Abstract: Considering that human being needs relying on all kinds of semantics to make correct judgments when reading the drawing, this paper presents a set of methods based on object figure semantics, joint relationship semantics and reference semantics to recognize the architectural structural drawing and reconstruct the building’s model. Taking beam as an example, the related information of wall, column, floor, reinforcement, symbols and annotation are worthy of attention too. The general principles and rules of recognition are illustrated in detail.