Abstract: The main retrieval method of content-based video retrieval system is query by example. If user submits a video as example, system returns a set of similar videos. But how to define whether two videos are similar is still a great problem. This paper puts forward a video similarity model to solve the difficulty. First, it advances the centroid feature vector of shot in order to reduce the storage of video database. Second, considering the latent factors existing in human's vision perception, it introduces a new comparison algorithm based on multi-level of video structure, such as from shot's view and from the overall view. This different granularity of measurement provides great flexibility, which is reasonable in real world. The final retrieval result demonstrates the validity of algorithm.