Abstract: Video caption could be used to index video stream with high-level semantics since it implied lots of semantics inherently．This paper presents an algorithm to localize and extract video caption using support vector machine（SVM）automatically．First the raw video frame is segmented into N×N sub-blacks，and the gray level value features are extracted from each sub-blacks；then a pre-trained SVMis used to classify each sub-blacks into two classes： caption and non-caption；finally pyramid model and post-processing process are applied to get the caption area in the video frame automatically．The experiment results show this algorithm works well．