Abstract: An algorithm using edge detector to detect the region of video objects in key frames of a video sequence and to build reference objects for the subsequent motion-estimate and compensation is proposed. First, the edge detectors of both Marr-Hildreth and Canny are used to build an edge image from the original key frame. Then the edge image is scanned to form mask. Finally, the original image is “AND” operated with the mask image to extract the video object. The algorithm is useful for many applications, especially for video conference and news. The experiment results show the algorithm based on Canny edge detector is better than the Marr-Hildreth and presents better reference frame for the whole video sequence.