Abstract: An automatic and accurate moving object extraction algorithm is proposed to segment head-shoulder video sequence. First, an improved background registration technique is provided to construct a reliable background image. The moving object region is then detected by combining the background difference with the frame difference. Then, a new temporal filter and a post-processing step are applied on the detected change to improve the temporal coherence and spatial integrity. Finally,an improved watershed algorithm is proposed to locate the precise video object boundary, with significant reduction of computation compared to conventional watershed algorithm. Experimental results show that the proposed method can segment the head-shoulder video object, provide accurate object boundary and meet the real-time requirement.