Abstract: In the video-based human animation，the effect of feature tracking affects the vividness of reconstructed 3D movement information．This paper offers a new two-camera based tracking algorithm which uses Kalman filter to estimate the coordinates of feature，then makes use of sub-block based model to match features，and uses epipolar line equation to guide tracking and to rule out the feature point with big error．The presented algorithm has been applied to VBHA system，the experimental results demonstrate that our algorithm can meet the requirements of speed and efficiency of video-based animation and can solve self-occultation problem partially．