Abstract: In this paper, a brief survey of knowledge-based animation techniques is given. Then we propose a video-based knowledge representation model for joint objects which includes: graphics object knowledge representation, action knowledge representation and videoflow knowledge representation, next we give the description of the system's UI frame based on the model proposed before. Finally, a conclusion is reached.