A Multi-modal Game Player Interface System Based on Video and Speech

In order to enhance game's interactivity and player's immersion in 3D computer games, we developed an easy-to-use and cost-effective interface system based on video and speech. This system integrates three modules, personalized 3D face modeling, head pose estimation and Chinese speech recognition. The first module allows a player to generate his or her personalized character with minimal time and manual interaction, the second and third modules are incorporated to control the character using player's head poses and speech commands in real-time. We also developed a multiplayer game demo, 3D Chinese chess, as a test bench to test our system. The test results demonstrate that the approaches and the system are feasible and practical.