Hidden Markov Models for Interactive Learning of Hand Gestures

Project Objectives and Current Work

We have developed a gesture recognition system, based on Hidden Markov Models (HMMs), which can interactively recognize gestures and perform online learning of new gestures. In addition, it is able to update its model of a gesture iteratively with each example it recognizes. This system has demonstrated reliable recognition of 14 different gestures after only one or two examples of each. The system is currently interfaced to a Cyberglove for use in recognition of gestures from the sign language alphabet. The system is being implemented as part of an interactive interface for robot teleoperation and programming by example.

Figure 1: Final position of hand for gesture "C".

The motivations behind this kind of interaction include increasing the effectiveness of teleoperation, enabling people to interactively teach robots new tasks or refine their skills, and allowing people to more effectively control systems such as semi-autonomous airplanes or systems for automated monitoring of industrial plants.

As people interact with machines which are autonomous and highly complex, they must be allowed to focus their attention on the content of their interaction rather than the mechanisms and protocol through which the interaction occurs. This is best accomplished by making the style of interaction more closely resemble that to which they are most accustomed: interaction with other people. We believe that online recognition of gestures and online training for gesture recognition are important components of this kind of interaction.

Approach

Our goal is to make a system which can not only interact with a user by accurately recognizing gestures, but which can learn new gestures and update its understanding of gestures it already knows in an online, interactive manner. Our approach is automated generation and iterative training of a set of Hidden Markov models which represent human gestures. Using this approach, we have built and tested a system which recognizes letters from the sign language alphabet using a Virtual Technologies `Cyberglove.'

Figure 2: Interface for using gesture recognition demonstration system.

In our current system, each kind of gesture is represented by an HMM, a list of example observation sequences, and an optional action to be performed upon recognition of the gesture. Our concept of interactive training is currently based on the following general procedure:

1. The user makes a series of gestures.
2. The system automatically segments the stream of data from the input device into separate gestures, and immediately tries to classify each gesture.
   1. If the system is certain about its classification of a gesture, it performs an action associated with that gesture (if one has been specified). Such an action could be be passing the result of the classification to a higher-level HMM, or sending a command to a robot.
   2. If the system is in any way unsure about its classification of a gesture, it queries the user for confirmation of its classification. The user either:
      • confirms the system's classification
      • corrects the classification, or
      • adds a new kind of gesture to the system's bank of gesture models.
3. The system adds the symbols of the encoded gesture to the list of example sequences of the proper gesture model, then updates the parameters of that model by retraining the HMM on the accumulated example sequences.

We have found that in our implementation, recognition of the gesture and automatic update of the HMM through the Baum-Welch algorithm is fast enough not to be noticeable during normal use of the system. This provides the system with a truly interactive character.

Future Research

Project Researchers

• Christopher Lee
• Yangsheng Xu

Publications List

• Christopher Lee, Yangsheng Xu. "Online, Interactive Learning of Gestures for Human/Robot Interfaces." 1996 IEEE International Conference on Robotics and Automation, Minneapolis, MN. vol. 4, pp 2982-2987. (Abstract, Postscript).