3D Human Kinematic Modeling and Markerless Motion Capture

German Cheung, Simon Baker, Takeo Kanade

Please contact German Cheung at german@ux2.sp.cs.cmu.edu for further details

Return to Homepage

RealTime 3D Reconstruction

Temporal SFS

Human Kinematic Modeling and Markerless Motion Capture

Human Motion Transfer

Objective

A vision-based human kinematic modeling system is built using the Temporal Shape-From-Silhouettes Algorithms (details can be found here ). The acquired kinematic model is then used to perform non-invasive motion capture of motions from video sequences using an idea similar to that used in our Temporal SFS Algorithms. Technical details of this project can be found in the documents listed below:

CVPR2003 paper titled `` Shape-From-Silhouette of Articulated Objects and its Use for Human Body Kinematics Estimation and Motion Capture''
(Download: pdf)
Chapters 7 and 8 of German Cheung's PhD thesis titled ``Visual Hull Construction, Alignment and Refinement for Human Kinematic Modeling, Motion Capture and Rendering''
(Download: pdf (7.94M))

Human Kinematic Modeling System

There are three tasks to the human kinematic modeling system. The first task recovers all the joint locations of the body. The second task acquires precise shape information. The final task merges the joint and shape together to form a articulated model of the person.

Algorithms

Results

SubjectE

SubjectG

SubjectS

Subject

Joint
Estimation

Video

Joint
Skeleton

Shape
(Voxels)

Merged
Shape
and
Joints

Video

Human Markerless Motion Capture

The kinematic model acquired above is used to track new motions of the person by an Image-based Articulated Object Tracking Algorithm. Eight cameras are used in each sequence. Mpeg videos showing tracking results of two synthetic data sets (KICK and PUNCH sequences) and five real data sets of simple motions (STILLMARCH and AEROBICS sequences) and complex ones (KUNGFU, THROW, SLOWDANCE and STEP-FLEX sequences) are included below.

Synthetic Data Sets

KICK Sequence	PUNCH Sequence

Real Data Sets

STILLMARCH Sequence Video	AEROBICS Sequence Video	KUNGFU Sequence Video	THROW Sequence Video	SLOWDANCE Sequence Video	STEP-FLEX Sequence Video

Applications for Human Motion Transfer

We also apply the above kinematic modeling and tracking results to an interesting application called Human Motion Transfer.