Temporal Shape-From-Silhouette

German Cheung, Simon Baker, Takeo Kanade

Please contact German Cheung at german@ux2.sp.cs.cmu.edu for further details

Return to Homepage RealTime 3D Reconstruction Temporal SFS Human Kinematic Modeling and Motion Capture Human Motion Transfer

Objective

Although Shape-From-Silhouette (SFS) is a popular 3D reconstruction method, the shape estimated from SFS can be coarse if there are only a few number of cameras. Better shape estimates or Visual Hulls can be obtained using SFS if the number of distinct silhouette images is increased. Instead of increasing the number of physical cameras (the across space approach), the silhouette images which are captured over time are combined (across time approach). The Temporal Shape-From-Silhouette Algorithm consists of two tasks: Visual Hull Alignment and Visual Hull Refinement. The temporal SFS is first devised for rigid objects and then extended to articulated objects (with rigid parts). Technical details of this project can be found in the documents listed below:

Temporal Shape-From-Silhouette: Rigid Object

Step 1: Constructing Bounding Edges Step 2: Extracting Colored Surface
Points (CSPs) 		Step 3: Aligning Visual Hulls through CSPs Step 4: Visual Hull Refinement

Synthetic data: Torso sequence (Mpeg video of this sequence can be downloaded here)

One of the input images Bounding edges Unaligned CSPs Aligned CSPs Visual Hull built using 6 images Visual Hull built using 126 images

Real data: Pooh Sequence (Mpeg video of this sequence can be downloaded here)

One of the input images Bounding Edges Unaligned CSPs Aligned CSPs Visual Hull built using 6 images Visual Hull built using 90 images

Real data: Dinosaur-Bananas Sequence (Mpeg video of this sequence can be downloaded here)

One of the input images Bounding Edges Unaligned CSPs Aligned CSPs Visual Hull built using 6 images Visual Hull built using 90 images

Temporal Shape-From-Silhouette: Multiple and Articulated Objects

To extend the Temporal SFS algorithm to articulated object, the individual parts of the articulated object are treated as separate objects. The algorithm then iteratively segments the CSPs to each rigid part of the object and then estimates their motions using the rigid object temporal SFS algorithm. Once the motions of the rigid parts are recovered, the joint locations are estimated using joint constraints between

Synthetic data: Left Leg Joints of Computer Avatar (Mpeg video of this sequence can be downloaded here)

One of the input images Unaligned CSPs Aligned CSPs Aligned and Segmented CSPs with
estimated joint location

Real data: Pooh-Dinosaur Sequence (two separate, independently moving objects) (Mpeg video of this sequence can be downloaded here)

One of the input images Unaligned CSPs Aligned CSPs Segmented CSPs Visual Hull built using 16 images Visual Hull built using 104 images

Real data: Right Leg of Real Person (Mpeg video of this sequence can be downloaded here)

One of the input images Unaligned CSPs Aligned CSPs Aligned and Segmented CSPs with
estimated joint location

Real data: Right Arm of Real Person (Mpeg video of this sequence can be downloaded here)

One of the input images Unaligned CSPs Aligned CSPs Aligned and Segmented CSPs with
estimated joint location