15-869, Image-Based Modeling and Rendering

THIS COURSE WILL NOT BE TAUGHT FALL 2001 due to Paul Heckbert's leave of absence and Jianbo Shi's other commitments. We apologize for the change of plans. We do not know when the course will next be taught. 9 Aug 2001

Professors: Steve Seitz and Paul Heckbert
Time: MW 10:30 - 11:50, Sept. 8 - Dec. 1, 1999
Place: Wean 5409B

Announcements:

old announcements

Administrative Info (prerequisites, grading, reading)
Syllabus

Programming Assignment 1: Image Mosaicing .
The results!

Programming Assignment 2: View Transformation .
The results!

Programming Assignment 3: Single View Modeling .
The results!

Final Research Project.
The results!

Class Schedule

Wed. Sept. 8.
Introduction
- what are image-based techniques?
- administrative matters, questionnaires
- preview of image mosaicing
- Assign presenter for Szeliski paper
Mon. Sept. 13.
- image mosaics
  - applications: remote sensing of earth and planets, panoramas, superresolution, video
  - projective mosaics: planar scenes and panoramas
- summarize image mosaicing assignment.
- image compositing (feathering / weighted averaging)
- image warping
  - transformations: rigid body, affine, projective, other
  - brief review of homogeneous coordinates
  - algorithms for image warping: forward, backward, two-pass
  - resampling, bilinear interpolation
Wed. Sept. 15.
- Reading for today: Image Mosaicing for Tele-Reality Applications, Richard Szeliski, Digital CRL 94/2, 1994. You can skim sections 6 and 7. Chris Rodriguez will present the paper.
- nonlinear optimization, Levenberg-Marquardt
- image registration for mosaicing
  - manual methods
  - automatic methods: sum of squared differences, mutual information
Mon. Sept. 20.
Image Mosaicing assignment due Sun. Sept. 19 at midnight.
- image registration (part 2)
- image morphing: Beier-Neely and other methods
Wed. Sept. 22.
- Discuss image mosaicing results from assignment 1.
- How to use the laser scanner for assignment 2.
- z-buffer
- Reading for today: View Interpolation for Image Synthesis, Eric Chen and Lance Williams, SIGGRAPH '93, ( PDF without color figures). Color copies of this paper for everyone in the class are available at Cleah's office (Doherty 4309).
  Presented by Mike Montemerlo.
Mon. Sept. 27.
- Head-Tracked Stereoscopic Display Using Image Warping, Leonard McMillan and Gary Bishop, Stereoscopic Displays and Virtual Reality Systems II, SPIE Proceedings 2409, Feb. 1995.
  Presented by Howard Leung.
- environment maps: cylinder, sphere, cube
- cylindrical panoramas
- Quicktime VR
Wed. Sept. 29.
- Reading: Layered Representation for Motion Analysis, John Y. A. Wang and Edward H. Adelson, CVPR 93.
  Presented by Kiran Bhat.
- see also Wang's web page with images and quicktime movies:
- occlusion, hole filling problem
- sprites (images in layers)
- automatic extraction of image layers from video
- layered depth images (LDI's)
Mon. Oct. 4.
View Transformation assignment due Sun. Oct. 3 at midnight.
- projective geometry
- Single View Metrology, Antonio Criminisi, Ian Reid, and Andrew Zisserman, ICCV 99.
  - Nice HTML presentation of this work
  - see Criminisi's examples on the web
- assignment 3 summary
Wed. Oct. 6.
- Discuss view transformation results (asst. 2).
- texture mapping, bump mapping, displacement mapping. See some example pictures.
- Reading: Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach, Paul E. Debevec, Camillo J. Taylor, Jitendra Malik, SIGGRAPH '96.
  Presented by Terence Sim.
- See also additional pictures and videos from SIGGRAPH '99 course notes.
Mon. Oct. 11.
no class - Mid-semester break.
Wed. Oct. 13.
- Ray-based representations (4D)
- light fields: acquiring, compression, rendering (Levoy, Gortler)
- Reading: Light Field Rendering, Marc Levoy and Pat Hanrahan, SIGGRAPH '96.
  Presenter: Mike Schuresko.
- You may want to also check out a very similar paper, The Lumigraph, by Gortler, Grzeszczuk, Szeliski, and Cohen, SIGGRAPH '96.
Mon. Oct. 18.
- the many-dimensional nature of radiance
- moving beyond diffuse, opaque surfaces
- light and color
- bi-directional reflectance distribution functions (BRDF's)
- environment maps
- Reading: Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography, Paul E. Debevec. SIGGRAPH 98.
  Presenter: Jon Rowlett.
  Browse also: History of Reflection Mapping, Paul Debevec.
Wed. Oct. 20.
Single View Modeling assignment due Tue. Oct. 19 at midnight.
- discussion of research projects: project topics, paper and talk requirements
Mon. Oct. 25.
- Discuss results of Single View Modeling assignment.
- global illumination and radiosity
Wed. Oct. 27.
- Reading: Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs, Yizhou Yu, Debevec, Malik, Hawkins, SIGGRAPH 99.
- projective geometry, epipolar geometry, the fundamental matrix
- View Morphing, Proc. SIGGRAPH 96, 1996, pp. 21-30.
- project meetings with Paul or Steve today or tomorrow. (you should have scheduled this)
- project proposals due by email on Fri, Oct 29 to Paul AND Steve. See instructions.
Mon. Nov. 1.
- Guest speaker: Chris McGlone on photogrammetry. See also his slides and links.
- projective geometry
Wed. Nov. 3.
- view morphing
- brief summary of chosen research project topics
- discussion/brainstorming image based modeling approaches (mosaics, view transformation, sprites, light fields, ...)
Mon. Nov. 8.
- texture analysis / synthesis (Markov random fields, other methods)
- Reading: Texture Synthesis by Non-parametric Sampling, Alexei Efros and Thomas Leung, IEEE Intl. Conf. on Computer Vision (ICCV'99), Sept. 1999.
  Presenter: Liu Ren.
- Reading: Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images, Jeremy De Bonet SIGGRAPH '97.
  Presenter: Jing Xiao.
Wed. Nov. 10.
- Reading: A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment, Takeo Kanade and Masatoshi Okutomi, Proc. International Conference on Robotics and Automation, 1991.
  Presenter: Li Zhang.
- stereo matching
- optical flow
- Visit to Takeo Kanade's Virtualized Reality Lab. Demo by Sundar Vedula.
Mon. Nov. 15.
- Students each give 3 minute progress report on their research. See projects page for a description of what is expected.
- Paul Debevec will show video examples of image-based modeling and rendering
- Paul Debevec will give a Robotics Seminar, Image-Based Modeling, Rendering, and Lighting in "Fiat Lux", Porter 100, 4:30pm.
Wed. Nov. 17.
- shape from motion (feature tracking, factorization method). See CMU Modeling by Videotaping web page.
- Reading: Rapid octree construction from image sequences. R. Szeliski, CVGIP: Image Understanding, 58(1):23-32, July 1993. (This paper is apparently not available online.)
  Presenter: Tad Drake.
- shape from silhouettes
- voxel coloring
Mon. Nov. 22.
- Reading: A Morphable Model for the Synthesis of 3D Faces, Volker Blanz and Thomas Vetter, SIGGRAPH'99.
  Presenter: Tom Kang.
- Reading: Voice Puppetry, Matthew Brand, SIGGRAPH'99.
  Presenter: Xuewen Chen.
- active vision, 3-D scanner technology
Wed. Nov. 24.
no class - Thanksgiving vacation.
Mon. Nov. 29.
- Student research presentations, Smith Hall 2nd floor, 3:30-5:30pm. Note special time and place! Each talk 20 min. total. (The links below point to progress reports from early November.)
  
  Terence Sim: Illuminating the Face
  
  Kiran Bhat: 3-D Mosaicing from Stereo Video
  
  Tad Drake: Generating Images with Arbitrary Focus Regions
  
  Li Zhang: Real-Time Image Mosaicing
  
  Mike Schuresko: Single View Modeling for Nonplanar Surfaces
Wed. Dec. 1.
- Student research presentations, Wean 5409B, 10:30-12:30. Note that class will end later than usual. Each talk 20 min. total.
  - Liu Ren: Texture Synthesis for Hole Filling
  - Xuewen Chen: Texture Synthesis of 3-D Surfaces
  - Jing Xiao: Video Texture Synthesis
  - Jon Rowlett: Using Texture Synthesis to Extract Occlusions from Images
  - Chris Rodriguez: Image Skeletonization for Motion Capture and Interaction
  - Tom Kang: Recovery of Camera Motion from Camera-Blurred Images
- written reports due Fri. Dec. 3

Steve Seitz and Paul Heckbert, Nov. 1999