Overview
Everyday, we observe an extraordinary array of light and color phenomena around us,
ranging from the dazzling effects of the atmosphere, the complex appearances of
surfaces and materials, and underwater scenarios. For a long time, artists, scientists,
and photographers have been fascinated by these effects, and have focused their
attention on capturing and understanding these phenomena. In this course, we take
a computational approach to modeling and analyzing these phenomena, which we
collectively call "visual appearance". The first half of the course focuses on
the physical fundamentals of visual appearance, while the second half of the
course focuses on algorithms and applications in a variety of fields such as
computer vision, graphics and remote sensing and technologies such as underwater
and aerial imaging.
This course unifies concepts usually learnt in physical sciences and their application
in imaging sciences. Students attending this course will learn about the fundamental
building blocks that describe visual appearance, and recent academic papers on a variety
of physics-based methods that measure, process, and analyze visual information from the
real world.
List of Topics
- Fundamentals of Appearance
- Principles of Photometry
- Light Fields
- Reflection, Refraction, Polarization, Diffraction, Interference
- Surface Reflection Mechanisms
- Signal Processing framework for Reflection
- Textures and Spatially Varying BRDFs (BTF)
- Lighting and Shadows
- Light Transport
- Caustics
- Scattering and Volumetric Light Transport
- Fluids
- Algorithms and Applications
- Photometric 'Shape-from-X' algorithms
- Image and Vision-based Rendering
- Inverse Rendering
- Understanding and Measuring Light Transport
- Appearances of Transparent, Transluscent, Wet, Woven Surfaces
- Appearances of Atmospheric and Underwater Scattering Effects
- Appearances of Fluids - smoke, fire, water
- Vision in Bad Weather
- Applications in Aerial, Underwater, Medical and Microscopic Imaging
- Principles of Nature Photography
Optional Texts
- Light and Color in the
Outdoors, M. Minnaert.
Grading
- One Project 50%
- Three Paper Presentations 30%
- Reaction Reports 10%
- Class Participation 10%
Lecture Presentations
[Acknowledgements]
A significant part of this course is similar to the courses offered at
Stanford (Pat Hanrahan, Marc Levoy, Ron Fediw), UC San Diego (Henrik Wann
Jensen), Columbia (Shree Nayar, Peter Belhumeur, Ravi Ramamoorthi), UW
Madison (Chuck Dyer), UWash (Steve Seitz), Utah (Pete Shirley), Rutgers
(Kristin Dana), Cornell (Steve Marschner, Kavita Bala), Technion (Yoav
Schechner), Princeton (Szymon Rusinkiewicz), MIT (Ted Adelson), Drexel (Ko
Nishino), TU Berlin and Deutsch Telecom (Rahul Swaminathan). These slides
were largely put together in previous offerings of the course by Srinivasa Narasimhan.
The instructor thanks the instructors of these courses for the materials (slides,
content) used in this course. In addition, several photographs and
illustrations are borrowed from internet sources. The instructor thanks
them all.
[Permission to use/modify materials]
The instructor gladly gives permission to use and modify any of the
slides for academic and research purposes. Since a lot of the material is
borrowed from other sources, please acknowledge the original sources too.
Finally, since this is a continuously evolving course, all suggestions
and corrections (major, minor) are welcome!
WEEK 1: INTRODUCTION
- Lecture 1 (Jan. 15th): Introduction + Course Administration
[PPT]
[PDF]
- Lecture 2 (Jan. 17th): Basic Principles of Imaging and Photometry
[PPT]
[PDF]
WEEK 2: BRDF MODELS AND MEASUREMENTS
- Lecture 3 (Jan. 22nd):
Basic Principles of Surface Reflectance
[PPT]
[PDF]
Specular Reflections from Rough Surfaces
[PPT]
[PDF]
- Lecture 4 (Jan. 24nd):
WEEK 3: STUDENT PRESENTATION I
- Lecture 5 (Jan. 29nd):
- Lecture 6 (Jan. 31nd):
CMU classes cancelled due to extreme weather conditions
WEEK 4: STUDENT PRESENTATION I (continued) + LIGHTING
- Lecture 7 (Feb. 5th):
- Lecture 8 (Feb. 7th):
WEEK 5: STRUCTURED LIGHTING (guest speaker)
- Lecture 9 (Feb. 12th):
No lecture
- Lecture 10 (Feb. 14th):
Guest Lecturer: Joe Bartels, "Episcan, Epitof, and Light curtains"
WEEK 6: LIGHT TRANSPORT
- Lecture 11 (Feb. 19th):
- Lecture 12 (Feb. 21th):
WEEK 7: LIGHT TRANSPORT (continued) + STUDENT PRESENTATIONS II
- Lecture 13 (Feb. 26th):
- Lecture 14 (Feb. 28th):
WEEK 8: STUDENT PRESENTATIONS III
- Lecture 15 (Mar. 5th):
- Lecture 16 (Mar. 7th):
WEEK 9: SPRING BREAK (NO CLASSES)
WEEK 10: STUDENT PRESENTATIONS IV
- Lecture 17 (Mar. 19th):
- Lecture 18 (Mar. 21st):
WEEK 11: TIME-OF-FLIGHT IMAGING
- Lecture 19 (Mar. 26th):
Time-of-Flight Revolution
[PPT]
- Lecture 20 (Mar. 28th):
Time-of-Flight Revolution (continued)
[PPT]
WEEK 12: TRANSIENT IMAGING + STUDENT PRESENTATIONS V
- Lecture 21 (April 2nd):
Overview of Transient Imaging
- Lecture 22 (April 4th):
WEEK 13: STUDENT PRESENTATIONS VI
- Lecture 23 (April 9th):
- Lecture 24 (April 11th):
No lecture (Spring Carnival)
WEEK 14: POLARIZATION + STUDENT PRESENTATIONS VII
- Lecture 25 (April 16th):
Basic Principles of Light Polarization
[PPT]
[PDF]
- Lecture 26 (April 18th):
WEEK 15: STUDENT PRESENTATIONS VIII
- Lecture 27 (April 23rd):
- Lecture 28 (April 25rd):
WEEK 16: STUDENT PRESENTATIONS IX
- Lecture 29 (April 30th):
- Lecture 30 (May 2nd):
FINAL PRESENTATION
- Project presentations (May 9th):
Project deliverables:
[PDF]
Format: 8 min. presentation + 2 min. Q&A
Time: 8:30am - 11:30am
Location: Doherty Hall 1211
Remember to send full project paper by 11:59pm on May 9th!
Relevant Papers
Topic I: photometry, BRDF
- Example-Based Photometric Stereo: Shape Reconstruction with General, Varying BRDFs, 2005
- Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction, 2002
- Specularity Removal in Images and Videos: A PDE Approach, 2006
- Color Subspaces as Photometric Invariants, 2008
- Projection Defocus Analysis for Scene Capture and Image Display, 2006
- A Coaxial Optical Scanner for Synchronous Acquisition of 3D Geometry and Surface Reflectance, 2010
- Principles of Appearance Acquisition and Representation, 2009
- A Basis Illumination Approach to BRDF Measurement, 2010
- Time-varying Surface Appearance: Acquisition, Modeling, and Rendering, 2006
- Generalization of the Lambertian Model and Implications for Machine Vision, 1994
- Surface Reflection: Physical and Geometrical Perspectives, 1989
- A Perception-based Color Space for Illumination-invariant Image Processing, 2008
- Microgeometry Capture using an Elastomeric Sensor, 2011
- A New Perspective on Material Classification and Ink Identification, 2014
- A Dictionary-based Approach for Estimating Shape and Spatially-Varying Reflectance, 2015
- Printing Spatially-Varying Reflectance for Reproducing HDR Images, 2012
- Fabricating BRDFs at High Spatial Resolution Using Wave Optics, 2013
- Light-Efficient Photography, 2011
- A Digital Gigapixel Large-Format Tile-Scan Camera, 2011
Topic II: signal processing, light field, lighting, shadows
- A Signal-Processing Framework for Inverse Rendering, 2001
- Estimating Natural Illumination from a Single Outdoor Image, 2011
- Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering Using Multi-flash Imaging, 2004
- Digital Photography with Flash and No-Flash Image Pairs, 2004
- Multiplexing for Optimal Lighting, 2007
- Light Field Transfer: Global Illumination Between Real and Synthetic Objects, 2008
- Light Field Microscopy, 2006
- Synthetic Aperture Confocal Imaging, 2004
- Light Field Rendering, 1996
- Light Field Analysis for Modeling Image Formation, 2011
- Light field photography with a hand-held plenoptic camera, 2005
- Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing, 2007
- Analytic PCA construction for theoretical analysis of lighting variability in images of a Lambertian object, 2002
- From Few to Many: Illumination Cone Models for Face Recognition under Variable lighting and Pose, 2001
- Webcam Clip Art: Appearance and Illuminant Transfer from Time-lapse Sequences, 2009
- The Bas-Relief Ambiguity, 1999
- What is the Set of Images of an Object Under All Possible Lighting Conditions?, 1998
- Light Fall-off Stereo, 2007
- Coplanar Shadowgrams for Acquiring Visual Hulls of Intricate Objects, 2007
- Compressive Light Field Photography, 2013
Topic III: light transport
- Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination, 2006
- Shape from Interreflections, 1990
- Dual Photography, 2005
- Optical Computing for Fast Light Transport Analysis, 2010
- A Theory of Inverse Light Transport, 2005
- A Combined Theory of Defocused Illumination and Global Light Transport, 2011
- Shape from Second-bounce of Light Transport, 2010
- Programmable Automotive Headlights, 2014
- A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus, 2012
- Compressive Light Transport Sensing, 2009
Topic IV: light transport
- Homogeneous Codes for Energy-Efficient Illumination and Imaging, 2015
- Micron-scale Light Transport Decomposition Using Interferometry, 2015
- Interreflection Removal Using Fluorescence, 2014
- A Dual Theory of Inverse and Forward Light Transport, 2010
- Recovering ThreeDimensional Shape around a Corner using Ultra-Fast Time-of-Flight Imaging, 2012
- A Practical Analytic Model for the Radiosity of Translucent Scenes, 2013
- Frequency-Space Decomposition and Acquisition of Light Transport under Spatially Varying Illumination, 20112
- Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport, 2014
- Primal-Dual Coding to Probe Light Transport, 2012
- Frequency Analysis of Transient Light Transport with Applications in Bare Sensor Imaging, 2012
- 3D Shape and Indirect Appearance by Structured Light Transport, 2014
Topic V: (mirror) reflection and refraction
- A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation, 2008
- State of the Art in Transparent and Specular Object Reconstruction, 2008
- Fluorescent Immersion Range Scanning, 2008
- A Multi-layered Display with Water Drops, 2010
- Reflection Removal using Ghosting Cues, 2015
- What a single light ray reveals about a transparent object? 2015
- Depth from Optical Turbulence, 2012
- STELLA MARIS: Stellar Marine Refractive Imaging Sensor, 2014
- Triangulation in Random Refractive Distortions, 2013
- Refraction wiggles for measuring fluid depth and velocity from video, 2014
- Image invariants for smooth reflective surfaces, 2010
- Specular surface reconstruction from sparse reflection correspondences, 2012
- Angular Domain Reconstruction of Dynamic 3D Fluid Surfaces, 2012
- Seeing through Water: Image Restoration using Model-based Tracking, 2009
Topic VI: polarization, scattering
- Polarization-Based Vision through Haze, 2003
- Separation of transparent layers using focus, 2000
- Acquiring Scattering Properties of Participating Media by Dilution, 2006
- Structured Light in Scattering Media, 2005
- Vision in Bad Weather, 1999
- DISCO - Acquisition of Translucent Objects, 2004
- Clear underwater vision, 2004
- What Do the Sun and Sky Tell Us About the Camera? 2010
- 3Deflicker from Motion, 2013
- Self-Calibrating Imaging Polarimetry, 2015
- Polarized 3D: High-Quality Depth Sensing With Polarization Cues, 2015
- Single image haze removal using dark channel prior, 2009
- On the Appearance of Translucent Edges, 2015
- Airborne Three-Dimensional Cloud Tomography, 2015
- Recovering Inner Slices of Translucent Objects by Multi-frequency Illumination, 2015
- Surface Normal Deconvolution: Photometric Stereo for Optically Thick Translucent Objects, 2014
- Shape from Single Scattering for Translucent Objects, 2012
- Compressive Structured Light for Recovering Inhomogeneous Participating Media, 2008