16-421: Vision Sensors, Spring 2009

General Information

Time : Tuesdays and Thursdays, 1:30 pm -- 2:50 pm

Location: Porter Hall A20A (NOTE CLASS ROOM CHANGE!!)

Credits : 12

Pre-requisites : Linear (or Matrix) algebra, OR any Vision or Graphics course, OR Instructors Permission

Instructor

Srinivasa Narasimhan

http://www.cs.cmu.edu/~srinivas

Email: srinivas@cs.cmu.edu

Office: NSH 4117

Office Hours: By appointment

Overview

This course covers the fundamentals of vision cameras and other sensors - how they function, how they are built, and how to use them effectively. The course presents a journey through the fascinating five hundered year history of "camera-making" from the early 1500's "camera obscura" through the advent of film and lenses, to today's mirror-based and solid state devices (CCD, CMOS). The course includes a significant hands-on component where students learn how to use the sensors and understand, model and deal with the uncertainty (noise) in their measurements. While the first half of the course deals with conventional "single viewpoint" or "perspective" cameras, the second half of the course covers much more recent "multi-viewpoint" or "multi-perspective" cameras that includes a host of lenses and mirrors.

Lectures and Readings

Lecture 1: History of Cameras and Images

Lecture 2: Camera Obscura and View Camera [Readings]

Lecture 3: Optical Elements of Imaging

Lecture 4: Lab demonstration - Lens undistortion

Lecture 5: Camera Calibration: Geometric (Thanks to Martial Hebert)

Lecture 6: Camera Calibration: Radiometric

Lecture 7: High Dynamic Range Imaging [HDRShop] [Fiat Lux] (Thanks to Paul Debevec)

Reading: Computational Cameras

Lecture 8: Programmable Imaging I (Thanks to Shree Nayar and CAVE Lab)

Lecture 9: Programmable Imaging II (Thanks to Shree Nayar and CAVE Lab)

Lecture 10: Lab Demonstration (Beam Splitters, NSH 2204)

Lecture 11: Structured Light and Range Imaging

Lecture 12: Lab Demonstration (Minolta Range Scanner, Martial Hebert's Lab)

Lecture 13: Super-resolution using Jitter Camera and Hybrid Imaging (Thanks to Shree Nayar and CAVE Lab)

Lecture 14: Deblurring: Flutter Shutter (Thanks to MERL)

Lecture 15: Camera Arrays (Thanks to Stanford Graphics Lab)

Lecture 16: Catadioptric Cameras (Thanks to Rahul Swaminathan, Ko Nishino and CAVE Lab)

Lecture 17: Stereo with Catadioptric Cameras with Planar Mirrors (Thanks to Joshua Gluckman)

Lecture 18: Polarization Imaging (Thanks to Yoav Schechner)

Lecture 19: Photo Tourism Google TechTalk (Thanks to Steve Seitz's group)

April 9-May 1: Student presentations of papers

Light Field Photography with a Hand-Held Plenoptic Camera
Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illumination
Rendering for an Interactive 360 Light Field Display
Compressive Imaging: A New Single Pixel Camera
Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing
Motion-Invariant Photography
3D Display Using Passive Optical Scatterers
Lighting Sensitive Display
Volumetric display with dust as the participating medium

Assignments

Geometric Calibration of a camera: Use the matlab calibration toolbox and a printed checker-board pattern to calibrate the camera intrinsic and extrinsic matrices. The re-projection errors must be very small. Create a webpage with your results. For extra credit, calibrate two cameras and apply the stereo algorithm for 3D reconstruction. (Due Feb 10)

Radiometric Calibration of a camera: Capture images of a color chart under uniform lighting with different exposures. Plot the response function of the camera. Create high dynamic range images of a scene with many exposures. (Due Feb 24)

The World in Eyes: Capture a high-res image of the cornea of the eye and compute the environment map and the retinal image. Use the VisualEyes software (thanks to Ko Nishino and CAVE Lab). (Due April 9)

List of Topics

PART 1: Perspective Sensors

Basic Principles of Still and Video Cameras
- CCD, Film, CMOS Sensors
- Electronics (A/D conversion, integration time, sampling, etc)
- Noise
- Device response
- Focus and Depth of Field
- Scheimpflug Photography
Basic Principles of Optical Elements: Filters and Lenses
- Filters (Polarizers, Neutral Density, Linear Interference)
- Fish Eye Lenses
- Single Lens Reflex
- Lens Distortions, Vignetting, Chromatic Abberations
- Autoexposure, Autofocus, Optical Stabilization
- Optical Transfer Function (OTF/MTF)
- Diffraction-limited Imaging
Camera Calibration
- Projection Fundamentals and Image Formation
- Geometric Calibration
- Radiometric Calibration
CCD Demosaicing
High Dynamic Range Imaging
- Sequential change in exposures
- Spatially varying exposures
- Real time exposure control over space and time
Other sensors
- Range Finders (structured light, time-of-flight)
- Gated Imaging

PART 2: Non-Perspective Sensors

Light Field
- Plenoptic function
- Light Field Parameterization
Single Viewpoint Catadioptric Cameras
- Omni-cameras
Multi-viewpoint systems
- Mathematical background on Caustics
- Refracting/Reflecting Caustics
- Multi perspective Images
Mosaicing
- Panoramic/Stereo Mosaicing
- Pushbroom cameras
Projectors and Displays
- Stereoscopic Displays
- 3D displays
- DMD for cameras, displays, projectors
- Projector-Camera Systems

Grading

Three Assignments 60%
Paper Presentation 20%
Final Project 20%

[Acknowledgements] A significant part of this course is similar to the courses offered at Stanford (Pat Hanrahan, Marc Levoy, Ron Fediw), UC San Diego (Henrik Wann Jensen), Columbia (Shree Nayar, Peter Belhumeur, Ravi Ramamoorthi), UW Madison (Chuck Dyer), UWash (Steve Seitz), Utah (Pete Shirley), Rutgers (Kristin Dana), Cornell (Steve Marschner, Kavita Bala), Technion (Yoav Schechner), Princeton (Szymon Rusinkiewicz), MIT (Ted Adelson, Ramesh Raskar, Bill Freeman), Drexel (Ko Nishino), TU Berlin and Deutsch Telecom (Rahul Swaminathan) The instructor thanks the instructors of these courses for the materials (slides, content) used in this course. In addition, several photographs and illustrations are borrowed from internet sources. The instructor thanks them all.

[Permission to use/modify materials] The instructor gladly gives permission to use and modify any of the slides for academic and research purposes. Since a lot of the material is borrowed from other sources, please acknowledge the original sources too. Finally, since this is a continuously evolving course, all suggestions and corrections (major, minor) are welcome!