Appearance Analysis

Near-Periodic Patterns (NPP) are ubiquitous in man-made scenes and are composed of tiled motifs with appearance differences caused by lighting, defects, or design elements. In this paper, we present a periodicity-aware framework to learn NPP representation, which enables various applications including NPP completion, resolution-enhanced remapping, and segmentation.

We describe a novel approach to decompose a single panorama of an empty indoor environment into four appearance components: specular, direct sunlight, diffuse and diffuse ambient without direct sunlight. Our system is weakly supervised by automatically generated semantic maps. A GAN-based approach supervised by coarse information obtained from the semantic map extracts specular reflection and direct sunlight regions on the floor and walls. These lighting effects are removed via a similar GAN-based approach and a semantic-aware inpainting step. The appearance decomposition enables multiple applications including sun direction estimation, virtual furniture insertion, floor material replacement, and sun direction change, providing an effective tool for virtual home staging.

In this work, we propose a two-stage near-light photometric stereo method using circularly placed point light sources (commonly seen in recent consumer imaging devices like NESTcam, Amazon Cloudcam, etc). Because of the small light source baseline, the change of image intensity for the input image is small. In addition, in the near-light condition, the distant light assumption fails. So the light directions and intensities are not evenly distributed across the scene. Our proposed method tackles both issues.

We develop a novel deep learning framework to simultaneously transform images across spectral bands and estimate disparity. A material-aware loss function is incorporated within the disparity prediction network to handle regions with unreliable matching such as light sources, glass windshields and glossy surfaces. No depth supervision is required by our method. To evaluate our method, we used a vehicle-mounted RGB-NIR stereo system to collect 13.7 hours of video data across a range of areas in and around a city. Experiments show that our method achieves strong performance and reaches real-time speed.

A new dataset that has 100 materials captured under under 12 different NIR lighting directions with 9 different viewing angles. A low-parameter BRDF model (in NIR) and fine scale geometry of surfaces are estimated simultaneously.

David Marr inspired hierarchical model of part mixtures is used to sample natural looking human poses. The model is learned from a human image dataset and can be used to estimate human pose from a single test image.

Imaging and measurement of crop and canopy in vineyards.

We present a novel technique to reconstruct the surface of the bone by applying shape-from-shading to a sequence of endoscopic images, with partial boundary in each image.

Scene points can be clustered according to their surface normals, even when the geometry, material and lighting are all unknown.

In this paper, we analyze what kinds of depth cues are possible under uncalibrated near point lighting.

We derive a new class of photometric invariants that can be used for a variety of vision tasks including lighting invariant material segmentation, change detection and tracking, as well as material invariant shape recognition.