A PDF version of this report is here
The problem addressed by this project is how to synthesize an environment map for lighting a synthetic object, given only the backdrop image in front of which the object will be placed.
Consider the pair of images below. Both images contain the same geometry, but the image on the left was illuminated using a standard stage lighting model, while the image on the right was illuminated using an environment map consistent with the image's backdrop.
The difference in believeabiltiy of these scenes is the motivation for addressing this problem.
The following section describes how to transform the image below into an environment map. An environment map is a function on the sphere, so the choice of representation of this function will affect the algorithm. The representation chosen is a Mercator projection. Consider the image below.
This image represents the intersection of an environment map centered at the viewer of the scene with a plane a given distance away. The first step in the algorithm is to warp this image to a cylindrical projection around the viewer. The resulting image is shown below.
Since objects will be inserted in front of the viewer, the assumption is made that this cylindrical projection represents 180 degrees of the cylindrical environment map centered around the object that will be inserted. To synthesize the other 180 degrees, the image is simply flipped horizontally and concatenated with itself. The resulting image is shown below.
A significant portion of the environment map, namely the details at the top and bottom of the map, are completely absent. To synthesize them, the texture from the cylindrical projection is stretched and repeated.
Because of the singularities at the poles of the spheres, and because it is desirable to have the environment map be continuous, a nonlinear filter is applied to the image, blurring the top and bottom of the image heavily while leaving the horizon untouched.
If the environment map is re-projected onto a cube, the result is as follows.
Finally, this environment map can be used to illuminate synthetic objects
In the following set of images, the image on the left was illuminated using a synthesized environment map while the image on the right was illuminated using an environment map captured from the location where the photograph was taken. Differences between the images are minimal.
A fault with this method is that if the image from which the environment map is being synthesized is not high-dynamic range, then the method described will not create a natural-looking environment map. As an example, consider the following low-dynamic range backdrop image.
The image on the left shows a matte ball being illuminated by the environment map synthesized by the method described. If high dynamic range is inserted (in this case by increasing the values of all saturated pixels), then the result is the image on the right.
Another fault of this method is that it depends entirely upon having the light sources for the scene present in the image from which the environment map is constructed. For example, consider the scene below.
The image on the left shows the scene illuminated with the synthesized environment map, while the image on the right shows the scene illuminated by the true environment map captured at the location where the picture was taken. The synthesized environment map has destroyed almost all sources of illumination from the image. Furthermore, even if there were cues to the true environment map (for example, a matte ball that was being light from above), they would be ignored by the algorithm.
The algorithm presented works well for select backdrop images, namely those that contain their illumination sources, but is certainly not a general solution. It is also questionable whether or not this compares favorably in terms of labor to having an artist create an environment map. But rather than dwell on such things, let's just look at more pretty pictures
Original backdrop image.
Simple non-interactive synthetic scene.
A matte cow (with shadow)
A shiny cow.
