Generating Spherical Mosaic using Calibrated Pan/Tilt Camera

Yanghai Tsin

1. Calibration of the Camera

The idea is to "register the images" beforehand by calibration.

I am using accurate pan/tilt cameras. The pan/tilt unit I am using has a resolution of 0.01 degrees/position. A pure rotation camera model is assumed. That is, I am neglecting the parallax introduced by camera translation. Compared to the distance of the objects of interest, this assumption is reasonable.

Pairs of images are taken. Their corresponding pan/tilt angles are read. The angle readings are trusted. Given the camera internal/external camera parameters, one image in a pair can be warped to the other. The camera parameters are estimated by minimizing the Sum of Squared Differences (SSD) of the pairs of images. The parameters I calibrated are focal length, image center, aspect ratio, radial distortion, and rolling angle of the camera. The calibration turns out to be accurate enough for image mosaicing purpose. Specifically, the standard deviation of the focal length is only a couple of pixels, while the focal length itself is about 400 pixels for the 320x240 images. For more information about the calibration step, please see my CVPR paper [1].

2. Taking images

44 images are taken and their corresponding pan/tilt angles are recorded. Notice in my example the camera AGC is not turned off. If the exposure is fixed, the high dynamic outdoor scene can not be captured.

3. Projecting to a sphere

Each of the 44 images is projected to a sphere. Each row in the image has the same altitude. And each column in the image has the same longitude.

For the overlap part simple average is taken. Notice the block effect is quite obvious, due to the AGC of the camera.

4. Image blending.

To smoothly stitch the images, I take the right bottom frame as reference. When a new image is pasted to the mosaic, I transform its intensity to that of the reference using a linear model. The parameters are estimated using weighted least squares. Feathering technique is used in blending the images.

5. Discussions

Images are "registered" before they are taken. That is, given the pan/tilt angles, the correspondence between each pixel in the image with the pixel on the mosaic has been determined, if the camera is properly calibrated.
Since the projecting step doesn't need further registration, building mosaic can be fast. In my case it takes less than 1 second for pasting an image. No optimization of the code has been attempted. All calculations are done using C++ with floating point.
Using this algorithm we can do image mosaicing for non-textured images. If exposure of the camera is fixed, we can do image mosaicing for dynamic scenes (scenes with moving objects). The still part of the mosaic will remain sharp.
To be improved: The color of the blended image is washed off. Instead of transforming RGB separately, I should first transform the RGB to YUV, blend Y compoent and transform back to RGB.
Drawbacks: calibration is critical. When pure rotation model assumption is violated, the algorithm fails. Limited to pan/tilt cameras.

6 Reference:

[1] R. Collins and Y. Tsin, "Calibration of an outdoor active camera", CVPR '99, Fort Collins.

Created: 9/19/99

Last update: 9/20/99