Project: Image stitching part 1 & 2

The project aims to showcase to solution to automated image stitching, given that that a user provides points to point correspondence for the pictures. The process is as such - I first take pictures of our subjects, usually taking a collection of pictures and enough pictures to form an image mosaic. These pictures are digitized. I then retreive the homography between the images that I want to form an image mosaic with. This is done by selecting a number of points which correspond to one another, and I need at least 4 points. The more points I choose, the better the homography will be. Then I perform an image warp on the image I want to warp to the base image. I can blend the two images using linear blending, or laplacian blending. I peformed both linear and laplacian blending, each with its own special effects.

The next part of the project is to automate the selection of the points. In part 1, I manually selected point-to-point correspondence for the two images, which can lead to inaccurate homographies as the human hand is incapable of such constant precision. Thus I perform a series of steps to obtain the best homography possible. First I obtain the entire set of Harris corners (using gaussian_1 of 4 and gaussian_2 of 6)from the two images, and perform adaptive non-maximal spatial filtering on these points to obtain the 500 points with the highest strength which are spatially distributed throughout the image. The strength of each neighbour must be 0.9 times that of the feature in order to be suppressed. Then I obtain an feature descriptor for each of these points(This is generated by applying a Gaussian filter to a 40x40 square around the point and subsampling it to get a 8x8 square). I then compare all features of image 1 to image 2, and retain the set of features that SSD(im1patch,im2patch)<=e where e is some epsilon. This threshold is obtained by averaging the error of the 2-nearest neighbour in order to get a good sense of the average outlier distance in the image. Lastly, I perform RANSAC on these feature points to choose the best 4 points to give us a final homography. It runs through some number of iterations, and in each iteration, chooses 4 points at random, computes a homography H from these 4 points. I then compute SSD(p', H p) <= e' , and keep the largest set of inliers which fulfil this condition. I used 2500 iterations and a e' of 0.7, which seemed to give the best effect.

I first tested my homography and image warping by trying to rectify certain objects that was known to be rectangular in shape. The images were warped to a rectangular shape.
This was a whiteboard with the words "hello world" on it.

This was a tile pattern on the floor of Wean Hall.

I then created image mosaics, or better known as panamoras, using the same technique. This was done incrementally i.e. first I stitched two images together, and using the new image, stitched it with another image of the image mosaic.

This image mosaic is captured down at Oakland.

This image mosaic is captured at the Software engineering institute.

This image mosaic is captured near the Cathedral of Learning.

There were several bells and whistles developed for this project. Firstly, I implemented laplacian blending, which took care of edge artifacts to a certain extent.

Original

This is the Laplacian-blended picture. You can see that the edge artifacts in the sky has disappeared. There are also less artifacts in the bottom of the picture, especially the fence.

The next cool thing I made was a graffiti implantation next to the Wean Cluster. I added some gradients and changed the background color of the graffiti using Photoshop, in order to better match the wall in Wean Cluster.

A giant propaganda poster in Wean of yours truly. I used Photoshop to add some noise and brush filter, in order to make it look as if it was truly painted.

The final result:

This is an example of the harris points generated for a single image.

This is an example of the interest points retained after performing ANMS on the harris points.

The first mosaic is of the cathedral down by oakland. This is the same subject as seen in part1. You can see that there are much less ghosting effect in the middle of the picture.

Manual

Automatic

The second mosaic is of the SEI down by the Fifth. Comparing the images created automatically and manually, there is much less ghosting effects in the automatically generated image.

Manual

Automatic

This is taken near the SEI institute too. The automatically generated image is better than the manually created image. There are some minor artifacts in the final image. This is possibly due to the highly detailed leaves, which have high feature strength but would actually look rather similar to one another, which would throw off the homography.

Manual

Automatic

This is a new image, taken near the Cathedral of Learning. Each image is made up of three source images. The aim is to merge these two composites to form a single large composite, however, for some reason, it refused to stitch. My hypothesis is that again, the good points are probably on the church, however, most of the picture is taken up by the trees and the sky, which would throw off the homography.

I implemented multiscale corner and feature detection. I accomplished this by first generating a gaussian pyramid for both images (using a subsampling rate of 2 and gaussian window of 3x3 and sqrt(2) sigma. Then I performed Harris corner detection and ANMS on all levels of both pyramids and matched features against each other for a given level. I then perform RANSAC on all the points collected from all levels. Naturally now that we have more points, we need to run RANSAC more. I used 8000 iterations and epsilon of 0.6.
One improvement was that pictures that did not stitch under the single scale could successfully stitch under multiscale. One example is the mega-composite.