Seam Cutting

Aaron Johnson, Computational Photography Fall 2007 Proj2

For this project I implemented the procedure from Seam Carving for Content-Aware Image Resizing, by Shai Avidan and Ariel Shamir. The procedure is demonstrated in amazing youtube video. Basically if you want an image to be a different size, dont cut pieces off of the side, or scale it and make people look stupid, remove only the non-important information. Pictures here are taken either from the class site listed above or from my website. Lets jump right into the images.

Starting off I took a picture from my Summer vacation a few years ago of a plane taking off from Contention Pond, in Hillsbourough, NH.

And here are the results of the algorithm set to remove 100, 200, and 300 vertical lines.

Not too bad but the plane is still getting squeezed. The reason for this is that when I removed a line I added the "energy" from that line to the neighboring lines so that the total energy of the image would stay the same and the algorithm would be penalized for picking neighboring lines over and over again. This forces the algorithm to take some lines from center of the plane. To try and combat this, and make the penalty for similar lines less severe, I only added back half the energy of the removed column:

This helped a little, but not enough. I realized that adding the energy of the removed seam changed the energy image too much and over time it became less and less relavant. So I decided to recalculate the energy every 50 lines (every line would take way too much computation):

This is looking better, but we are starting to get artifacts from too many similar lines (see the edge of the bush). Up untill here I had been using entropy as the energy function, using matlab's entropyfilt function. I tried running the algorithm with the e1 energy function from the paper (basically a gradiant). Here are the results of using e1 on the left, with the entropy images shown again on the right for reference.

That last image is a simple scaling of the origonal image for reference. You can see that the e1 version produces the best result, in terms of how the overall scene looks as well as looking at the plane and the rock.

Looking at these energy functions more carefully, I used a different picture of mine taken on Star Island, Gosport, NH.

The entropy energy of this image is as follows:

And the e1 energy, at two contrast ratios, is:

Now onto the results, e1 on the left, entropy on the right again, at 100 and 200 line reductions:

Neither result is very satisfying especially along the right side of the image, but the entropy result is much more believeable and smooth. Taking this image to 300 line reduction isnt really practical:

Next I considered a picture my Dad took on his trip to Uganda:

This time I tried removing horizontal lines (shrinking the height). Here the two energy functions seem to preform simlarly, as seen in this 100 line, e1 on left entropy on right:

So from now on I will consider only the e1 function. Taking the vertical shrinking further, at 200 and 300 line reductions:

But is this algorithm really doing anything special? Consider the same image cropped and then streached:

The cropped image lost more of the sky, the tree, and the grass, and the streached image looks completely wrong.

Annother thing you can do with this algorithm is streach images by adding low energy lines instead of subtracting them. Here I tried adding 100, 200, and 300 pixels, and it worked so well that I continued to 400,500, and 640, thus doubling the width of the image.

The last image does look somewhat streached, but that is unavoidable since we are trying to double the amount of information in the image. Also note the streaching on the far right side. This often happens at some point as you continue to streach an image, and is a degedation of the energy function as you push the algorithm to its limits. Compare this last image to the same image simply streached by 640 pixels:

Next I tried a picture from the assignment page. Here is the original, 100, 200, and 300 pixel reduction, then 100, 200, and 300 pixel expansion:

I wanted to try and find a picture that would challenge the program, so i dug up this funny picture of some signs stored in a parking garage near my house:

It did surprisingly well with squeezing, at 100, 200, and 300 pixels with e1:

That last image is noticeably squished but the algorithm did what it could to keep the signs, the important part, together and just squeezed them. This implied that the image would do well with expansion, since there were a number of low energy channels between the high energy signs, and the results agree. The only problems are with the vertical supports on the left and the bleeding on the right. Expansion at 100, 200, 300 pixels with e1:

Here is a flower from my house in Silver Spring, MD:

Squeezing by 100, 200, 300 pixels with e1 energy function:

And now expanding 100, 200 pixels with entropy, followed by 200 with e1, then a simple streaching:

Here I took a picture of my cousin from the NH pond, starting with a disapointing result from entropy:

So I moved onto the e1 metric ad did better at 100 pixel reduction, but starting around 200 it also started to fail:

Next is a cool picture I took while flying into the Pittsburgh, PA, airport of CMU's cmpus.

Both energy functions did pretty well at 100 pixel reduction, first entropy then e1:

At 200 pixel reduction though e1 did a lot better. Note in all of these that most buildings do alright but high rise in the upper right always gets messed up:

Now onto a fountain from Uganda again, with 100 and 200 reduction, then 100 and 200 pixel streaching.

Some images dont do as well. Here is one from the front lawn on Star Island:

First I tried squeezing it horizontally, with e1 then entropy:

Neither look too hot, though the foreground did fine both times. Next I tried squeezing vertically:

The right side (entropy) is the best of these 4, though not great. Maybe streaching is the way to go?

Hm, not so great. Entropy running into the bleeding on the right, while the e1 has all sorts of problems with strait lines. Maybe we can streach vertically?

A little better, especially on the e1 side, but this image is really no good.

With a time extension, I decided to add a feature discussed in the above video where the user can specify some regions as good and others as bad. By first removing lines from the bad regions, then adding lines while avoiding the good regions you can delete part of an image without resizing it. The algorithm is basically the same as before but you add or subtract from the energy function. Lets see how that works on the chapel picture from above. First, I made the following mask:

The green part of that image will be considered much higher energy than the rest of it. Running the algorithm at 100, 200, and 300 pixel reductions (with mask version on left and origonal version on right for comparison):

The last image there it ran out of seams that didnt have some mask pixels in it, so it had to start cutting through the mask. It ended up breaking the roof but doesnt look that bad.

Now lets look at that picture of my cousin from before. I added this mask:

I filled in his face so that if the algorithm had to start taking seams from his body it wouldnt touch the face. Here was the best result from before, at 200 pixels:

And now the results with the mask, at 200,300, and 400 pixel reductions: