Final Project: Object Removal
Introduction
It is desirable to remove certain objects from an image.
An example would be a passer byer ruining an otherwise pristine shot of nature.
It would be nice to remove the offending portion of the image and fill it in in such a way that it is impossible to notice that it was ever there.
Most object surfaces tend to exhibit regularity or repeating patterns at some scale or another.
Typical examples include grass, carpets, building walls, etc.
Even a seemingly irregular pattern such as text has a repeating context when viewed letter by letter, line by line.
Under this observation, texture synthesis methods seem to be a natural choice for filling voids left by removed objects.
All that is required is for the context of the void area be defined.
This is done by examining the close area around the object.
The size of the area can be defined by a single variable that the user may specify.
Examples
good, no visible artifacts
reasonable, but smooth gradient not preserved
bad, boundaries between differnt texture regions wack
bad, texture taken from wrong regions
What went wrong with the bad cases?
A simple pixel based method based on MRF has no explicit concept of boundaries.
The texture from one region may bleed into the another where it's not supposed to.
This can depend on the order the pixels are being synthesized.
Since my current implementation works in onion-peel order, texture tends to bleed outside inward.
Boundaries seem to be the key point.
As long as the synthesizer knows which section to sample from when synthesizing a pixel, there should be no problem.
Boundary detection seems the natural solution.
But this is not the whole solution.
We need to define the boundaries inside the void through some method of interpolation.
Furthermore, in some cases the boundaries themselves are not regular. This appears mostly in nature shots
where sharp boundary lines tend not to exist.
Improvement
Let the user discretize texture regions. Although my original plan was to automize this process, the problem
turned out much harder than it seemed and due to time constraints I was not able to pursue
this to an end. On the other hand, by letting the user define the regions, the process seems
to work for a wider range of images than would be possible by an automatic method.
Here are some improved results achievable by user defined boundaries. (note: 2 or 3 iterations were applied to enhance results)
Largest result. I hope Prof. Kuffner doesn't get mad for using his pic.
Conclusion
Each of the improved images took about an hour in total to generate.
This includes the time to draw out the boundaries by hand.
Even for a professional restorer, I think the same task would likely have taken a day by hand.
Texture synthesis methods definitely seem to be a valid method of efficiently removing objects from an image.