William Wedler
Computational Photography
15-463 Fall 2007
My final project is based on implementing the texture synthesis presented by Efros, known as Image Quilting. An interesting extension to texture synthesis is texture transfer. I will implement texture transfer and will work on extending it even further by generating animation sequences that show texture transfer as it changes with changing parameters.
Presentation powerpoint file. Animation results are in the animations directory.
The final goal of my project is to generate interesting animations of texture transfer at work. That is, to simulate how texture transfer develops over time. To reach this goal, I broke the project up into three parts.
The texture synthesis process involves extracting blocks from a given sample texture and fitting them together in the final texture result image.
Appropriate blocks can not be selected purely at random. Instead, the sample image is searched for a block that when placed into the texture image, it will match well with the already synthesized region near the block. This is determined by minimizing the SSD error between a block and a region of overlap.
Below is an image showing a block outlined in the sample input, and the area where that block should be placed, with the region of overlap shaded.
Once an appropriate block is selected, it is placed into the image by cutting through the region of overlap along a minimum energy seam. The seam is found using Dynamic Programming, and the algorithm is almost identical to seam carving from project 3.
Why is a block that matches well on the boundary an appropriate block to use? It may not be, but the algorithm makes an assumption that it will be appropriate. This intuitive assumption corresponds to the idea of the Markov Property for a sequence of states governed by a distribution.
In texture synthesis, the state is the location in the synthetic image. And the output at that location is the block that is extracted from the sample input. The Markov Property says that the output at a state only depends on that state. So in texture synthesis, the region of overlap is all that matters in finding an appropriate block.Texture transfer has two inputs: the sample texture and the target image. The texture synthesis algorithm is then modified to account for the target image. This is done by considering how well a block corresponds to the target image. In terms of the Markov model, this means adding more information to each state (location in the synthetic image). In addition to finding blocks that matches well in the region of overlap, blocks must also have high correspondence with the target image.
To vary texture transfer over time, I vary how much the synthesized image should correspond to the target image. This is done by changing the window size for the blocks. A smaller window allows the synthetic image to match the target image better than a larger window. I also blur the target image when the block sizes are large to de-emphasize correspondence.
However, the texture transfer algorithm is has a random component and successive frames are very jumpy. To reduce this, I added more sate information. In addition to forcing the blocks to correspond well to the target image, they also have to correspond well to the previous frame.
Since my initial texture synthesis and transfer results were not very good, I borrowed code from Professor Efros and based my final implementation on his code.
I tested out my implementation with images from the quilting paper because. I first generated an animation without correspondence with previous frames. This is rather jumpy and I was able to improve after adding the previous frame correspondence.
Firs try, with large jumps between frames.
Smoother frame transitions after adding previous frame correspondence.
Animations are uploaded to YouTube and can also be found in the animations directory.
| Texture | Target | Animation |
|---|---|---|
 |
 |
|
 |
|
|
 |
|
|
 |
|
|
 |
 |
|
|
|
|
|
|
The images used are from the image quilting paper.
| Texture | Target | Result |
|---|---|---|
 |
 |
 |
Input
Output
Good results can be difficult to get, partially because they depend on the block size and overlap size used. Below are synthesis results with varying parameters. The sample texture is from the image quilting paper.
The foundation of this project is texture synthesis and I have worked towards getting good synthesis results. My results illustrate the texture synthesis concept, but are not as good as I would like them to be. Below are my texture synthesis results at mid-project.
Input texture
Block size 50
Block size 25
Input texture
Block size 15
Block size 25
Block size 50
William Wedler. 17, Dec, 2007