Abstract
Existing methods for video completion typically rely on periodic color
transitions, layer extraction, or temporally local motion. However,
periodicity may be imperceptible or absent, layer extraction is
difficult, and temporally local motion cannot handle large holes. This
paper presents a new approach for video completion using motion
field transfer to avoid such problems.
Unlike prior methods, we fill in
missing video parts by sampling spatio-temporal patches of
local motion instead of directly sampling color.
Once the local motion field has been computed within the
missing parts of the video, color can then be propagated
to produce a seamless hole-free video.
We have validated our method on many videos spanning
a variety of scenes.
We can also use the same approach to perform frame
interpolation using motion fields from different videos.
|
Results
Hole-filling
We set the spatio-temporal hole to the input video sequences.
Then using our motion field transfer, the hole is filled with the estimated local motion field.
Finally, video completion is achieved by propagating color.
|
Walking (DivX avi)
Performer (DivX avi)
|
Object removal
This movie shows a useful application of video completion: object removal.
Here, we manually removed the foreground person and automatically filled the hole using our
motion field transfer technique.
Even though there is complex motion in the background, the fill-in was accomplished well.
|
Performance (DivX avi)
|
Frame Interpolation
This movie shows another result of video completion for a breakdance scene.
Unlike previous examples, here we recover entire frames.
The input video contains rather complex motions that are not periodic.
Despite this, our method was able to interpolate frames by treating the new intermediate frames as holes.
|
Breakdance (DivX avi)
|
Frame rate recovery with prior video
Given a low-frame rate input video, we recompute the local motion in the prior video with the same frame rate as the input video.
Motion field transfer is then performed to find out the most similar motion patch from the prior video.
Once the patch is found, the full-resolution motion field is warped to the low-frame rate video to achieve a higher frame rate.
In this way, the frame rate recovery is performed.
The video shows the video chat scenario where prior video is especially effective.
|
Video chat (DivX avi)
|
(The latest DivX encoder is downloadable from DivX website)
Media
Acknowledgement
This work was done while Takaaki Shiratori was visiting Microsoft Research Asia.
|