Z-key is a new image keying method which merges synthetic and real image in real time.
Image keying is a method which merges two images. The two input images are connected to a switch which is turned according to some information (key) attached to image pixel. Chroma-key is the standard video keying method used in TV industry to select part of real images - e.g. a weather man in front of a blue screen - by using chromaticity as the means of differentiation. However, chroma-key implicitly assumes that the real objects are always in front of the synthesized objects or scene.
In contrast, the z-key uses pixel-by-pixel depth information in the form of a depth map as a switch, thus it is able to separate based on the detailed shape and position of surfaces. Therefore, we can place real and virtual objects into a synthesized image according to their local and global positional relationship.
Figure 1 is a schema of the z-key method. The z-key method requires images with both depth information (depth map) as inputs. The z-key switch compares depth information of two images for each pixel, and connects output to the image which is the nearer one to the camera. The result of this is that real and virtual objects can occlude each other correctly. This kind of merging is impossible by the chroma-key method, even if it is accompanied with some other positioning devices such as magnetic or acoustic sensor, since these devices provide only a gross measurement of position.
We have developed a simple demonstration of depth-key video keying with a video-rate stereo machine developed at Carnegie Mellon University. The demonstration can create virtual images like one at the right side of the figure at a rate of 15 frames/sec.
The z-key method will extend the possibilities of synthesizing virtual reality and make applications such as tele-operation, tele-presence, training systems with simulation, and games playing in virtual reality, possible and/or more effective.