Environment Matting Extensions

Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture

Yung-Yu Chuang¹ Douglas Zongker¹ Joel Hindorff¹ Brian Curless¹ David Salesin^1,2 Richard Szeliski²

¹University of Washington ²Microsoft Research

Abstract
Environment matting is a generalization of traditional bluescreen matting. By photographing an object in front of a sequence of structured light backdrops, a set of approximate light-transport paths through the object can be computed. The original environment matting research chose a middle ground using a moderate number of photographs to produce results that were reasonably accurate for many objects. In this work, we extend the technique in two opposite directions: recovering a more accurate model at the expense of using additional structured light backdrops, and obtaining a simplified matte using just a single backdrop. The first extension allows for the capture of complex and subtle interactions of light with objects, while the second allows for video capture of colorless objects in motion.

Citation (bibTex)
Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H. Salesin, and Richard Szeliski. Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture. In Proceedings of ACM SIGGRAPH 2000, pages 121-130, July 2000

Paper

SIGGRAPH 2000 paper (1.5MB PDF)
Technical Report UW-CSE-2000-05-01
(This paper is identical to the SIGGRAPH 2000 paper, but also includes a proof in the appendix explaining why an object never amplifies light from a background.)

Video in SIGGRAPH 2000 video proceedings
720x480 DivX Avi (76.9MB)
360x240 DivX Avi (56.4MB)
(Download DivX codec from www.divx.com)

Results

Towards Higher Accuracy

(a)

(b)

(c)

SIGGRAPH'99

SIGGRAPH'00

Photograph

Comparisons between the composite results of the previously published algorithm, the higher accuracy environment matting technique described here, and reference photographs of the matted objects in front of background images. Lighting in the room contributed a yellowish foreground color F that appears, e.g., around the rim of the pie tin in the bottom row. (a) A faceted crystal ball causes rainbowing due to prismatic dispersion, an effect successfully captured by the higher accuracy technique since shifted Gaussian weighting functions are determined for each color channel. (b) Light both reflects off and refracts through the sides of a glass. This bimodal contribution from the background causes catastrophic failure with the previous unimodal method, but is faithfully captured with the new multi-modal method. (c) The weighting functions due to reflections from a roughly-textured pie tin are smooth and fairly broad. The new technique with Gaussian illumination and weighting functions handles such smooth mappings successfully, while the previous technique based on square-wave illumination patterns and rectangular weighting functions yields blocky artifacts.

(a)

(b)

(c)

(d)

Oriented weighting functions reflected from a pie tin. (a) The previous method yields blocky artifacts for smooth weighting functions. (b) Using the higher accuracy method with unoriented Gaussians (\theta = 0) produces a smoother result. (c) Results improve significantly when we orient the Gaussians and solve for \theta. In this case, \theta is about 25 degrees over most of the bottom surface (facing up) of the pie tin. (d) Reference photograph.

Towards Real-Time Capture

(a)
(b)
(c)
(d)
Sample frames from four environment matte video sequences. Rows (a) and (b) show bubbles being blown in an Ehrlenmeyer flask filled with glycerin, while rows (c) and (d) show a glass being filled with water. Sequences (a) and (c) were captured with no lighting other than the backdrop, so that the foreground color is zero. Sequences (b) and (d) were captured separately, shot with the lights on, and the foreground estimation technique is used to recover the highlights.

Additional results

7.5MB AVI 7.3MB AVI 7.1MB AVI

cyy -a-t- cs.washington.edu

(a)
(b)
(c)
(d)
Sample frames from four environment matte video sequences. Rows (a) and (b) show bubbles being blown in an Ehrlenmeyer flask filled with glycerin, while rows (c) and (d) show a glass being filled with water. Sequences (a) and (c) were captured with no lighting other than the backdrop, so that the foreground color is zero. Sequences (b) and (d) were captured separately, shot with the lights on, and the foreground estimation technique is used to recover the highlights.