fxpodcast #334: Paul Debevec on IBLs + Virtual Production LED Stages

Dr. Paul Debevec is a good friend of fxguide, we have done many stories on this work on Lightstages and his huge contribution to Visual Effects. Paul is now Director of Research at Netflix. Previously, he was a Senior Staff Scientist at Google Research and an Adjunct Research Professor at the USC Institute for Creative Technologies. For this discussion, we are also joined for the first half by Weta Digital’s Senior VFX Supervisor Erik Winquist, an expert in VFX and IBLs.

Paul Debevec’s research in photogrammetry, image-based modeling and rendering, high dynamic range photography, image-based lighting, appearance acquisition, facial scanning and photoreal digital actors has won many awards including, an ACM SIGGRAPH’s first Significant New Researcher Award in 2001, a Scientific and Engineering Academy Award in 2010, the SMPTE Progress Medal in 2017, and a second Technical Academy Award in 2019.

In the late 1990’s, Paul developed the technique of High Dynamic Range Image-Based Lighting, where HDR panoramic HDR images are acquired on location and then used to illuminate computer-generated objects, making them appear as if they were truly in the scene.  Paul created two films, “Rendering with Natural Light” and “Fiat Lux” to demonstrate the technique, which is now a standard visual effects process for film and television visual effects.

In 2002, he originated the technique of surrounding actors with real-world image-based LED lighting for virtual production.  These techniques have been used to create key visual effects sequences in The Matrix, Spider-Man 2, Benjamin Button, Avatar, Furious 7, Blade Runner: 2049, The Irishman, Gemini Man, and to create a 3D Portrait of US President Barack Obama.

Erik Winquist joined Weta Digital in 2002 to work as a compositor on The Lord of the Rings: The Two Towers. Erik’s first supervisory roles at Weta Digital were on I, Robot, and Peter Jackson’s remake of King Kong.  His credits as a VFX Supervisor include Avatar, Rise of the Planet of the Apes, and Dawn of the Planet of the Apes. He has been very involved with integrating Weta Digital’s cutting-edge performance capture technology into the production workflow and integrating digital performances into physical sets. He is an expert in Image Based Lighting and Weta’s VFX team has one of the most comprehensive and impressive IBL pipelines ever assembled. Note: Erik was called away for the Director’s screening partway through this podcast.

 

This year, Paul presented at the View Conference in Europe on Getting the Lighting Right in Virtual Production. (His talk is available via the View on-demand website). The talk explored making Image Based Lighting (IBL) HDR captures at both interior and exterior locations and then exploring how these could be used in high-end professional LED Virtual Production Volume.

Paul Debevec did this experiment with the ETC team at USC, ETC is the Entertainment Technology Center at USC and this group has contributed enormously to the understanding of new technologies in the M&E space. As part of this ETC team, Erik Winquist offered to calibrate and assemble the IBLs used, while Paul shot the HDRs, Weta made the highly accurate IBL’s used in the test.  We have a detailed look at Weta’s IBL pipeline coming up at fxguide, including loads of tricks and specifics on equipment to allow you to produce professional IBLs.

The main test took place at Ryot Playa Vista, CA in May of 2021 and on the Line 204 XR Smart Stage in Pacoima CA. The LED volume is a very high-quality stage with ROE Black Pearl 2, 2.84 mm Pixel Pitch (~1350 nit) wall panels, and Planar 5mm pitch (~5000 nit) Ceiling panels.

As Paul and Erik outlined there are 3 main reasons to use LED VP Stages:

1. Lighting: Generating IBL on to the actors so they are lit as if they were in that location
2. Backgrounds to provide in-camera effects without green screen compositing
3. Displaying the virtual world to the actors to help with their process and aid in eyelines and other story elements.

The experiment was to explore how easy or appropriate it was to sample a location with an IBL and then use that for a faithful reproduction back on the LED stage.

Also our podcast, the guys discuss the LED spectral response curves of LED walls. As seen here with the three spikes that Paul refers to.

Links to Paul Debevec’s research papers can be found here  www.debevec.org.