Digital humans are a key aspect of the evolving area of virtual production. After most of our stories are about people, and the most expressive part of any performance is the actor’s face.
In our latest episode of the Visual Disruptors: Exploring the Future of Virtual Production podcast series with EPIC games, we speak to Digital Domain.
In the first episode (click here) of the series, we explored the basics of what virtual production is, the impact it’s already having on film, and what the future could hold for emerging technology and techniques.
In episode two, Digital Humans: Actors and Avatars, fxguide’s Mike Seymour (who has himself been digitally cloned in the name of research for the MEET MIKE project) interviews Digital Domain’s Scott Meadows, Head of Visualization, and Doug Roble, Senior Director of Software R&D, about the studio’s trailblazing work in the area of virtual production and digital humans.
Listen to the podcast here (or Subscribe at Apple iTunes or where ever you get your podcasts)
The podcast starts with Meadows explaining how the original toolset he assembled for previs on Tron is constantly evolving, as the capabilities of both graphics hardware and real-time engines advance to states that were unthinkable a few years ago. Having honed his tools and techniques on Beauty and the Beast, Black Panther, and Ready Player One, Meadows is now approaching a point where—instead of creating rough 3D models for previs and then throwing them away—assets can move through the pipeline from previs, to postvis, and right through to full VFX post-production.
Diving deep on digital humans
The director’s ability to see CG elements on set while shooting the live-action components is fundamentally liberating. But what if they could see the actual high fidelity CG faces reacting in real time to actors’ input, so they could really judge the performance of the character, and not just the actor? That’s what Roble and the Digital Domain team set out to explore with their ‘Digital Doug’ project.
The team undertook to create a real-time, highest-resolution, markerless, minimally rigged, single-camera facial animation system (no small challenge!), using Doug himself as the guinea pig. To achieve this, they had to combine several existing techniques with some proprietary Motion Capture tech, before turning to machine learning to complete the picture.
First, ultra-high-resolution scans were taken in several poses at the USC ICT’s Light Stage. These scans deliver incredible photorealism as photogrammetry based on still images, but have no temporal coherence—there’s no way to move from one pose to another without the textures swimming.
Next, Roble turned to Dimensional Imaging (DI4D) to get scanned data of himself that was temporally coherent, albeit at a lower resolution.
The third step saw Roble taking to Digital Domain’s traditional motion capture stage with his face adorned with 160 markers, to capture full-motion animation for a range of emotions and facial positions. But with only 160 vertices, the meshes this produced did not deliver sufficient resolution for photorealism. Not to be deterred, The team was then able to use machine learning within Digital Domain’s own Masquerade software (used on Avengers: Infinity War)—to take information and up-res the motion capture data to 80,000 vertices. These high-resolution meshes could then be used to provide a much wider data set for the machine learning. Digital Domain was able to use the additional motion capture data to create 20,000 pairs of very high-resolution images and meshes for a wide range of poses.
The final step was to feed this data into their Machine lLearning /Deep Learning programs, and use it to train Digital Doug.
At the end of this training period, the system was able to compute the right combination of high-resolution textures and mesh for any live-action input at speeds as high as 70 fps. While the training takes a long time, the Machine Learning system runs incredibly fast. Whatever expression or look Doug makes, digital Doug (or another re-targeted CG character) can now do too—in real time. This is all realised and rendered in Unreal Engine UE4.
