The Irishman is an epic crime drama directed and produced by Martin Scorsese and written by Steven Zaillian. It is based on the 2004 book I Heard You Paint Houses by Charles Brandt.
The film starts primarily in the 1950s when truck driver Frank Sheeran (Robert De Niro) gets involved with Russell Bufalino (Joe Pesci) and his Pennsylvania crime family. As Sheeran climbs the ranks to become a top hitman, he goes to work for Jimmy Hoffa (Al Pacino), the powerful Teamster leader tied to organized crime.
The film is now on Netflix but after a brief cinema run, it is also an early contender for the VFX Oscar and it was just voted National Board of Review: Best Movie of the Year.
As the film’s stars are all much older than most of their on-screen characters, ILM was brought it to de-age the actors for the various time periods in the film. The film covers decades in the lives of these characters. Robert De Niro and Joe Pesci are both 76 years old, and Al Pacino is 79, yet in the film, De Niro alone appears as Frank Sheeran aged: 24, 36, 41, 42, 47, 55, and then eventually his actual age 76. The 209-minute film contains 1750 visual effects shots, and as the film was for Netflix is was mastered in 4K, including all the VFX.
One of the most important aspects to grasp for ILM Visual Effects Supervisor Pablo Helman was that the director wanted the actors aged to their characters’ younger selves, not the actor’s younger selves. For example, the de-aged Joe Pesci was de-aged to a thinner version of Russell Bufalino than the actual actor Joe Pesci was at that similar age. In this way, Helman was focused on the arc of the characters in the film, not necessarily the perfect likeness of the same actor but younger. The difference is subtle but very important to Scorsese. This meant that the ILM team could not just match the actors to their features in film from earlier in their careers.
The second issue that was important to both Scorsese and Helman was to not restrict the actors with any form of headgear or intrusive technology. Thus remarkably the actors performed on set as normal and ILM solved the tracking and face replacement thanks to some major innovations in technology.
New Flux Camera Rig
The first major innovation was a new camera rig. To build a computer model of the actors the team filmed with the primary camera on a rig with two additional cameras offset on either side. Helman knew that if the team was not going to have tracking markers or head rigs, then they had to get the most information they could from the actors’ texture and lighting. Work started in pre-production to solve how to de-age the actors without interfering with either the director process or actors’ performances. “We came up with this three-camera rig, but if we were going to be basing our solution on lighting and texture, then changes between the level of the key light and shadow side were going to be very important” explains Helman. This created a problem of how to light the actors without affecting how the DOP would normally light the scene. The solution was to make the extra cameras be infrared.
To help with obtaining consistent lighting each side camera had an infrared light ring fitted around the lens. This new approach was born during pre-production which started in 2015, “which is kind of funny because at that time in 2015, we were all doing basically the same thing in terms of faces – which was markers, helmets, and witness cameras” recalls Helman. Robert De Niro was not only the lead actor in the film but also a Producer. He was at the early ILM meetings and Helman recalls De Niro saying, “you know, I’m not that kind of an actor, I want you to develop an approach that doesn’t have any markers on my face. I don’t want to wear a helmet or anything like that. And I want to be onset with the correct lighting, working with other actors”. The director agreed, telling ILM in the friendliest terms “why don’t you think about it? And when you come up with a system – give us a call back”, Helman laughingly recalls.
The ILM team went to work and invented the core of the new three-camera new approach. To test it, they had De Niro re-enacted the pink Cadillac scene from Goodfellas but with De Niro aged now 74. ILM then de-aged De Niro to match himself from 1990 and compared this digital performance to the original Goodfellas‘ De Niro performance. At this early stage of testing, the cameras were all just RGB. This initial 10-second test greatly impressed the filmmakers and so the ILM team set about working out how to turn the bulky test rig into a financially viable and production-friendly model.
The script grew from an original 130 pages to 160 pages and filming would end up taking a lengthy 108 days. The new approach worked well and the director ended up shooting a lot of improvisation. For example, the ‘fish in the car’ scene and the scene of Russell Bufalino (Pesci) coming home covered in blood, were all not in the script. Even the argument and brawl in Miami where Jimmy Hoffa (Al Pacino) argues about being 15 minutes late, was not in the script. This meant that the ILM team could not use body doubles or put heads onto other stand-in actors.”If they are going to improvise I couldn’t say ‘hey Marty, you need to select your take now and have the body double do exactly the same thing’, it just would not work, ” Helman explains. This means that the ILM team had to digitally alter the actors’ actual bodies in hundreds of shots to make, not only their faces, but their bodies look younger. Helman also found that the actors’ 70-year-old hands required a lot of work. Hands age very distinctly, so while audience members may naturally be aware that there are a lot of digital faces manipulated in the film, few will stop and think how many digital hands have also been added and altered.
The camera rig incorporating infrared did provide evenly lit face footage, but as the infrared cameras have a different depth of field, this also required doubling the camera department as all the focus marks were different. Infrared also responds to incandescent lighting, as it is hot, and the production was unable to always use LED lights, so the infrared cameras needed special filters. “The other thing the infrared cameras influenced was the production design. Because we’re dealing with cars are from the fifties and sixties and even the seventies, we had to remove the windshields to film”, explains Helman. Windshields in the past contained lead. Infrared would not penetrate the cars’ windshields if filmed from outside the cars. If the windshields were not removed the infrared footage would show just a black windshield and no faces. The issue also happened if a character was smoking, the heat contained in the smoke would cover the actors’ faces far more on the infrared than on the main camera, requiring much of the smoking ‘smoke’ to be VFX. Also, the rig was big and bulky. Helman explained that the multi-camera rig was “a 64lbs ‘brick’ on a 30-inch wide rig,” adding, “- which is the widest rig that can still go through a standard door frame!”
To move the cameras, most of the film was shot on Technocranes, Jib arms, and dollies. The only limitation was Steadicam. At 64lbs, the rig was too heavy for Steadicam. “In the film, there were only 3 Steadicam shots. These had to be done with just two cameras on the Steadicam and another full rig put somewhere so we could triangulate between the Steadicam rig’s two cameras and the other three cameras” Helman explains. This was physically possible as the production had a complete second three-camera rig. They needed two full rigs as normally Scorsese shoots dialogue with two cross camera rigs, one on each side of the conversation, so the actors could be captured authentically together and not in separate single shot setups.
Flux software
The Flux or Face Lux (lighting) system software solved the faces, somewhat as a ‘black box’. The base geometry and textures of the face were informed from both Light Stage and Medusa scanning sessions with the actors, but the process did not involve the traditional driving of an animation rig. Stephane Grabli supervised the Face Capturing. After building the younger versions of the actors’ digital doubles as re-targeted versions of their current real age, the pipeline diverged from a normal ILM face pipeline. “We did re-targeted the faces,” explained Helman “but having said that, – besides the modeling and the painting as with normal 3D, – there was no ‘animation’ on the show, there was no keyframe animation”. The key to Flux software working its understanding of the lighting and the physical set. All the film’s sets were LIDARed and captured with HDRIs. While the HDRIs sample the onset lights & Illumination, they do not include the Infrared component, which are key to the facial illumination and thus need to be accounted for separately.
For a typical de-aging sequence of two actors in conversation (each with their own multi-camera rig focused on them), the process was as follows:
- ILM combined the HDRI and LIDAR and solved for the location and movement of the cameras,
- this provided the location, intensity, and color temperature of all the lighting on set.
- This is combined with the infrared illumination solution.
- This is then processed and the Flux output is compared against what was shot for alignment.
- The digital doubles of the actors are then added and they are deformed on a per-frame basis to match the Flux solution of the actors’ actual faces.
“It’s a completely different way to look at (facial) geometry and how it deforms. Obviously Flux checks with the shape library, but there is no ‘animation’, no process to adjust it. I mean, we could adjust it with shapes and sometimes we had to, but not in an animated kind of way”, explains Helman. This is a completely different pipeline to the previous ILM Academy Sci-tech winning approach. The new Flux pipeline is designed to produce results without an explicit animation stage. If one watches the final credit roll, Helman points out, “there is not one facial animator listed in the credits – since Marty did not want to change the actors’ performances…we did not change one blink. The only thing he wanted was to make them younger”. Helman loves animators and has worked with facial teams for decades on many ILM feature films, but “on this project, there was no facial animation. There was no way to do it. There was no development of an animation facial rig”. The Flux 3D output was rendered in RenderMan and composited in Nuke.
The new process dramatically changed dailies, which ran each day from 3.5 hours each day, from 8:30 am until after Midday. If a problem was found in the Flux output it was not a matter of adjusting Flux to find a different solution, “perhaps a compositing solution or a layout thing might be needed to be done”, Helman explains, adding that “the reviews were never done viewing intermediate work. Marty would never be shown an old fashion ‘plastic render’… the dailies reviews were: ‘here is the render, almost everything is done to 95%’. I’d show a shot to Marty and then show it side by side with the original take to see how he reacted to it,” Helman recalls. “I don’t think I ever heard Marty say anything like, ‘the likeness was not there’ and he would only say something like ‘I think we more texture in the skin’, etc”.
The AI Face Finder
ILM developed a whole new artificial intelligence Face Finder program between 2015 and 2017. The team at ILM assembled an “an incredible library,” comments Helman. “We collected clips and images of the three main actors at all their targeted ages. We then created a library, which we divided up so you could look for say the nearest eyes to what you needed”. For example, the artists could search for De Niro’s eyes: exterior or interior, at a particular age. “We broke it down into all these different components of performance and lighting that we were going to need”. Once a shot was rendered in 3D, based on the captured performance at the contemporary age of the character, the “AI program would go through our library and come up with alternatives with similar angles, ages and alike, to what we were rendered. This provided a huge sanity check” says Helman.
The AI frames were not offered up as replacements but visual reference from a vast library of previous De Niro performances over decades of films. Of course, Director Marty Scorsese having known and worked with Bob De Niro for 50 years, was an incredible judge of fidelity and authenticity. The director also had fun with referencing his own work. For example, the set where De Niro’s character confesses he is about to blow up the laundry is actually a faithful reproduction of a set from Goodfellas. The director is known for his incredible depth of film knowledge and detailed eye.
To allow ILM to get up to speed, the director turned over about 50 shots, while principal photography was still filming. This allowed the 30 odd person team at ILM who were developing the new pipeline to start tackling difficult final shots and not wait until the shoot was completed. The ILM team was extensive, with three additional Associate VFX supervisors required to help Helman, Leandro Estebecorena, Nelson Sepulveda, and Ivan Busquets. The complex issue of texturing based on Flux was headed by Digital Texture Supervisor, Jean Bolte and worked alongside Look Development supervisor Kevin Reuter. Additionally, Damian Steel was the Digital Texture Lead, John Levin, Layout Supervisor and Paul Giacoppo, was the Model Supervisor.
Not all the de-aging was fully 3D, there was also a lot of compositing work and 2D solutions. For example, in the earliest of the flashbacks to the War in Europe, all of De Niro’s body was adjusted with 2D composting to look thinner and younger. Helman preferred this solution to using a body double. The goal was to always use the lead actors’ bodies wherever possible and modify them without losing the original performance.
Helman closes by reflecting that “the major achievement here is the capture method. I think the way we captured the performances, without markers and without having to ask the actors to go somewhere else to record their performances, is our most significant achievement”. For Helman himself, he adds when working so closely with some of the greatest actors of our era, he realized “the incredible subtlety of their performances and how they work with each other”. For ILM this new pipeline aims to capture and maintain that detailed performance through an incredibly complex technical process and still deliver emotionally engaging, honest and riveting performances. “I think that was the thing for me, being able to get these performances with such fidelity, – frankly I was surprised.”