Lola are arguably the world’s leaders in human face and body manipulation. In The Social Network Lola completed a hundred or so shots, but a key 20 of those involved delicate face replacement to allow one actor to play two roles – the Winklevoss twins. We spoke in-depth with Lola’s VFX supervisor Edson Williams about the technical process. UPDATED 21st Oct
Director David Fincher with a Red MX camera on Set
Surprisingly for anyone who knows the tech behind the Oscar winning work of Benjamin Button, Director David Fincher’s previous film, Lola ended up producing perfect twin face replacement in The Social Network without a single HDR sample from on set. In fact, while it might appear after Benjamin Button face replacement was ‘solved’ , – the approach taken on this film was actually technically quite different.
Before we investigate why, one needs to understand the two processes.
Benjamin Button worked primarily on having an actor’s head replaced with a fully digital version of Brad Pitt’s head. A digital version based on Brad Pitt’s performance but fully rendered in 3D. HDRs were taken on set not in just one position but through a range of positions. Brad Pitt was scanned in an ICT Lightstage allowing for a near infinite range of lighting environments to be recreated and derived from the sampled Lightstage session. The HDRs from the set were then used to produce a frame of Pitt’s real head with the correct HDR lighting.
In parallel a scan was used to make a very accurate digital head and this was animated to match the performance of Brad Pitt delivering the lines on a special set. The digital animated head was then composited over the scene but not before it was compared with the Lightstage sampled image of Brad at the same lighting point. This way the final compositors had
a) a digital animated ‘aged’ Brad Pitt
b) a sampled Brad Pitt but computed or dialed in with the correct lighting – side by side.
Of course the old head had to be removed and the background patched, but the effect was breathtakingly real and rightly earned Digital Domain and the other companies (like Lola) that worked on the visual effects a well deserved Oscar and massive industry wide respect for spanning the so called “uncanny valley”
Click here for the fxguide full story on Benjamin Button
For The Social Network, Lola’s approach was actually quite different. The problem in The Social Network was to have twins on screen played by the same actor. This is a problem that has appeared and been attempted by almost countless vfx artists since visual effects entered film making, Dead Ringers, Multiplicity, The Parent Trap, Adaption , The Man in the Iron Mask, Hot Fuzz etc, and with the exception of Matt Lucas as Tweedledum and Tweedledee in Tim Burton’s Alice in Wonderland, – these rarely involved attempting face replacement, (and even then Alice is a special case due to costume).
In the Social Network Armie Hammer convincingly plays twins Cameron and Tyler Winklevoss faultlessly thanks in large part to his and Josh Pence brilliant performances and the work of Lola in placing actor Arnie Hammer’s face onto the face of Pence. Fincher could not find a pair of twins that satisfied him creatively, and matched the real world Harvard rowers. Unlike Benjamin Button the solution was not to replace Pence’s head with a digital head matched with HDRs and animation to Hammer’s performance. Instead it was to map Hammer’s face onto Pence’s face.
In the words of Lola VFX supervisor Edson Williams, David Fincher worked all this out, ” a lot of this was his idea, the basis of this technique was his, it was his fundamental idea to project on to geometry, my job was just to simplify it, he did everything he could to help us, apart from shoot HDRs ”
Steps for face replacement:
Stage 1: Pre-production The actors Pence and Hammer train to both move and look alike. Gym work and acting training produced two body shapes that matched the close similar physical presence of identical twins.
Stage 2: The two actors simply acted out the scene with Pence wearing dots on his face. “Fincher described the effect as the ‘hockey’ mask”, explained Williams. As the plan was only replace the face, hats and head bands were used to provide a clean line for Lola.
No one from Lola was on set for the twin’s scenes and no HDRs or special measurements were taken. (Although Fincher is extremely CGI savvy and one could argue no Lola VFX presence was needed).
Stage 3: Lola worked based on just the principal photography and any on set texture photos and production stills to recreate the lighting of the set, as it would appear on the face of Pence as he acted. “We just had the final principal photography, but nothing shot for us, we did have a lot of reference shots – just because Fincher is very thorough” points out Williams. This CGI lighting would need to animate, if say Pence walked past a door or window. At this stage a scanned version of Pence (not Hammer) is used to check accuracy. “We just recreated it roughly, we just built a simple mockup, it was just a general look of the environment”, adds Williams.
Pence’s real head is object tracked primarily in PFtrack. In a perfect world the real Pence and his digital double would match perfectly in lighting and orientation. Once this match was achieved that head is discarded – but the lighting design is used in the next stage and the object track was kept.
Stage 4: This lighting information obtained from 3D was used to program a rock concert style DMX lighting board. Actor Armie Hammer was then filmed delivering his performance sitting in a chair filmed with 4 Red cameras. To obtain the correct eye line a back projected 30ft x 12ft screen was placed in his eye line and a projected dot was animated to move – providing the correct eye line position.
This was also derived from the 3D animation done in stage 3.” We would break down the quicktime of the performance frame by frame, and control the lights, …we had a bank of 12 lights around him (Hammer) ” says Williams, but he points out that at this stage there was some flexibility. “You had to be pretty good, but if you got within 80% you were doing pretty good, you actually had a surprising amount of leeway”. That being said it is still an art to match lighting “Michael Watson helped me most with the lights, he’s worked with David Fincher in the past, he is very Red savvy a very good cameraman, he helped me the most with matching the lighting”.
Stage 5. The four Red cameras are combined to produce one full face texture mask, with the Red captured performance with the correct lighting from the DMX panel – but the orientation of Hammer’s head during the performance is unimportant as this moving textured mask is now just a disembodied 3D texture.
Stage 6. A scanned version of Hammer’s head is now carefully animated to match the 4 camera filmed performance. The digital animated head is hand animated to do this, but the animator has the action filmed from 4 angles to reference. Josh Singer was the Lead 3D artist. The digital head needs to animate to match the performance but it does not need to be lit in a CG sense as it will have the combined 4 camera RED texture map projected on it.
For example if the digital Winklevoss twin smiles – the CG face moves to match the smile – but the subtle skin texture creases and lighting all come from the live action projected performance. Interestingly while the cg nose and ears need to align to the projected texture, – the mouth was not internally modelled, the mouth was just a flat surface that the interior mouth texture was projected on to.
Stage 7. The original Pence head PFtrack is now applied to the animated – texture projected digital Winklevoss face. This provides a moving – correctly positioned, correctly lit, very real face mask.
Stage 8. This face mask is now integrated with the original background plate. In some cases this meant mesh warping the body double’s ears, “as Arnie’s ears were smaller, and that’s more of a traditional Lola kind of thing, we do that in traditional flame work”. They did not ever need to use digital hair, “we used the hair of the body double” says Williams.
Then the team remove any elements such as Pence’s chin that tended to be slightly longer than Hammer’s chin. “we did do the last 15% in the comp”, he adds ” we were able to get pretty close with just the 12 lights and the DMX”.
Stage 9. This comp was then beautifully and very accurately color graded to match the on stage multiple RED performance to the exact lighting of where Pence’s body was in shot. While the technique would get the artists most of the way there, subtle differences would still need to be adjusted, shadows or highlights added (some shine or spec highlights were rendered digitally using standard CG approaches).
“It is really interesting how different their faces were tonally, as far as colour correcting, I mean two guys standing next to each other, Arnie’s face was always a bit more magenta.. it looked a bit odd when you AB’ed with the original body double but when you look at Arnie standing right next to him – it matched… it is pretty interesting how people look different in the same light”.
EXCLUSIVE UPDATE:
The original photography
The final comp Click on the images for larger versions
This approach was used for some 20 shots in the film, Lola did a further 90 odd shots, which included traditional split-screen work, with Hammer’s separate performances as each brother stitched together in the same frame. However, due to the film being unable to shoot at Harvard, computer screen replacements and blue screen etc , in total it had almost 1000 visual effects shots in the film.
So why did Fincher pick this route over the Benjamin Button route?
Well firstly he was right, the effect is perfect and there is no arguing with the amazing result.
The good point from Williams point of view is how much this approached allowed Fincher to work with Hammer to get a great performance, “David Fincher was being so specific (for example he’d say) be a bit more pensive on this take… he was getting exactly the performance he wanted, and the actor was only worrying about his performance, he wasn’t worried about where he was walking or landing on his mark, – he did not have any of those (technical) concerns, his only concern was delivering the line David was wanting him to recreate… it was a very controlled way to get an accurate performance”
Director David Fincher and writer Aaron Sorkin here preparing for a cameo
It is William’s opinion that David Fincher does not like HDRs on set. It is no secret that complex HDR sampling on set can take a fair amount of time, and perhaps more importantly break the mood and pace for actors in a very strong performance driven piece. In The Social Network it is also a less dramatic difference than in Benjamin Button. In that film the character had to age and be remarkably different than Brad Pitt in size and head proportions.
In The Social Network, by design, the actors were very similar to start with in both stature and of course importantly in age. In breaking with such a break through proven method and adopting a new innovative approach Fincher simplified on set. The Social Network is believed to have been only a $50million film vs an estimated three times that for Button. Perhaps “necessity was the mother of invention”, but working with professionals like Lola certainly stacked the deck in successes favor.