fxguide recently spoke to Weta Digital senior visual effects supervisor Joe Letteri about the sensitive task of re-creating Paul Walker’s performance in James Wan’s Furious 7 after the actor passed away mid-production. Now we talk to Weta Digital visual effects supervisor Martin Hill who explains more of the process behind that incredible work. Find out how a combination of methods – existing Paul Walker footage, trawling through mountains of reference, creating a photoreal digital Paul Walker head, having his brothers and another actor act as stand-ins, and relying on re-projection and meticulous compositing – allowed Weta Digital to bring the character to the screen.
fxg: What were the early conversations you had about how Paul Walker would be brought to the screen for scenes he hadn’t actually be filmed in?
Hill: Well, even before we thought about how we would do it, we got together with Mike Wassel, the visual effects supervisor, and the filmmakers, and talked at length about what they wanted to make the film. The whole point was to get the film they wanted and not compromise the film, and therefore compromise Paul’s performance. We went through the range of potential shots and would categorize them. Some of them we knew would be reasonably straightforward in that they’re mid-shot digi doubles with their faces in action sequences. That’s stuff we’ve done before, and we would categorize as a stage 1 type shot in a 1-7 scale.
Then we divided them all up into levels of difficulty going up to 6 which was full frame dialogue in-camera. Then there were shots which were 7s which we basically called impossible, which were dialogue in full sunlight conditions as long shots while interacting with other actors. Then of course by the end of the film, all those shots that were categorized as 7s we actually did! And we were wishing we had some more of them. Hindsight is a great thing, but we thought that as we were working on the character and getting into more difficult shots, we thought it was really great fun and had wished there was more of it to do.
fxg: From those categories, how did you then decide on techniques?
Hill: There were essentially three techniques. One was using existing footage – we’d take an existing plate and add Paul Walker in, we’d track the shot, re-shoot it with motion control cameras. There was like a pre-edit process that found exactly the pieces we wanted. And quite often it didn’t completely work so there was compositing and re-timing to make sure they did work for the shots we wanted. That counted for less than 10 per cent of the shots. And it wasn’t actually necessarily more efficient to work in that way – there was a lot more time on set in re-creating those shots via motion control, but it was a general mantra that if we can get Paul in the film, let’s put him in there. It’s about respecting his performances and you can’t get more of his performance than what he’s actually done.
fxg: Where was that first method used?
Hill: There’s a section of shots where Paul is talking to Mia (Jordana Brewster) by his computer. We used John Brotherton as a stand-in there for Jordana to act off. We’d use John’s body and Paul’s neck up and head. That was a very emotionally intense sequence for Jordana because she had previously acted that scene with Paul, so it was quite an emotional day on set.
fxg: What was the second technique for re-creating Paul’s performance?
Hill: This was more of a two-and-a-half-D CG re-lit version. This we used for the very end sequence when he’s driving in the car. James Wan was adamant that the last shot we see of Paul should be of Paul. We didn’t really have anything for it, for the shot that he wanted. But we did have three shots from the fifth film that we could use as existing footage. Now, that was obviously shot on film, it was grainy and not as close-up as we needed. It was all lit with metal-halide mercury vapour-type lamps. It was very green, there wasn’t much color to it. But James wanted it in full LA sun with lots of bounce. So there was a lot of re-lighting. Essentially we matchmoved Paul and then using our digital Paul asset we completely reconstructed the shot digitally. Then rather than putting the digital model in, we used it to re-light a lot of the plate that was there and bring up the resolution.
What you ended up with was basically 60 to 70 per cent CG but then we would keep portions around the eyes from the plate to keep Paul in there. It was heavily re-graded based off a CG re-performance. Because you have the matchmove and it’s perfect – and it had to be – you can project into the 3D space and move around a fair degree. If you go too far anywhere you just need to reconstruct.
fxg: So the third technique I’m guessing is the completely CG Paul Walker head.
Hill: Yes. It’s a particularly challenging CG character to build. Normally when you’re doing something like this, you’re either creating a character no one’s ever seen before, or you have access to information like scans and texture reference from the actual actor to make the digi double. Or if you’re inventing the character you don’t need to nail the representation of the actor. Quite often our digi doubles are because of an augmentation or a digital makeup step in the process. We weren’t doing any of that. So we didn’t have a whole lot of reference in the traditional sense to go off. One of the first things we did was contact the family – they were very amenable and helpful in the process. Caleb and Cody let us scan them and run through the USC ICT Light Stage process. That was really the next best thing in terms of their skin quality, their wrinkles, their pores, their skin tone. That was really useful to get something that as close as we could get to Paul.
fxg: How did you then use that to start building a digital model?
Hill: For the model build we took the reference we had, including some texture stills from the sixth film – these were taken to build a stunt digi double asset and we used those to get to the ground truth of Paul’s face in terms of its structure. Then we went through a process of augmenting the Caleb and Cody textures and building up the texture layers for our skin shading models which we re-wrote and improved for the show to match Paul as he was in Fast 7, using some of the footage that had been shot prior to the accident.
Then we went through a two and a half month process of building up a reference library. This was a huge task in itself. We took reference from all of Paul’s films from the last five years, and whatever else we could get access to – media and publicity photos, footage that didn’t make it into the films from the studio. Then we would break it down into camera angles, breaking it down into lighting directions, and we had this very quickly navigable library. It was a huge time saver. From that we could essentially validate our work by matchmoving these various shots and build up a faux FACS session where we could go through all of the poses and make sure all of our facial poses were on character. That was really important to us, to not make anything up.
One of the really interesting things we found was that we had to go and discard anything from a media shoot or that was from a different film. The reason being was that we had to keep Paul as Brian as much as possible. Paul has played this character in seven films now and he plans it a very distinctive way, so it was very important for us to keep Paul’s interpretation of the character Brian and not have our artists come in and improvise something that perfectly looked like Paul but just didn’t have the character as Paul would play him. So we learned to make some of our reference texture-only reference rather than our character puppet reference.
fxg: You mentioned re-writing the skin shaders – can you talk about what was involved there?
Hill: We improved the entire shading pipeline for skin, to get the skin quality right. Everything was rendered in Manuka, our physically-based path tracer. It has enough hooks in it to be able to create very technically accurate energy conserving shaders, which is essential for this kind of work. We rewrote the hair shaders for the hair, eyebrows and eyelashes. We added a lot of textural variation based off poses, so if Paul is screwing up his brow his procerus is firing. There’s a lot of vascular constriction where the blood rushes into areas and out of areas and skin color changes as he moves. That was a very important detail that really brings life to the face. We’d also add in all the delays for the blood rushing back in.
And we made a load of shader variance for various levels of exertion that could be dialed on and off on a shot. Paul’s performance is quite physical in some of the car sequences. So that kind of redness you get from running for five minutes – that could be brought on and off. Also, tiredness. The shoots were very long days on set. All the other actors would be, shall we say, varying across the day and we wanted our character to be able to do that as well. It’s a story point, too. For the Petty death sequence, the sun’s coming up and they’d just had a big fight the night before and hadn’t slept all night. We needed a way of having this extra dimension of tiredness to the character, which affected both the model and the textures and shading, particularly around the eyes. Our shading lead was Olivier Lesaint. Our digi doubles were already very high standard and I think he managed to raise the bar on the skin rendering and the hair rendering enormously.
fxg: That tiredness aspect is interesting – did you ever feel pressure to make your digital Paul Walker always look amazing in still frames? Sometimes an actor needs to grimace or is making a funny face in still frames, for instance, but I wonder if there was a tendency to change that.
Hill: That’s an interesting point. The thing is, for any actor, not every frame is a poster. There’s always going to be a frame where one eye’s half-blinking and the teeth are getting a really weird specular highlight. The next frame, it could be put on the poster. There is a level of polishing that’s sometimes tempting to do by stepping through every frame, but it’s really important to stop things looking synthetic and too perfect. Something can look bad but it can still look real. It has to look real and like the character.
fxg: Weta Digital has done such great work with eyes in recent films, such as the Apes films. What were some of the challenges with Paul Walker’s eyes?
Hill: Again, we didn’t have access to very high photographic reference of Paul’s iris or anything. For his scleras we used the texture reference from his brother re-mapped to what we could see in close-ups from other film and media photography. So eyes were the one time where EPK from other films was really good because they were really high-res stills that we could use as reference. Getting the fiber density of the coronet and the iris right was really important.
fxg: You also mentioned doing some scans at USC ICT – did you adopt any of their research in the area of microstructure deformation and stretching of skin?
Hill: I did talk to Koki Nagano who had developed that and he’s doing terrific work with that. Unfortunately that research came out after we had done our scans. But we do have something here at Weta Digital that works in a very similar way. Based off of pose-base and tension and compression, we stretch the wrinkles and have them non-linearly disappear or stretch in certain directions. So if we’re pulling in one direction and it becomes an extreme pull, the orthogonal wrinkles will disappear and the longitudinal wrinkles will become enhanced. Even at a distance, through LEAN mapping it becomes a highlight BRDF anisotropic change to the skin. It’s not done in exactly the same way as the ICT method, which I think is terrific, but it covers the same problem.
fxg: In terms of the brothers and any stand-ins on set, how was their performance captured?
Hill: Whenever they were performing they had dots on their faces. We had a vaccuform mask that every morning when they came on set we would put the mask on and dot them up. It became an interesting process. Cody and Caleb are very naturally talented actors which was a huge help. When Caleb and Cody came on set, it was about us getting a nice physical body performance from them. Paul’s a very athletic guy and is naturally relaxed the way he moves. The first couple of days on set, sometimes our guys were a little bit stuff. We had an acting coach, Jay Scully, who was very good at getting them into character. He’d worked with Paul before too.
Every time we shot those guys, the filmmakers were very accommodating with our witness cameras. It wasn’t just the standard visual effects setup, we had focus pullers and proper camera operators on each cam. What this gave us was the ability for James to direct the shots and for Steve Windon the DOP to light the shots and then we had something to edit with, and of course for the other performers to interact with. They could shoot the film normally in that sense.
And then for visual effects, it gave us timings. With the best will in the world, the performance that the guys gave we couldn’t one for one match that and put it onto our Paul digital puppet. If we transferred it procedurally mapping, whether it’s a pose based thing – well, what’s the equivalent of Caleb furrowing his brow to Paul furrowing his brow – they’re actually quite different because they’re activating different muscles. But if we map them that way how well does it look into character if we did straight on muscle mapping as
well. What we found was we would get a really convincing looking Paul character but it didn’t look like he was playing Brian. It looked like it was a rehearsal or something.
If you were taking Caleb or Cody’s performance and mapping that onto someone that looked essentially very much like them – a very similar facial structure – we could tell whether the shot was from Caleb or Cody. This meant we were imbuing too much of their personality into our Paul Walker puppet.
So we would just always go back to our reference library – back to the source. We would take our cues from Caleb and Cody in terms of timing and dialogue delivery and would find similar shots or portmanteau’ing shots from various places of Paul performing, matchmoving those and then using those as a feeder into the final performance of what went into the film. Some shots we just matchmoved Paul from a completely different shot or film and it just worked terrifically in a different camera angle and lighting and you had a performance that was directly from Paul. That was really pleasing because you’re getting all of Paul’s essence into the final performance. Obviously that didn’t work all of the time, but by always going through the process of finding the best reference, often we would matchmove to a high level of scrutiny three or four previous shots and just take pieces of them as various beats in the shot to inform our facial animators to build that final performance as faithful as Paul would have been to the character of Brian.
fxg: To me that’s what seemed to bring the character out of the Uncanny Valley, because you were always going back to the original performance, where you could.
Hill: It’s interesting you mention the Uncanny Valley because when we were starting the project and building the model, I just remember the first milestone, coming in and our digi double looked like a completely convincing human. That was great and that was something everyone was happy with and you’re looking at close-up turntables of the character performing, but it wasn’t Paul. It looked like Paul, but it wasn’t Paul. You turned it on and as soon as you saw it you instantly knew it wasn’t him. So then there was more development and it got to the point where our pose library for Paul was more than twice it was for any other character we’ve done here. By refining and the artists observing and matchmoving and getting tighter and tighter, suddenly it got to the point where I’m absolutely convinced now that that is Paul. We got over this sort of second Uncanny Valley of ‘is it a human being?’ to ‘is it that actor?’.
That was a really interesting process for me, because perceptually our brains are really hard wired to recognize people instantly. It’s something we can do out of the corner of our eye in a fraction of a second. It’s not an analytic process at all. So having that recognizability for someone like Paul Walker was of course essential.
But then we reached the really strange third barrier, which we had sort of anticipated. At times it felt like he was in a different film and he wasn’t playing the character of Brian. That was where the way he smiles and squints and he has those really powerful looks as he does as the character of Brian – we needed to make sure we were always emphasizing those character traits. At that point we had to discard those elements that were, although they were completely legitimate on the level of referencing Paul, but things like his smile on the red carpet were nothing like his smile he would use as the character of Brian. Even footage taken during filming when the camera is rolling and then cut is yelled, his face changes – it’s subtle and almost imperceptible – but you can tell when he’s in character and when he’s not. It tells you, oh yes we need to tighten our digital representation into that realm of just the character.
I have to say that our facial modeler lead Alessandro Bonora and our animation lead Dan Barrett were fantastic. Every day the three of us would get together and would analyze hundreds of shots and bits of reference – those guys did a fantastic job of representing Paul.
fxg: This might sound like a funny question, but how did you work with the overall supervisors on the film for Paul Walker shots – did you feel like you could only show them something that was close to final from your point of view. I wonder if you showed them a work in progress you would be constantly saying ‘it’s not done yet’. How did the review process work for something that has to be so convincing?
Hill: The overall VFX supervisors, Mike Wassel and Kelvin McIlwain, were terrifically knowledgeable guys. They understand the process. They understand the complexities of what we were embarking upon, and that we were trying to do something that had never happened before. They were well aware that things we were going to show them weren’t always going to be perfect, particularly at the beginning of the process. We were just very open and collaborative with them, taking their input. Mike came down to New Zealand to go through the whole process. We were very frank about where we were at any stage. Back when we had what I thought was the convincing human – early on – that didn’t quite look like Paul, we said we knew we had a long way to go here in making this look like Paul, but here’s a milestone step. And they were really happy to see that. They were also really involved in the process – they were going through libraries with their editorial of Paul Walker references. They would call up and say here’s a great reference for this shot here, what do you think? It would all be part of a collaborative discussion.
fxg: What do you think were the most challenging or successful Paul Walker shots from your point of view?
Hill: Well, it’s interesting because it’s not a fantasy film, it’s an action film. You’re in very realistic lighting, there’s nowhere to hide in dark shadows or fantastical lights. So one of the things I was most worried about were the prosaic shots of full daylight, Paul standing and talking. He’s not moving around, there’s no motion blur to hide, the camera’s going to be fully focused on him. That’s something we encounter ourselves, hundreds of times a day. You know when it’s wrong. My fear was that the simplest shots were the hardest. The complexity of getting the phoneme shapes in the mouth and all the muscles around the mouth working correctly, the vascular constriction of the lips and the wrinkle changes and flexes as the mouth changes form are all very difficult. And it’s full screen on the cinema.
On the LA Overlook – where they’re all in a line talking together – I’m very proud of how that one came together. Also in the Petty death sequence there’s a few shots of him driving where there are a few emotional beats. It’s a subtle performance where Petty is dying and gets let out of the car – they’ve just been fighting all night in this industrial complex. Brian doesn’t know Petty very well, he doesn’t know whether to trust him but at the same time the guy is being left in the desert to die – getting that level of performance was what James the director was asking for. James wasn’t saying just ‘Oh he’s not happy here’, he was looking for something very subtle.
Another shot I’m really proud of is in the apex of Brian’s story arc, and it’s right at the end of the tower jump sequence. They’ve just jumped from tower 2 to tower 3 and tower 3 to tower 4 in the Lykan. Brian’s story arc is that he’s missing the action and adventure being at home being a family man. He leaps out of the car holding the MacGuffin, the Gods Eye, and Vin’s character asks, ‘Are you still missing dodging bullets?’ or something like that. And Brian’s face is just meant to drop for this moment of realization that his family life is more important. That’s where he decides this is the last time he’s going to do this. James was very adamant that this was a critical character moment in the film where we needed to nail the performance. We did an awful lot of work in just getting the facial performance correct, not in terms of photorealism, but it was almost a fourth barrier after looking like human, like Paul, like Brian – it was then Brian on his best take that the director wants. Not even Paul could get that the first time. It was lovely to be at the level where we could just sit and work with the director and get the performance that he really wanted for the film, confident that we were making it believable.
fxg: It feels like you nailed the facial animation and rendering, but you also must have had lighting and compositing challenges – can you talk a little about that side of the work?
Hill: Our compositing lead Florian Schroeder did an amazing job. You think about all of those head shots, not only the compositing work from existing footage, but any one of those shots if there was any form of seam or motion in the neck, where we transitioned into the body (and sometimes we were augmented the body as well in a 2D or two and a half D way) – making sure that every frame there was seamless was a terrific effort.
