2004 Oscar Winner Joe Letteri

Winner of 2004 Academy Award for Visual Effects Joe Letteri sat down to talk to Fxguide just prior to heading over from NZ to this year’s Oscars, to discuss his work on Lord of the Rings.

FXG: Congrats on your Oscar nod and last year’s Oscar, and also for your technical achievement award for Sub Surface Scattering – As well as Stephen’s for Massive.

JL: Thanks

FXG: Two Towers was your first LOTR film wasn’t it ?

JL:Yes, I was at ILM and I joined for Two Towers.

FXG: Does your work on sub surface scattering on say Gollum – relate in any way to your time on dinosaurs in Jurassic Park ?

JL: It does all go back to the Jurassic Park days in a sense, we were studying skin, of course human skin is different from Dinosaurs skin – and that of lizards or reptiles you might reference for a dinosaurs. It really was a pretty good application of a first skin pass. We were looking at getting a good surface texture quality, we were looking at specular models, colour and diffusion maps, scans of textures and applying fine detail, to get really good folds and creases, and make it look very organic. And this served us well for many years, but on much smaller characters, you end up seeing that there is much more going on – with any character smaller in size really. A lot of people have looked at skin technology, but none of it really changed the shading model, it all ended up giving a layered look, and just didn’t get to the heart of the matter, it wasn’t until I saw the 2001 Siggraph paper on the multiple scattering of light, and I knew that this was the thing that was missing. If you look at master painters it is all about diffusion, there are no lines, it is all very soft, there is a quality, a softness of tissue that they capture. The most obvious example is candle wax, it is the often quoted classic example of light scattering. So we threw out everything else and applied the principles of subsurface scattering, and then applied “makeup’. I went to see the the physical effects guys and there was a head cast lying there and it looked just like the person was dead, it was so well done. And I asked the guys how they did that… how they made the skin seems so real, and they discussed the way they approach making life life casts, and the use of pure silicon to and then the layered paint approach, to capture the diffuse way the light is handled and so we applied these techniques to our 3D. In reality we simplified the approach – the solution was simpler and it reduced the overall complexity, – even if the solution is incredible computationally complex, the approach is actually simpler.

FXG: But you didn’t just use the subsurface approach on Gollum, did you?

JL: Sub surface scattering is actually very valuable and we see it in all creatures, and we used in on all the creatures, the first being of course the digi-doubles ( digital actor doubles) but we used it also on all the big animals – even in the wider shots, of course it is very hard to see in some of the wider shots but it does make a subtle difference. We used it everywhere in the keyframe animaion and in the Massive animation sequences too.

FXG: Goullum moves and changes in motivation and physical levels of dirt from film 2 to the end of film 3, was this just plot driven or was there an underlying technical set of changes to Gollum through the production of ROTK?

JL: No he just stayed at the level of the end of Two Towers, there was not really any need to change him. Oh well, we may have changed a few wrinkles for the close ups in ROTK but nothing much.

FXG: He wasn’t always a human form was he? How much previz was there of Gollum?

JL: No in preproduction, even in film 1 he was less humanoid, there was not much previz for gollum, initially Andy was not cast to play him,- well he was only cast to play the voice, but when Andy was there, you could see “oh yeah Gollum needs to do this”, and while you could look at the other actors for reactions – it is not the same as an actor with peculiar actions – which is what you want of course, you want each character to have action peculiar to that character. Once we had Andy there we could see exactly how Gollum should react.

FXG: How did previz generally move through the pipeline, was there reuse of previz models later in production for example ?

JL: Not much in film 2, of data moving from previz to editorial drop-ins to final, but much more in film 3. By the third one the data would move from prepiz to rough cut editorial, back to technical breakdowns, and on to final plates. We did end up re building a lot for film 3, PJ did say after film 2 that “the third one will be easy – as you have the characters built”.. but it did not work out that way. PJ threw quite a lot at us, and we welcomed it, but he did push hard and we hard to go and re work many of the models.

FXG: Can you talk a bit about the compositing, what layers are provided from 3D to 2D?

JL: Oh we render all characters out in one pass. Each character is complete, – well except for the hair, we would often provide the hair as a separate layer. But that is more for the revisions we would do on the hair and just how long it took to render, sometimes the hair would take as long as Gollum himself. The animation is locked down early on – but we do like to have a very flexible pipeline and keep it very flexible right up to the end .
In terms of matching live action cameras to digital moves, that just comes down to having good compositors really, the compositors handled most of the motion dynamics. They are very good at having to add elements to moving shots and they are very experienced at it. They just study the camera move and handle most of the match moving – in this film – more than before – the compositors handle motion dynamics, so they can add 3D transforms to make 2D elements fit so well.

Courtesy of Apple

click for a larger version
large file over 500k

FXG: In the end lava sequence your quoted as having a team of 12 people on that, was that common -or was that the limit?

JL: Ahh well the lava sequence was very complex, there was a lot going on there, the body was locked down very early, but the lighting is very complex and there is a mix of bits and pieces, – bits of miniatures blending into digital lava etc, and it was a very long shot. There is a different bit falling, – to the bit when he splashes into the lava, etc and the deadline was so tight, we like to have basically a TD and compositor per shot, but there were 3 or 4 compositors on that shot, and it was about the maximum number of people who worked on any gollum shot, just because it was so complex.

FXG: Alias have announced a strategic Alliance with Weta, what are the details of that ?

JL: Maya is a primary 3D package and they did a lot of work with us. In fact, a lot of the work they did with us is being folded into the next release, so it will be interesting to see just what is in the next release. But we try and work very closely with all our suppliers.

FXG: Do you have any other plans to sell software in the way you sell Massive? Such as your muscle system?

JL: I haven’t thought about it, it is a difficult thing to do, I think you either do it or sell it. I mean Pixar do it with Renderman and Digital Domain do it with Nuke but it can be hard separating it out of the pipeline and I think your either going to be in production or not (and thus selling the software).

FXG: Massive was designed to control and animate realistic vast crowds, but could that artifical intelligence be applied down the track to a single character? Could you end up with a version specially for say King Kong in principle ?

JL: It could be expanded into that – it is certainly a very interesting idea, the problem is that if you’ve got something like that then you tend to rely on it too much – you don’t develop the skills you need for when it fails or doesn’t work. You need to build up those skills in the team otherwise when you get to a shot it can’t do – you have no skills to solve it. By the end of film 3 (ROTK) in many ways Gollum was easier to animate as we knew the character so well. So much is done by hand and you need the practice and knowledge to be able to solve those problems. It was much more of an up hill battle at the end of Two Towers, actually it was much easier by the end of ROTK.

FXG: What governs if you use Massive vs Key-frame vs Motion capture?

JL: In Two Towers we had more Massive work closer to camera, where in film 3 (ROTK) most of the action close to camera is actors, Massive is designed for the background, and it is not just Massive, Grunt – which is the Massive renderer is efficient at dealing with large numbers of characters, you could render a single character but it would not be as efficient as Renderman. We can move very seamlessly between the two, it is very easy to move back and forth, we can animate and then take those animations and make them actions for Massive actors. It all works together really well.
It is really about performance and when you want a particular type of performance.

FXG: What about just using particle systems for distant shots?

JL: We tried that, I thought we could just make these really distant characters particles but they just don’t react. They have a different motion, there is some clean up required but our animators are pretty good now. We’ve all seen those animation tests when characters run the wrong way, but that doesn’t happen now because our animators wouldn’t set up a simulation that allowed that. Some times our first pass simulation is pretty good – due to the teams experience they know how to do good simulations, and their first attempt is often very good.

FXG: On ROTK – you managed to do on set motion capture, that is amazing – just in terms of on set organization…

JL: Yeah we got every shot, except one I think, and that was the schedule changed. The day we had to move the rig was no longer there and so we could not do it, but apart from that, it worked really well and even that shot did not matter much. There is a heck of a lot of clean up required but the fight scenes worked so well, – the fight at the foot of Mount Doom worked so well. It was very valuable.

FXG: On film one (FOTR) there was much made of the use of forced perspective and motion bases, but we haven’t heard much about that since? Was it still used?

JL: I think that was just a plot issue, in Two Towers Sam and Froddo were off by themselves and the plot did not just call for it much, in the third film we did need some but it was done more traditionally with mostly blue screens, scaled moves and comping.

FXG: In the middle of Two Towers you got Renderman vers. 11 – which was the first Renderman version to have High Dynamic Range(HDR) support, did you use HDR in ROTK?

JL: HDR is really about capturing and maps. The thing about HDR is it works really well in something like Fiat Lux where the idea is to put hard shinny 3D surfaces into a real environment, but it doesn’t work so well on set – ’cause on set it is all greenscreens and things you don’t want to have captured for an HDR!

And we actually get a pretty good dynamic range off the film. We have experimented with a few shots but we already have a floating point pipeline and so HDR itself doesn’t tend to be something we use in production. As our pipeline is already very flexible, it does not add much – in a real production environment like ours – with loads of blue screens and nothing on set you want to capture for maps (- that is going to be worthwhile).

FXG: Thanks and congrats again !

The 76th Annual Academy Awards : Best Visual Effects 2004
The Lord of the Rings: The Return of the King,
Jim Rygiel, Joe Letteri, Randall William Cook and Alex Funke.

Fxguide wishes to congratulate the winners and all the visual effects artists who worked so hard on all 3 outstanding films that were nominated this year.