Visual effects supervisor David Vickery discusses Double Negative’s work for Harry Potter and the Deathly Hallows: Part 2, including work on the digital Hogwarts and the Gringotts bank environment and dragon. We also speak to Dneg executive producer Dominic Sidoli, a veteran of several Potter films, about this last VFX outing.
fxg: What was Dneg’s general scope of work for the film this time around?
Vickery: We did 410 shots overall in the final edit, over about 50 sequences. The main shots were the arrival at Gringotts, and then all of the cart ride and cavern and the dragon’s vault. We then did the dragon escaping across the roofs of Diagon Alley. Then, pretty much everything to do with Hogwarts we were involved in – set extensions, exterior sets of the courtyard or battlements, the viaduct collapse, the shield creation, the shield attack and its destruction. We did the Death Eater plumes and Death Eaters in the air as well. We also did the title sequence.
fxg: Hogwarts was such a big part of this film – how did you plan out that work given that it would be such an enormous digital creation?
Vickery: It was definitely the biggest thing we had to do – we started working on it in summer 2008 and we were still working on Half-Blood Prince at that point. Tim Burke (the overall visual effects supervisor) and the director David Yates were very keen to see whether it was plausible to do a digital Hogwarts, to free them up in filmmaking terms. We went on set and photographed the miniature before it was deconstructed. Then we started doing tests to show the client that a CG believable Hogwarts would work in both wide shots, full close-ups and as set extensions. That was the easy part in a way. The difficult part was then taking that and creating an asset that could be used in the film. When we were building our asset, we still had no idea what any of the shots would be. They also didn’t really know at that stage how much set they would be building.
fxg: So where did you start with modeling Hogwarts?
Vickery: Ultimately we were tasked with building this asset that was essentially seven miles from one end to the other, with a completely digital school in a completely pristine state and a destroyed state. We weren’t sure how close we were going to get to any of it. But we had to prepare for it both in terms of the environment and in building the asset. I think one of the cleverest things we did was that our lead environment artist – Pietro Ponti – was tasked with going to Scotland where they shot the mountains surrounding the school and capturing that data. He set up a three-camera rig, left, center and right. The cameras were all set up with remote trigger shutters. He went up in a helicopter and they would fly semi-circular flight paths around every single mountain they needed to capture, at different heights.
Pietro plotted all these points out using Google Maps and Earth, so that before he went he was able to work out the best time of day depending on the time of year they were shooting, so that they got the best lighting for every mountain. The helicopter pilot was really great and also super-excited that Pietro had managed to do all of this GPS work. They just fed the co-ordinates into the helicopter nav system and just flew perfect positions all the way around the sides of these mountains. It was an amazing little trick.
We got all of this information back at Double Negative, and used a piece of software called Photo-fit, which is a photogrammetry tool we used to re-create the mountains. Once we had all the mountains, we worked closely with Stuart Craig, the production designer, and he created a collage of geometry surrounding the school and we could put the mountains anywhere we wanted. The next tricky thing was actually just modeling the school. In the end we had a team of about 30 people working for two years building the school. It was fully pristine and brand new, but then also torn to pieces.
fxg: Because it was such a big asset, did you build it in a modular fashion?
Vickery: We have an asset system at Dneg which allows us to build models in multiple levels of detail. So we’ll build a low-res model for previs purposes and low-res rigid body sims. Then we’ll build a high-res model that can take the textures through to final lighting. Each of the pieces of Hogwarts could be separated. So Dumbledore’s tower for example was made up of eight or nine separate models, and we would never have to have any one of those things loaded into the computer at any one time if you’re working on a live scene. We could just have a preview version of the school visible, but at render time it switches out for a higher detailed version.
Depending on how close to camera any one item or asset was, it would automatically select a higher level of detail. Close to camera would use our ‘A’ model and stuff in the distance went with the lower res geometry. It meant we had a very flexible lightweight Maya scene that was usable for the artist. We also did a lot of displacement extraction, so for the terrain we would use fairly high resolution meshes, then go into Mudbox and paint incredibly high resolution textures and displacement maps. Then we’d use displacement extraction to apply all of the high resolution fidelity detail to an incredibly low res model. It got so involved building this asset that we ended up having to put grass on the hills, using a Houdini tool for sculpting grass and shrubs across the landscapes. We had such brilliant photography of Scotland that we had to match our work always to the source material.
fxg: What about the detail on the stonework and Hogwarts building itself – where did reference for that come from?
Vickery: Real buildings. It was a combination of Stuart Craig’s amazing team which have literally done thousands of architectural drawings of literally every single school building. You could re-create Hogwarts from these if you really, really wanted to. We had all these blueprints that went into the tiny details of each of the tiles of the gapping stones in the Great Hall, to the arched windows, and even profiles of hand rails that ran around the inside of the staircases. So that was the starting point, then Pietro and his team went to the actual filming locations that Stuart had used to design the school – Durham Cathedral, Edinburgh Castle, Winchester Cathedral and others – which we extensively photographed.
The actual physical textures were painted by hand, but when you look at real buildings you realize there can be some really weird things going on in there. You’ll see there are even small trees growing out all over the place in the higher reaches of old cathedrals, coming out of old brickwork. And just the way that one face will be worn because that’s the predominant side the wind or sun is coming from, which fades things differently. And different buildings sag slightly too.
Stuart would come into Double Negative a number of times to review our work. His keen eye for detail was put to the test. He would scrutinize every little thing. We’d produce these renders that we thought were perfect – beautiful and photoreal renders that would match their courtyard perfectly. Stuart would look at it and go, ‘Ah, the mortar lines look a bit like they’re a bit too wide over there…’. And then when we started destroying the school, we started getting into what was inside the walls. He would say, ‘You can’t just break a wall and have three layers of bricks, because actually if they did that there would be a course of brick and then inside that there’d be a course of rubble, and behind it another course of brick.’ So we ended up getting a specification for how thick each wall around the school should be and what would be in the middle and on the outside. It was literally a masterclass in architecture.
fxg: When it did come time to destroy parts of Hogwarts, how did your model interact with some of Dneg’s destruction tools?
Vickery: Well we already have this great pipeline of tools, so that meant we could spend a lot of time doing the creative side of the work, rather than too much R&D, because it was already done. Essentially we took a fairly old-school approach to destroying Hogwarts – so if say a roof collapsed we would make 3,000 roof tiles and build the roof as a real object from Stuart Craig’s set plans, and then we would drop a canon ball on it or fire big forces at it and see what happens. Because we had built the entire school we could actually knock it down for real.
We’d also run these physical simulations on huge walls or rooftops or turrets and knocking them down and seeing what happened – that gave us a beautiful organic feel to the destruction as well. They would stop working somewhat if we needed to art direct the destruction, at which point we would go in and sculpt and model holes in the sides of walls. We also had areas of destruction that had to match the real set, so we had to sometimes physically place planks of wood and beams and masonry detail, especially for the courtyard set.
fxg: How was the protective shield that goes over Hogwarts generated?
Vickery: Firstly, Tim Burke gave us a real free hand in the aesthetic and let us put some versions of that in front of him for ideas. So we had big creative input, and then that was taken on and driven by him. It also became a shared effort between 2D and 3D. Sometimes in visual effects, shots like this are driven by the 3D particle simulations, and then comp is doing the finishing touches. But in this instance, because it was such a creative concept, it came from a lot of different inputs.
Tanya Richard did a lot of concept work in the first instance, and then Alexander Seaman, our effects lead, put together some fantastic Houdini effects work, particles and cloth simulation and some straight-up texture painting. He handed it to 2D artist Christine Wong, who was a really creative compositor and defined the look of the shield in Nuke, also using a lot of the 3D tools in that software.
fxg: There was a lot of sharing of shots and assets between Soho vfx houses on this film, especially for Hogwarts. Can you talk about how the work moves between facilities?
Vickery: The amount of collaboration was pretty incredible. If I wanted something from MPC, I’d pick up the phone and ring Greg Butler, their VFX supe, and say, ‘Hey, are you working on this, can we have the stone knights?’ And he’d send that over in the afternoon. We weren’t really competing with each other, because when the work had been awarded, we all sat in the same room together. Tim Burke and Emma Norton (the overall VFX producer) would divide the work the way they thought best, and then let us sort out the details of interacting with each other. We’d have these meetings where we’d work out the easiest way to work on the shots and sometimes things shuffled back and forth between facilities.
Sometimes we’d do things like the backgrounds and they were relatively easy shots to share. It got more complicated sharing things with say MPC, who had done a lot of previs for the battle sequence. Their previs would be passed over to us to do the shots. Early on we shared our previs Hogwarts asset with them, and they built stone knights and viaduct elements that they gave to us. We also of course kept Tim and Emma and Warners in the loop.
fxg: Changing tact to Gringotts, can you talk about the cart ride down into the cave? How was that sequence planned and shot?
Vickery: It started with Kieron Helsdon, our environment lead, who was installed at Leavesden Studios for a short while in 2008. He was there working with Stuart Craig and Tim Burke putting together the previs for the cart ride. It meant that all the ideas they had came to the visual effects department first hand. So there was a lot of planning and early previs. They used a motion control base they’d been using for the last few films – a huge six-axis base, and they built the cart and had the actors on the front and back. There were fully-controlled animating axels that we did end up replacing in CG.
We would essentially control their motion base to move the rig based on animation extracted via our previs. Sean Stranks, our 2D supervisor, Rick Leary, 3D supervisor, and myself were on set when they shot it. We took the HD feeds from the rushes and do bash comps and test that the whole thing was working. David Yates was very specific about the roller coaster aspect. He said that’s what it needs to be – a huge, fun, bone-shaking roller coaster ride. I think it works really well in stereo as well because you get that massive sense of vertigo as you’re dropping down.
fxg: How was the whole environment built? Could you actually run your track and cart through it?
Vickery: It was built like that to start with. Stuart Craig’s team built a scaled miniature about 10 feet tall and six feet wide of the entire cave, which we scanned. It had the location of the waterfall and the vault, and various rails running around it. Kieron’s team then essentially animated a sequence which travels through that cave in real time and real speed, and set about rendering those shots. The only problem then was that once they’d filmed it, editorially they’d decided that certain shots worked better in different orders, so it was jumbled up a little. But Keiron had very cleverly built the cave in a modular way, so when they needed a new shot to slot in between A and B, Keiron could then re-create a bit of cave using the rock from one shot and a ledge from another, and create new environments that way.
Rather than the cart running along the rails, the rails were generated by the movement of the cart. We would animate the camera, animate the cart running through the frame and animate the axels of the cart. Then we’d run a script which would essentially generate the rails that the axels would run on. It meant that we had control over where the rails were in the frame the whole time. Originally, because they were spinning and twisting through the frame, we often had some problems where the rails were suddenly going through the field of view of the camera, and that was causing aesthetic and technical problems to render. So by using the position of the axel to tell where the rail should be, we were able to keep them out of the field of view of the camera.
fxg: Were the actors in that sequence sometimes done as digi-doubles?
Vickery: There were a couple of shots of them as digi-doubles. There’s a whip pan of the cart running left to right, and then later on in the sequence of them falling in a big wide shot. But in the majority of the shots it was the real actors. Also, because we knew this would be stereo, we knew we’d have to do full body tracks of all the elements that were shot, and so we were able to re-project all of the plates onto digital versions. That gave us flexibility for moving around the elements and people in the frame quite a long way to re-scale them or even re-angle the cameras quite considerably, because we were re-projecting onto 3D geometry.
fxg: When they meet the Gringotts dragon, you can really feel the pain it is going through – how did that aspect of the character drive the design?
Vickery: It was always supposed to be a really tortured, malnourished, emaciated dragon. We even had to look at some horrible RSPCA photos of dogs that had been mistreated for reference, which had a certain look in their eye. The design process was a collaboration between us, Stuart Craig and Tim Burke. We got some initial concept art which was handed over to our 3D concept artist, Kristin Stolpe, who sculpted the dragon in Mudbox and did concept-painted 3D versions to get sign-off on the overall look.
The tricky bit we came up against very quickly was that as soon as we got it working and had a believable look signed-off, we had to think about how we were going to film the kids. What we ended up doing was passing our 3D version of the dragon to special effects supervisor John Richardson who then laser cut a 10 to 12 foot section of the back of the dragon, that the kids would sit on. Then he sculpted a full scale version of that part of the dragon. He then created a silicon hide.
At the same time back at Dneg we were working on the animation rig and the motion control rig that we would use to drive the on-set practical dragon that the actors would sit on. It was a standard motion control base and then a rig that allowed us to move both of the shoulders and three points of the spine along the dragon’s back – it could be controlled and driven to match the animation in our shots.
Back on the design process, once we got past the concept stage, we were keen to essentially do a shot. It’s so easy to do turntables on a static background, but when you put it in a shot it looks so different and you start appreciating the finished product. We took one of the shots in the bank when the dragon bursts up just through the floor and then falls down. We started that in late 2009, and just kept getting feedback from the director.
We spent a long time on the getting the right color of the skin. They didn’t want it to be a pink dragon, or a blue dragon. It was amazing how sensitive everyone was to the tinniest little hint in the skin. The other thing was making it tortured and emaciated enough that you felt sorry for it. David Yates was adamant that it not be a creature you look at and be disgusted by – you were supposed to emote with it. You had to look at it and feel simultaneously terrified and sorry for it.
fxg: What was behind the skin and muscle development of the dragon?
Vickery: Well, the animators would firstly take it to a certain level. They had some very finite controls over really small details. Aside from your usual rig elements, they had wobble controls for loose skin underneath the neck, they had fairly accurate controls for positioning the panels of the wings and the cloth between the wings. Chris Lentz, the animation supervisor, and his team would go through and do that and then it got passed over to the dynamics team.
The first thing they would do is simulate the muscles. We had built an accurate skeleton of the dragon, which had a fairly simple muscle geometry built in. We then had a Dneg plug-in called Beefcake which allowed the geometry of the muscles to then deform and inflate or deflate the exterior of the skin. We then had a displaced set of tendons across the neck, the shoulders and the hips, which were controlled by the effects guys and animated to fire on and off all the time. They also added in controls for veins – essentially where they start moving underneath the skin.
On top of that we added in skin-sliding over the top of the bones and muscles. We painted maps to allow the skin to slide in some areas and not others. Then we added the cloth simulation for the wings. Plus we had all the chain simulations around the neck and shackles for the feet. And of course we then went through a whole lighting and rendering process. We would follow these processes as add-ons – so we would do the animation, take a cache of that, add the skin and bones, and on top of that cache we’d add the cloth simulation and so on.
fxg: In the shot where the dragon does emerge in the foyer of Gringotts, how did you accomplish all the debris and destruction there? Was that chandelier in the shot also something you had to replicate?
Vickery: Oh yeah. And, you know, I found one in a pub! It was a very bizarre moment early on when we were first working on the shot. These chandeliers in Gringotts are huge – they’re probably six feet wide and very tall. They’re made up of individual little crystals on wires. I walked into a pub in London and there was a miniature version right in front of me. So I spent the next 20 minutes shaking it around and filming it with my phone trying to work out what it did, getting lots of stares.
On set they built the lowest section, a hemispherical part, and we had to do set top-ups for all of the foyer. The chandelier itself had to be sim’d. It’s quite simple to build it because it’s just one string of crystals duplicated round and round. Our rigid body tool Dynamite got a little bit annoyed with 20,000 to 30,000 individual crystals, but it handled it. It was literally sim’d as a full rigid-body object and then allowed to collide with a fully rendered cache of the dragon in a completely digital set.
Actually, for all those shots of the dragon, we started out mostly with a background plate of the vault or the foyer and then expected to put the dragon into the plate. I think the reason the final results are so successful is that we actually ended up replacing all of the plates and just using CG environments, then adding in the dragon and CG fire and general destruction. It gave us an unprecedented level of believable interaction between the dragon and its environment. We would match the live-action sets exactly. And then that ends with probably my favorite shot in the sequence which is the dragon clinging to the rooftop and destroying all the chimneys before it flies off. It just represented all the good work we had done.
__________________________________
We also asked Dneg executive producer Dominic Sidoli about having worked on Deathly Hallows Part 2 and other Harry Potter films.
fxg: Can you talk about managing the Potter VFX work from a production point of view?
Sidoli: From a production point of view, the Potter movies have always been a dream to work on. The projects have had a near continuous visual effects production department over the past several years and I’ve worked on each of the movies near back to back (not all at dneg of course), so it has been a little like working on one very long movie where you constantly have the time to finesse your production processes and develop the relationship with your client. That’s a rare, if not unique quality amongst movies.
As a result, the whole process is very well organized and lends itself to spending the maximum time developing the visual effects which are themselves so broad ranging. The plan for developing a Patronus spell is going to be very different to the plan for facilitating a fully digital Hogwarts, and we often find that we have tens of dozens of uniquely different challenges within the same movie on Potter, so being able to forward plan as much as possible puts you in the best possible position.
fxg: There seems to be a unique way in which the work was shared amongst the London facilities – what’s your perspective on that?
Sidoli: Sharing work on complex sequences increases the number of dependencies in your production pipeline and introduces additional technical and creative considerations. Sometimes a number of different vendors may be involved in a single shot: You provide me the environment, I animate creature A, someone else animates creature B and then finally someone else altogether converts the final result to stereo. It’s sometimes a necessary byproduct of splitting types of work between different vendors to sensibly spread the load and to enable each vendor to focus on a predominant type of work.
Tim Burke (the overall VFX Supervisor) and Emma Norton (the VFX Producer) spend considerable time deciding how to split up the work beforehand to find the best overall result and they involve the vendors in that thought process, so it’s something that we’ve all bought into and planned for. The vendors have established very good working relationships over the years and we all want to see the best image possible on screen, so we make every effort to make the process as transparent and straightforward for each other as possible. Sharing work with stereo vendors on occasion is a relatively new complication to the already complex puzzle, but everyone has adapted quickly – planning and re-adapting our plan is what we do every day.
fxg: From your point of view, what kind of impact has the Potter films had on the London VFX industry?
Sidoli: The franchise has contributed hugely to the work that we now see in Soho. Harry Potter was one of the first movies that demanded such a variety of work, and at such a scale that each of the vendors were able to take on these amazing moments and expand their repertoire. When I joined Double Negative several years ago we were about eighty staff, we’re now over a thousand and Potter has played its part in that expansion; for this film alone we had some two hundred and fifty plus crew working on it.
All images courtesy of Warner Bros. Pictures.