A Game of Shadows: more to the mystery

With A Game of Shadows, director Guy Ritchie follows up on his first Sherlock Holmes adventure with even more impressive special and visual effects artistry. Overall VFX supervisor Chas Jarrett returned to oversee sequences completed by Framestore, MPC and BlueBolt. We dive into just some of the major shots, including the waterfall plunge, the train battle, digital environments and the escape through the woods.

Robot arms and waterfalls

Framestore’s most significant, and complicated, contribution to A Game of Shadows came in the form of the climactic plunge from the fortress balcony by Holmes (Robert Downey, Jr.) and his arch nemesis Professor James Moriarty (Jared Harris) into the Reichenbach Falls. The camera follows the fighting duo from below as they plummet towards the water, switching to slow-motion and ramping up again for a 29-second shot. In the end, Framestore composed the scenes from high-speed greenscreen footage of the actors strapped onto robotic arms, live action waterfall plates, digital extensions and CG fluid sims.

To orchestrate filming of Downey Jr. and Harris for the waterfall scene, Framestore began with a previs. “Guy was very particular about how we came into and out of that slow motion moment,” says Framestore visual effects supervisor Sirio Quintavalle. “It had to feel natural and properly reveal the faces in the right order. You see Moriarty first with that sense of panic and then it comes around to Holmes who has a sense of calm, so there’s a hint that Holmes has some other plan in mind.”

Watch behind the scenes of the robot arms shoot for the waterfall plunge.

The previs data from a Maya scene was converted for an on-set motion control camera and two large KUKA robot arms, which are used mostly in construction and on factory lines. “They were very accurate for placing things but not great for animation,” admits Quintavalle. “It was tough to keyframe them and smooth frames out. When we got there with our doubles to practice it took about a week solidly just to get these robot arms set up with the motion control cameras to get the whole move worked out.”

“Still, adds Quintavalle, “the robotic arms were absolutely repeatable – their precision was very high. The animating of them is not something they are used to doing – when you build a car it doesn’t matter how you get from point A to point B, they just do it in the quickest time, but with animation you want to get some curves in there and give more of a sense of it. There was also a slight language barrier at first when we tried to describe keyframes and ramping up – in the visual effects world these are key concepts – but at first this didn’t make sense.”

Holmes and Moriarty prior to the jump.

In fact, the arms were so accurate that multiple takes lined up well enough to enable Framestore to use performances from different run-throughs. “We ended up preferring Jared Harris’ performance from one take, Robert’s from another, and preferred his face from a different take altogether,” says Quintavalle.

The arm plates were shot on a Phantom HD Gold camera at 432fps. “We ended up going 19 times slower than real-time for the frozen time moment,” says Quintavalle, “with things moving enough to see stuff rippling. The Phantom can go a lot higher speed than that still, but we didn’t need it to. We shot the whole thing fully slow motion, something like 7,000 or 8,000 frames of footage which we then ramped in and out of.”

Significant clean-up work was required to accommodate the safety cabling harnessing in the actors which caused bulges underneath their suits. Fans, the robot arms themselves and safety crew also needed to be removed. “The Phantom footage was pretty good,” notes Quintavalle. “There are some challenges with keying compared with normal footage, but we could get away with that given it was a nighttime shot. One thing the DOP did was put Mirrorflex on the floor to bounce up extra light and reflect all the greenscreen around and let us key off the floor really well.”

The waterfall was a complicated build for the Framestore team. The initial plan involved inserting Holmes and Moriarty into live-action plates of real waterfalls shot also with a Phantom. After considering falls in Norway, Switzerland and Yosemite, a waterfall in Canada was deemed suitable and a Phantom set-up on a cable was filmed. “This ended up being really helpful,” recalls Quintavalle, “firstly as reference for the water, but we could also use it for the base part of the falls.”

Since the waterfall would essentially be a one kilometer drop, it was ultimately constructed from a combination of Naiad sims, Maya particles, Houdini and the live action footage. “Since we were focused on their faces,” explains Quintavalle, “much of the background is out of focus and so we could use live action plates in multi-layers to get some sense of movement. Then we took over into Maya towards the base before we merge back into live action for the final drop-off section.”

Framestore also completed wide views of the fortress in the Swiss Alps.

Interestingly, the Framestore team found that after looking at significant reference, much of the ‘water’ that flows from the source of a waterfall quickly becomes something else. Says Quintavalle: “As soon as the water comes off the top, everything gets very aerated and loses its liquid quality and becomes much more mist and foam and white aerated water. Still, it had wonderful V-shapes with leading edges with bits trailing off that we could do with different techniques.”

For the frozen time moment, water droplets are seen passing close by to Holmes and Moriarty’s faces, an effect achieved inside Nuke. “We wanted to make them feel in the moment,” recalls Quintavalle, “so we added some droplets of water right up close to their faces. To give us some control and to get the look right we used Nuke’s particle system which had just come out. It very quickly gave us control of the distribution and movement of the droplets. We also put some sprites onto the droplets and it worked well but they weren’t refracting light and giving us the right movement, so we tried doing a warp on them and passing a refraction through them and kept at it until we got the look we were after.”

Nuke was also relied upon for compositing the fall, and in particular orchestrating the time ramps. “We spent a lot of time getting the time warp curve sorted out during the previs to make sure we got the nice ramping in and out,” says Quintavalle. “Once we had that we knew the frame rate we wanted to shoot at – it was a multiple of 24 frames per second so when we needed normal live action we could use exact frames. And then when we were going slow motion we would again do whole frames. We slowed it down 19 times, so at that point we could just drop frames to get real time, and just use all the frames to get the fully slow motion.”

Going by train

Much earlier in the film, Holmes and Dr. John Watson (Jude Law) encounter a group of assassins on a train as it rides through the countryside, engaging in a dramatic gunfight that takes place both inside, and on top of, the moving carriages. Filmed on a practical train set-piece against greenscreen, the sequence required extensive digital environments and CG train replacements to enable interactive lighting and show wider shots. “The environments were a mixture of some footage we shot on panorama-cam cameras pointing in different directions at different angles so we could reconstruct some of the scenery,” explains Framestore visual effects supervisor Sirio Quintavalle. “We also used a crane arm on a train to get some moves as people are thrown out of the train or as things explode for those high, wider shots. We also had to do some fully digital environments.”

Director Guy Ritchie during filming of A Game of Shadows.

Wide shots of the train were filmed on an empty railway track by helicopter with a CG train added in. Smoke and steam from the train stack simulated in Houdini and rendered in Framestore’s fRibGen, an in-house rendering interface. Artists also used a proprietary node-based particle processing tool called FMote to manipulate particles pre-render.

Holmes escapes certain death despite having a gun fired on him – it has been sabotaged with a lipstick. The misfiring gun is revealed as a CG fly-through featuring the pin striking the percussion cap of the bullet which pushes through the barrel past camera. “Guy loves his guns and he was really particular about the shape of the firing pins and what would happen inside,” says Quintavalle.

The final flythrough shot was previs’d, then animated in Maya with a range of shaders used to achieve the metallic surfaces of the gun innards. Compositors used a Nuke plug-in contributed to the defocus and depth of field look. “In terms of the plot,” adds Quintavalle, “it was fairly confusing with the lipstick going into the barrel causing the gun to misfire, so we felt that by going through the lipstick, back through inside the gun and seeing that all through a macro sort of world would not only be a cool scene but also sell what was happening plot-wise.”

See my breath

A small but crucial effect Framestore completed for the Swiss Alps castle scenes was the addition of breath and condensation when the characters talk. “Guy wanted it to look cold while they were on the balcony,” says Quintavalle. “So when they shot the scenes they would open the doors and make the set as cold as possible, but then once the lights came on all the breath would go away.”

Looking for a suitable and believable CG solution, Framestore created a hybrid live-action/particle system triggered by the audio soundtrack. “There were about a 100 shots which needed cold breath, a lot of them dialogue heavy,” notes Quintavalle. “A lot of the conversation is also quite quiet and not always noticeable so we had to make sure it worked for every syllable.”

The system relied on scripting in Nuke to take filmed cold room footage and find suitable ‘puffs’ for the relevant syllables. “In the first pass we’d get a good look at what was happening,” says Quintavalle. “We’d then select different breaths for different sounds – there were about five key ones to match different types of puffs. It’s really one of the effects in the end you won’t even notice, but it’s there.”

A gritty London and Paris

To bring to life many of the environments in the film, Jarrett engaged MPC to re-create London’s Baker Street, several Parisian streets and its Opera house, as well as a weapons factory. For an establishing shot of Holmes’ apartment at 221 Baker Street, production filmed a crane move over a grassy field at Leavesden Studios onto with a partial set piece. “Our work was to build a city to look like we were looking over London,” says MPC visual effects supervisor Seth Maury, “and extend the street to give it a period feel. We matchmoved it and shot some mo-co elements to extend the crowd down the street. Then all the buildings extending down the street were CG, so the shot was made up of the plate, some mo-co crowds and people riding in carriages and then CG buildings and matte paintings. We used mostly painting and reprojections done in Maya and comped in Nuke.”

MPC looked to reference from the era for period London, and also to the first film. Chas Jarrett’s brief was for things to feel uneven and not repetitive. “It needed to be old, gritty London without too much greenery, but it was still a stylized reality,” says Maury. “I wanted the effects to all be invisible because as much as it’s all a period piece I didn’t want the effects to be noticed. It was hard to sell it was London without any greenery but that gave it an industrial feel.”

– Above: watch MPC’s breakdown reel for Game of Shadows.

The film moves to Paris, where exterior shots of the Parisian square, Place de l’Opera, and the Opera house were also completed by MPC. “Chas wanted it to feel like a big gala night at the opera,” notes Maury. “Most of the environment was CG buildings based on textures of buildings we photographed that were actually in the square, but we closed it off to make it feel more intimate. In the big reveal as Holmes and Watson come running out of the opera house back to the hotel after they release they’re in the wrong place – that was made up first a crane move that was shot at Greenwich and we matchmoved it, went back out to an airstrip north of London where we shot some mo-co elements for people, and then added CG buildings behind the crowd. There was a lot of texture painting and then went in with DMP and projections.”

Interior audience shots for the Opera house came from texture stills, since production could not film inside. “First they shot the audience plates,” says Maury, “then we matched all the lenses for the shots they were going to use, and then for the views from the stage looking back out at the audience we shot people on sets of rises doing different poses and actions so that we could then comp together to fill up the audience. So we had a high-res stitch of the background of what it looked like without the audience, and then we had a rough model of the interior and projected that stitch onto that. Then in comp we layered in all the people.”

A bomb is discovered in a hotel across the square, with the resulting explosion a combination of digital and practical work from MPC. “We started with a practical plate and then shot an explosion element,” explains Maury. “It didn’t work too well at first – we did two explosions and the first was too smokey and did not have enough fire for what was required. The second one went right off the greenscreen and after a lot of iterations it still wasn’t quite working. By that time, though, we had a CG version of the whole build so we replaced the area with our CG version and could put interactive lighting on it. Then we found some other library explosions elements and executed the final shot in the comp.”

Referencing Krupps

Holmes and Watson later find themselves at an ammunition factory, intended to be based on the real Krupps war time factory. The scenes made use of CG MPC buildings, matte paintings, a CG watchtower and a digital Big Bertha gun fired by Watson. “Chas had shown me a book on the history of the Krupps factory and how it had grown up over time,” recalls Maury. “What fascinated him was how nothing was in a straight line, it had all been bolted on – it was build it as you need it. Chas thought of it as a functioning city, not just a factory. So we studied the book and built a library of about 20 CG buildings based on factories in Germany and England that the art department referenced. We shot textures and then just kept building up the city. There are some high views down, and a major establishing shot of the railyard.”

Watson fires the Big Bertha gun and destroys the watchtower. On set, only a light on a crane stood in for the CG building, which MPC then destroyed using its finite element analysis FX tool called Kali to realize the brick destruction, and then Flowline for the accompanying smoke and dust. An earlier practical effects set-up for a brick wall explosion also informed MPC about how the watchtower should come down. “There’s a shot at the railyard when the Big Bertha fires through the wall – initially that was meant to be a practical effect,” says Maury. “The special effects team rigged up a ‘shell on a wire’ – it was a shell on a tension wire hooked up to crane and the wire was pulled through a brick wall laying on its side. It punched through the wall so fast that it didn’t destroy any of it, so it wasn’t exciting enough, so we did a CG brick wall. But we actually kept that in mind the whole time when we were destroying the tower because what we had seen was that a big metal object traveling quickly won’t actually blow out as big a piece as we thought it would have.”

Escape from the woods

Holmes and his pals make a dash through the woods in a sequence that again drew on slow motion and ramped up effects work. For that scene, MPC enhanced some of the practical explosions, bullet hits on trees and through clothing. “The scene also has an effect of punching in and out of the action,” adds Maury. “You’re traveling with the action and then you’ll just punch in tight on it. They had two Phantoms mounted side-by-side – one with a wider lens and one with a longer lens. Each one had a camera operator. The editor took the footage and cut it together to give us an idea of how he wanted to punch in and out of the action. We took that as a guide and polished it and made it smooth.”

Most of the explosions in the chase were practical and detonated nearby to the actors, but to enhance the danger, Chas Jarrett shot secondary elements of mortars, dirt and trees exploding that were added in by MPC. “The one that sticks out most clearly to me,” comments Maury, “is where we’re tracking sideways with the guys and we zoom in as a bullet goes through Watson’s jacket and we pull focus with a tree right in front of us which explodes. The bullet going through his jacket was a practical element – Chas took a dummy and bent it over in the position that Watson was. They shot it to get all the cloth movement. We took that and replaced that on top of the live action.”

Watch behind the scenes of the woods shoot.

Digital destruction via FEA tool Kali was only used sparingly for the sequence, including for the shot of a bullet shell coming in sideways, impacting with a tree and causing it to explode. “They had earlier shot some Phantom footage of a balsa wood special effects tree exploding,” says Maury. “It was super-cool – it just had all different sized pieces spinning quickly, slowly. It was great reference but the trouble was it had a lot of dust and sometimes that would ignite. Guy Ritchie didn’t want any fireballs, so we took that reference and used Kali but I think Chas did a great job of getting that practical reference first, because there’s no substitute for in-the-lens, right?”

All images and clips copyright (c) 2011 Warner Bros. Entertainment Inc.