One of the fun panels we attended at SIGGRAPH 2011 in Vancouver covered the VFX in Marvel films Thor and Captain America: The First Avenger. With that in mind we present a look at the work of several of the studios for Captain America.
Conceiving an old-fashioned comic book outing for Marvel’s Captain America: The First Avenger – set mostly during World War II – director Joe Johnston still relied on just shy of 1600 visual effects shots, and more than a dozen vendors for the stereo release. Visual effects supervisor Chris Townsend and visual effects supervisor Mark Soper, along with Method London VFX supe Stephane Ceretti, led the worldwide effects effort on the film. “When we read the script and spoke with Joe Johnston,” recalls Townsend, “we came away saying, ‘Oh, it’s not an effects movie, then.’ There are a lot of shots, but it’s still not a big effects movie, it’s more of a character piece and an origin story about how a skinny guy becomes Captain America.”
In the film, soldier hopeful Steve Rogers (Chris Evans) makes repeated attempts to enlist for military service but is rejected for being small, skinny and sickly. Rogers’ determination is finally rewarded when he is picked for a top secret ‘super solider’ project, transforming him into Captain America and eventually taking on a Johann Schmidt (Hugo Weaving) and his Nazi research department, HYDRA. Chris Evans played both Rogers as a skinny man and as the superhero. ‘Skinny Steve’ would be an effect completed by Lola VFX and one of the most challenging aspects of the production. A number of approaches were considered to complete the final effect, including doing a fully CG head, 3D projections and head replacements, before the 2D manipulation approach was tested. “We asked Lola to do a test which we shot in England,” says Townsend. “We took a really big guy and smaller guy and we said, ‘Make the big guy look like the smaller guy.’ We did that with a shirt on and off, and wanted to see what they could do with mesh warping and 2D manipulation.”
“They came back with some pretty phenomenal results,” adds Townsend. “And we then based our entire shooting on that, and would capture Chris Evans’ performance on set, and the only thing we would be doing would be reduce his size down. It’s not just a ten per cent reduction in scale or thinning up – it’s gaunting up his face, reducing any healthy skin underneath the chin, sinking in the eyes a little bit more, thinning the neck, adding in some shadows on clothing.” Ultimately, Lola completed the Skinny Steve effect mostly via 2D compositing, but also with a number of other techniques, body double reference and CG work.
For an in-depth look at Lola’s work for the film, check out Mike Seymour’s article: Case study: How to make an American wimp.
Planning and previs
Prior to Captain America’s release, Joe Johnston had gone on the record to say that he would not be relying on significant previs for key scenes. Townsend notes that although previs and certainly storyboarding was used, they were just part of the planning process for various shots and not followed to the letter. “So I would turn up on set each day,” says Townsend, “and discuss the scene and the script and Joe would say, ‘Oh, don’t really go by the script.’ And I would say, ‘OK…’. But then I would say, ‘But I saw the storyboards you’ve done.’ – and his storyboards were amazing – and he would say, ‘Yeah, you don’t really want to go with that – that’s probably not what we’re going to do.’ And then I’d say, ‘Well I’ve seen the shot list which says you will be doing a push-in here…’ and he would say, ‘Well, I’m probably not going to be pushing in, I’ll probably do a push-out.'”
“In hindsight,” notes Townsend, “it was the best way to do it, especially for the Skinny Steve shots, because it gave Joe the freedom to do whatever he wanted. And he understood the parameters and knew what we could do. I told him we knew we could do a guy standing still, and it just got progressively harder if he moved his head or took two steps. We would adjust the shooting of the scene to make it easier for us, like shooting from above the feet or working out how his clothes might crinkle.”
That shooting style, says Townsend, also suited the stereo nature of the film, which was mostly a 3D conversion completed by Stereo D. “Joe naturally shoots in a very 3D style in that he uses wide lenses and pushes in on objects, and gets close to things. And the DP Shelly Johnson did a fantastic job. So it became very 3D-friendly, but it’s not a gimmicky film by any means.”
Digital streets and digital feet
Following Rogers’ bulking up to superhero-status, the secret military installation in London is sabotaged and its chief scientist murdered by a HYDRA spy. The newly bulked up Captain America chases the spy through the streets of Brooklyn, scenes that were actually filmed in Manchester, England dressed to look like the 1940s American setting. The Senate provided street extensions and replaced Chris Evans’ rubber skin-colored boots with CG feet.
“There were two streets they mostly used,” explains Senate visual effects supervisor Richard Higham, “and then they kept switching the action around to get a sense of progression. That gives you the first issue, which was continuity and repetition. The buildings themselves are fairly nondescript, but we still had to replace from about half way down the street to all the way back.”
Using a self-devised map of the area based on the edit to maintain consistency, The Senate built only necessary extensions. “We initially considered doing this as a 2.5D projection in Nuke,” says Higham, “but we needed to do a bit more, so we took it into Maya to create the basic geometry. Then we went around London looking for reference to match. We photographed them from many different angles. We also added flags and did simulations to give it some life and created some lampposts in 3D and fire escapes to give it that Brooklyn feel.”
Artists built three CG cars to populate the backgrounds, and would roto live-action cars and people to re-purpose them. Smoke from a car explosion required enhancement to maintain consistency in the scenes. The Senate also created a CG Brooklyn Bridge that could be placed in the background.
For scenes of Captain America racing barefoot after the HYDRA spy, production shot Chris Evans running in skin-colored boots to protect his feet, which would be replaced in CG by The Senate. “We started off by receiving a cyberscan of Chris’ feet plus photographs,” explains Higham. “The cyberscan was very complex, so we re-built the model based on his feet and added in textures. We built an IK system – it has several bones and joints, right down into having joints in the toes themselves. We added in muscle as well, and a fur dynamic for close-ups to re-introduce hairs.”
Tracking the blurred boots was The Senate’s biggest, but most interesting, challenge. Says Higham: “The rubber boot moved like a boot with a slight stiffness. We had to make up the articulation you get in natural feet movement. And feet are very curious objects and there’s a lot of interpretation in what they do. The 3D team, led by Martin Walters, ran about the studio trying to emulate what Captain America was trying to do, filming their own feet from different angles.” The final shots also involved creation of clean plates as the boots were slightly bigger than the feet would be, with Evans’ trouser shorts roto’d over the top and the trouser shadow hand-animated onto the CG feet.
Down at the docks
Steve Rogers pursues the HYDRA spy to the dock area, with Luma Pictures adding in large transport steamer ships and backgrounds. As the camera came very close to the CG ships, Luma relied on its Arnold rendering pipeline, having recently switched from mental ray. “Usually for a close-up shot you would have a very high resolution map,” notes Luma visual effects supervisor Vince Cirelli. “Then for wider shots you’d use a lower resolution map. Well, with Arnold we didn’t need to do that – it handles textures in memory so well, we just painted one very high-resolution map and the renderer solved it.
Luma created four boats in total, three being steamers and one fishing boat. The work also included fluid sims for water and foam and wake creation for interaction between the boats and the harbor. “When they shot it,” explains CG supervisor Richard Sutherland, “they had a stand-in float rig that gave us a little bit of water interaction, but the overall supe Chris Townsend wanted to have some foam interaction in the water to help seat the CG boat into the environment.”
To model the steamers, Luma researched boats from the 1940s eta, ‘Frankensteining’ the reference to achieve the desired composition. Says Cirelli: “On top of the boats we also populated them with little people carrying items and walking across. We placed them on cards inside of Nuke, and also did some cloth sims for flags, just to add movement and sweeteners to help sell the shots.”
In one scene, Rogers is seen running down the dock, and the camera pans from screen right to screen left, following him as he dives into the water between the dock and a steam ships. “To execute that on set,” says Payam Shohadai, executive visual effects supervisor, “they filmed the actor jumped onto a blue airbag. They did a later take with the stunt double running and jumping into the water. It was not shot on motion control and the camera placement was slightly different. So we had to re-create the entire dock by projecting the plate back onto geometry. By re-projecting it and changing the perspective of the geometry, we were able to line the two plates up.”
HYDRA factory and vehicles
As the film continues, more is revealed about Johann Schmidt, also known as the Red Skull, and his mission to destroy key world cities. Learning of this, Captain America breaks into a HYDRA weapons factory. Double Negative, under visual effects supervisor Charlie Noble, extended a factory set shot at Pinewood Studios and added fire and explosions as the building is destroyed.
“The sets they built were quite awesome,” says Dneg’s 3D supervisor Peter Bebb. “We LIDAR’d it and photographed it, and then replicated just about everything in there. We added in big items like the jet engines, the land cruiser and parts of the bomber and a massive chimney system to warrant the external views of the factory.”
Many of the explosions seen inside the factory were practical elements combined with digital fire. “We would shoot massive fuel explosions right into camera,” notes Bebb, “but they would break frame at some point so we did our parallel CG explosions using Dneg’s pipeline here.”
Double Negative also worked on the train heist sequence, as Captain America’s team zip-lines onto a fast moving carriage. “We had a storyboarded sequence and they went and shot plates in the Alps based off those,” says Bebb.”We ended up with both a full 3D and two and a half D environment. When the train was going to go rushing close past rock, we found the same type of rock in England and scanned that and created a bespoke piece of rock. Then there was also a sky dome environment and matte painting.”
Double Negative then completed shots of HYDRA’s bomber, a large winged aircraft that Schmidt plans to use to destroy a number of cities. Production shot the hangar environment for the bomber mostly on greenscreen, with Dneg creating much of the area in CG. As it leaves the hangar, the bomber climbs through a Dneg cloud environment, which is also seen as Captain America engages HYDRA soldiers in a pod fighter sequence. “For that,” says Bebb, “we looked at other films like Joe’s The Rocketeer to see what he wanted, and we went for the magic hour kind of look. We had a cloud pipeline here with sub-surface scattering and deep shadows using our DNB tool. It worked out really well, and we could tweak it on a per shot basis artistically.”
Additional effects by Double Negative included the Land Kreuzer, Triebflugel and Schmidt’s Mercedes, parts of the Arctic sequences, many of the bluebolt gun and shield effects and the cube taken by Schmidt, which emits a strong blue light. “We shot a basic cube and had LED lighting on set to try and get as much blue spill onto Hugo Weaving,” explains Bebb. “It had to be ramped up to give it an awesome power and we were trying to get something that was incredibly powerful that you could hold in your hand, but without it clipping out everything with white light. We used Nuke and full 3D methodologies to get that working. When Red Skull gets vaporized by it, it’s all dialed up to 11.”
Schmidt’s is ultimately revealed to be affected by the mysterious powers of a tesseract, which he had earlier stolen. Underneath his human face is in fact a red-colored, skull-like structure. Hugo Weaving was filmed in a full facial prosthetic piece as Red Skull, with his nose area painted black to read as a cavity and a few tracking markers. The final effects were completed by both Framestore and Lola using a combination of 3D and compositing techniques to depict Red Skull’s striking head and face.
“Part of the development we’ve been doing in the last few years is to hone our ability to rig any creature based on a number of standard defaults,” says Framestore visual effects supervisor Jonathan Fawkner. “We were able to roll out our human being and fit him to Hugo Weaving in his cast. It’s still a lot of bespoke tuning, but it gets you off to a good start.”
The tracking of Red Skull was carried out in two stages. The first was to acquire an overall head track based on the principal photography and a LIDAR scan of Weaving. “We did a lot of hand tracking, but it needed to be absolutely rock-steady,” says Fawkner. “We were building on the experience we’d had with Harvey Dent on The Dark Knight. Only some parts of the face and head are good for tracking the skull – the bridge of the nose and the skull and the eyebrows.”
“Once we had a rock-steady skull track,” continues Fawkner, “which also included a jaw track, we would then go and do what we called the feature track. This was to track all the wobbles and expressions on the face. We did that in almost a 2D way. We unwrapped the texture of his face, and then warped the tracking markers which were there back into place. That pretty much enabled us to render a fully lit and textured Red Skull head.”
Framestore’s compositors could then choose how to adjust Red Skull’s facial features to the desired amount, based on either an unwrapped re-projection of the prostethic make-up onto a model of the Red Skull face, or an actual lit and rendered CG Skull. “We needed to reduce the fatness of the lips,” notes Fawkner. “We needed to bring his chin in, square up his jaw, add sallow in his cheeks, deepen the eye sockets, put the nose cavity on. Sometimes we needed to change the volume of his head because the mask felt too big or bulged over the collar – basically all of this could be done in Nuke.”
“For instance,” says Fawkner, “if he does a big smile, and the complicated smile lines that were describing his expression weren’t there in our rig, it would be a straightforward case of keeping those from our re-projection, but use the CG for his chin line. So it was a case of muddling one in with the next. There’s even one shot where he drinks a glass of water and it goes right past his nose cavity, so that all needed to be refracted and dealt with, but it was all done in Nuke – composited refraction through the glass water.”
In one sequence, Schmidt tears off his entire facial skin to reveal the Red Skull underneath. “The foundation for that was a fantastic performance from Hugo Weaving,” admits Fawkner. “He mimed just as him as Schmidt – literally pulling his own skin off, and did a really excellent job. After we got that, we also got the B-roll of Red Skull pulling off a rubber skin mask. So we had the two sides of that to deal with, but what we ended up with was him revealing a fully CG Red Skull, with a CG warped plate for his skin.”
Artists modeled a piece of skin as a cloth sim to depict Weaving ripping his face off and then re-projected that back onto the plate, adding shadows and CG lighting. “Underneath that,” adds Fawkner, “we had our fully CG Red Skull, which was matched by and large to the B-roll side of that plate, which had the made-up Red Skull in it. So we had nice reference to match to. That scene and all the Red Skull shots just proves that even though it looks like a little thing that needs doing, for the quality we now all expect, it can’t look like a visual effect. It means that you have to take it to another level, which was largely impossible to do as a 2D approach – it needed to be a whole 3D track and render. ”
Rounding out the effects
Several other effects houses also contributed to Captain America – here’s a look at some of the key shots.
Evil Eye Pictures delivered car composites for scenes of skinny Steve Rogers and Mi5 agent Peggy Carter (Hayley Atwell) traveling to the secret military lab. Moving backgrounds were filmed in England and then composited with interior car shots in Nuke, along with extra matte paintings and backgrounds. “Chris Townsend understands that these kinds of shots can be both simple but quite complicated in the details,” says Evil Eye visual effects supervisor Dan Rosen. “We had to add in things like dirt on windows and make it feel like a plane of glass, with shifting shadows on people and camera shake.” Then for a final scene of Captain America waking up in hospital environment, and then breaking out of his room, Evil Eye provided window comps, wire removal and stunt-pad clean-ups, stitching together photography of exterior walls and steam pipes for the outside views.
In a James Bond-esque motorcycle chase, Captain America is pursued by enemy HYDRA bikers, a sequence requiring complex plate manipulation and integration of fire effects by Fuel. “It was shot just outside of Pinewood Studios in the backlot area, which is surrounded by forests,” says Fuel’s Dave Morley. “Cap was filmed bluescreen so there were a bunch of comps. In the live action background plates, which sometimes had the HYDRA soldiers on motorbikes, there were quite a lot of atmospherics – some with fog and some without. We had to tie all those in, along with dappled lighting and make him feel like he was coming in and out of shadows from passing trees.”
Fuel also extended practical 12 foot high tree stumps with digital tops, grappling hooks and some digi-double HYDRA bikers. “They did have a flame-thrower on the back of one of the bikes,” notes Morley, “but we would have to connect that fire into actually hitting people and engulfing them.” Rounding out Fuel’s work for the film were several set extensions, including the design of the HYDRA Base ‘Box Canyon’, an almost entirely CG shot of Cap’s bike impacting on the HYDRA Base door, numerous blue-bolt effects, and a reconstruction of Radio City Dance Hall. The studio also created several shots in the ‘Arctic Discovery’ sequence where the mysterious cube is retrieved by a submersible Bathysphere.
Method Studios provided shots for interior and exterior views of Howard Stark’s plane, commissioned by Captain America to jump over enemy lines, which eventually faces gunfire before Cap parachutes away. Actors were filmed on a gimbal set set with green screen outside the plane’s windows and doors. Method, under visual effects supervisor Sean Faden, altered live action day plates of the Swiss Alps for the nighttime backgrounds, adding clouds and then tracer fire and flack as the plane comes under attack.
“It’s a shiny metal plane, but we also took exterior shots of the airplane a little further than just darkening it and dampening the highlights,” says Faden. “We match moved the plane and used V-Ray to render a night version of it that was lit by moonlight. We rendered a moonlight pass, which gave us more realistic moonlight highlights, and added Stark logos and lights inside the windows.” For the jumping out shot, Method relied on a digi-double of Captain America modeled from a scan and reference photos. Says Faden: “That’s the kitchen sink shot – digi-double, flack, clouds, tracers, door mist, window comps – everything.”
Visual effects were required for Howard Stark’s prototype hover car seen at the World Expo. Although the car was filmed on a rig that allowed it to levitate, Joe Johnston requested that it be given slightly more motion. LOOK Effects, under visual effects supervisor Max Ivins, used 3D projections on a 3D model of the car, re-animated the model in Nuke to the director’s desired motion. The studio also replaced the hubcaps and added interactive lights inside the hover-pods and sparks for when it falls to the ground.
LOOK also contributed to a montage sequence as Captain America takes his team to the fight. After receiving green screen plates of Chris Evans, LOOK used multiple backgrounds, both CG and live action to create the shots, adding CG debris, Cap’s shield and other elements for the shots. Other environments were handled by Whiskytree and Matte World Digital, while Rise FX delivered some complex disintegration effects.
Trixter handled interior shots of the train heist – made up of green screen composites, blue-bolt firearm effects and a digital double of Cap’s friend Bucky who falls from the carriage – sharing some of this work with Double Negative. The studio also completed a flashback sequence in which Dr. Erskine explains the history of Johann Schmidt and the Red Skull character. For the flashback, Chris Townsend requested Trixter visual effects supervisor Alessandro Cioffi give the shots an awkward, almost disturbing atmosphere to show Schmidt’s descent into madness. “Editorial had a number of cross-dissolve transitions and had made it very pacey in the cut,” says Cioffi. “We were provided with all of the layers from editorial and we were allowed to play a little bit with it in terms of color corrections and in-camera effects starting in Photoshop and then completed in Nuke.”
“In the very beginning,” adds Cioffi, “the sequence was meant to be in black and white, but we moved to a more three strip Technicolor type of look, where every channel was color-corrected individually and recombined with the other two, so the final effect had an aged footage look.” Trixter also dimensionalized the sequence, initially giving it quite an unusual conversion with intersecting layers that aimed to disorientate the audience, although this became more toned down in the final piece.
Finally, Rok!t Studio worked on several title cards and the film’s main on end titles, an elaborate stereoscopic move through 18 World War II propaganda posters. “We pulled them apart and redesigned and dimensionalized them,” says Rok!t’s creative director Steve Viola, “with camera moves and transitions from one to the next and made it one long seamless piece.”
The studio initially presented its own designs to Joe Johnston based on the period look before settling on real posters from the era (as they were commissioned by the government, the posters were generally in the public domain). “We took the pieces of the posters and turned them into full-on environments,” explains Viola. “We pulled off some of the typography of the posters in the messages, and instead they are the titles of the film.”
To dimensionalize the sequence, Rok!t used Maya to model on top of the flat imagery and the projection-mapped the original textures onto the 3D objects so that a 3D camera could move around the scene. “We also wanted to keep the integrity of the look and feel of the posters,” says Viola. “For objects that were metallic, say, we painted metallic highlights from the posters – not using photorealistic shaders – but keeping the original textures.”
All images courtesy of Paramount Pictures and Marvel Studios.