Effects you never knew were there

We continue our invisible effects series with a look at some recent productions with incredible, yet seamless, visual effects work. Read about Tippett Studio’s historical VFX in Hemingway & Gellhorn, Digital Domain’s retro LA scenes and crowds for Rock of Ages, Mikros Image’s limb removal carried out in Rust & Bone, miniature combinations by Look Effects for Moonrise Kingdom, effects by Level 256 for The Dictator and how DIVE pieced together a subway scene for Safe.

When Nicole Kidman and Clive Owen met FDR: Hemingway & Gellhorn
Original plate.
Final shot.

We’ve seen on-screen characters interact with famous persons of history before – think Woody Allen’s Zelig and Robert Zemeckis’ Forrest Gump. But in Hemingway & Gellhorn, an HBO film on the lives of journalist Martha Gellhorn and her famous writer husband Ernest Hemingway, director Philip Kaufman wanted to use the latest invisible effects techniques to place the two in actual archival footage spanning the Spanish Civil War through to other historically significant events of the 1940s. That effects work was led by Tippett Studio and visual effects supervisor Chris Morley.

“Phil Kaufman directed The Right Stuff and used a lot of stock footage, and saved so much money to create an epic feel,” notes Morley. “He did it again on The Unbearable Likeness of Being with a Russian invasion sequence, and then shot reverse angles in the same city square with the actors. Then editor Walter Murch cut those two together and you had this feeling of being there. So with Hemingway & Gellhorn, he wanted to use a third dynamic – not only do we use archival footage and shoot reverses, we actually embed the actors into the archival footage.”

Morley was entrusted with figuring out how to shoot the actors – Nicole Kidman and Clive Owen – in a quick way and on a television budget. He came up with a greenscreen/match move solution that involved sourcing and scanning archival footage, bringing in a 3D camera, working out the perspective through that 3D camera and using the data to line up the actual camera on a greenscreen stage. Here’s how the process worked:

1. Sourcing archival footage

Morley worked with the director Kaufman and editor Walter Murch for a year before principal photography to conceptualize the shots based on the archival footage (found in personal collections and university film archives). “We sat with a stock footage editor and decided which shots would work best for the script,” outlines Morley. “We’d talk about where the actors would be and what they’d be doing. Then when we were on set I was armed with that information, and if there were other people in that archival footage, if they were doing something and looked in a particular direction, I would clue Nicole or Clive or any actor that they were in there – we would suggest a little nod to them for example.”

Morley needed the scans of archival footage before the greenscreen plates were filmed, but did not want to do any clean up. “We wanted to embrace every scratch, and I just loved that,” he says. “So often in our field everything is pixel perfect but I loved messing it up a little bit. So it was really therapeutic.”

A ‘rag-tag team’ of the filmmakers also carried out some tests using a Canon DSLR to test the process. “One of the PAs would play Martha, the location manager would play Hemingway and we would go out and shoot these little tests,” says Morley. “Phil Kaufman is not used to shoot digital effects films so it was an educational experience for him.”

2. A digital grid for match moving

The digital match move process began with the creation of a digital 20×20 foot grid in Maya. “I then purchased a black shade awning that was also 20×20 and I used orange duct tape to make a 20×20 grid that matched the digital grid,” explains Morley. “So we had a digital and a practical element. We’d bring in the digital grid with the computer generated match move camera, line that grid up on the ground of the scanned in archival footage, export all the camera data – how high it is off the deck, tilt, roll, lens size and everything. We would export a snapshot image of that grid – the grid’s perspective through that camera – and have that on set.”

Watch Tippett’s behind the scenes video for Hemingway & Gellhorn.

3. The greenscreen set

On the greenscreen stage, Morley had storyboards, the archival footage, the digital grid and the camera data – all this was fed through video assist software known as Conduit. “It was nodal-based a bit like Shake, and was a real time compositing video assist tool,” says Morley. “So you’re able to bring an archival footage plate, bring in the feed from the camera that’s shooting the actors on the greenscreen stage, real-time key out the green and composite them roughly into it.”

The production shot on the ARRI Alexa and tended to rely on very soft lighting, despite the stark lighting generally found in the archival footage. “One huge problem was, ‘You don’t light Nicole Kidman with stark lighting,'” recalls Morley. The DP was like, I can’t light Nicole with a stark light from the top – she’s going to have monkey eyes. So what we did was shoot everything very soft but with a definite directionality to it. We were able to control the contrast. It was a balance between putting it into reality and making it cinematic.”

“We used a zoom lens on the greenscreen stage so we wouldn’t have to switch lenses out,” adds Morley. “We could go from a 28mm to 32mm really quickly. We’d get those settings onto the camera and then we would load up the digital grid image and overlay it over the greenscreen where I had laid out the shade awning – then we would A/B those together and check out the cameras to make sure they were lined up. Once those two grids lined up perfectly, we locked down the camera and then pulled out the practical grid and we were left with nothing but green.”

One of the benefits of the archival footage was that it was generally filmed locked-off without two many sweeping camera moves, enabling Morley to also film locked-off scenes. “Wherever there was a pan in the archival footage,” he says, “what we would do is stabilize the pan, then create a single image of every view of where that pan would pan to, so we’d be left with a long panoramic image and have enough real estate to do camera tracking.”

Morley had a monitor that allowed him to walk around the set ‘in’ the archival footage. “That way I could set marks where I needed the actors to look or for them to go around a piece of debris. The director knew exactly what we were going for. Of course, all the full integration isn’t there but the idea is very clear. For Nicole and Clive – it really helped them a lot but they were just in a sea of green. It was a magic box to show them that we weren’t crazy.”

4. Creating the final shots

Original greenscreen plate.
Final shot.

While shooting was taking place, visual effects assistant Nathan Hackett carried out slap comps, feeding those to Murch for editing and providing a template for Tippett Studio in creating the final shots. For these, the first step was to take the archival footage and stabilize and track it. “When we put the actors into the archival footage,” says Morley, “we needed to integrate them into that certain film stock with scratches. There was also a lot of production footage of taking the audience out of a traditionally shot film into that archival world. So we’d put dust and grain, scratches and jitter over pristine production footage shot with the Alexa.”

To get the right kind of dust and scratch look, an old Finnish newsreel was scanned. “It had this really wonderful long black leader at the beginning and with the numbers,” recalls Morley. “We scanned all that, so it was 380 frames of just black leader. I clipped out all of the numbers that would pop up on single frames. We were left with this beautiful sequence of black with dust, dirt and everything. We used a key for the darks and a key for the bright, and used that to color correct the luminance of the production footage. So what it does is it actually uses all the pixel information of the production footage but tweaks it – wherever the dirt and scratches are. It gave a real authentic feel to it.”

This work was done in Nuke, which also served the visual effects artists in tracking the original archival footage. “Nuke’s tracker let us switch between single frame and normal mode,” says Morley,” which was really robust when doing frame by frame tracking for the tiny moves in the archival footage.” To composite the greenscreen actors, Tippett paid close attention to matching shadow density, mid-tone density and detail depending on whether the footage was 16mm or 35mm, and finally added in matching defocus and gate weave jitter.

In one particular scene, Hemingway and Gellhorn meet Franklin D. Roosevelt and Eleanor Roosevelt at the White House. “They had almost studio lighting in the archive footage,” says Morley, “which we could pretty much match perfectly. I had C-stands for Nicole to look at and then told Clive just to look at the back of Nicole’s head. One challenge was when a gentleman comes in, blocks the light and sits down. In order to get that kind of feeling, I was off camera with a flag and I covered the light hitting Clive. We then just roto’d Eleannor and painted out the existing diplomats, tracking in a still to fix the background, put Clive and Nicole back in and matched the grain.”

5. Non-archival shots

Amongst the archival footage effects were scenes filmed by the production that had to match real locations and the era. One such shot featured Gellhorn running down a Madrid street as it is bombed by advancing planes. “Phil Kaufman wanted this intimidating epic shot showing Martha’s courage,” says Morley. “We looked at archival footage of Grand Via in Madrid, but it didn’t have the detail we needed. So we used it as a guide and did a complete matte painting of 1930s Madrid. We also animated CG planes and used all of our dust and explosions from our elements library here at Tippett.” Realising the greescreen set would have to have been 50 feet high for the designed shot, Morley ended up having a portable greenscreen pushed behind Kidman as she runs to camera.

Another scene features Hemingway’s Cuban residence, Finca Vigia. For a shot of Gellhorn walking up stairs into the house – complete with signature cats – Tippett Studio combined elements from various places. “It has a very signature facade at the front with stairs,” says Morley. “We found that at a Catholic school which had a garden with really beautiful steps and a religious shrine behind the steps. At first they weren’t going to let us shoot there, but I put them at ease because I only wanted the foreground stairs. We had a bluescreen covering that and we had a cat wrangler.”

Then on set in San Francisco, the front steps section and door was built to match the Finca. Now Morley just needed the actual house. “We originally thought we’d have to matte paint the house, but after we finished shooting, I decided I wanted to go to Cuba. I went to Cancun and flew to Cuba – and I went to Hemingway’s Finca. I couldn’t bring my laptop, but I had my camera, and I had to try and remember what perspective we shot it in. I timed it right in the morning to where the sun would be. I took pictures of the Finca and we also used that element in comp and it worked amazingly well.”

Digital Domain rocks out: Rock of Ages

Much of Adam Shankman’s musical film Rock of Ages takes place in 1987-era Sunset Boulevard. This LA location had to feature iconic landmarks; nightclubs, stripclubs and the Tower Records building to name a few. But to shoot in present-day LA would have been cost-prohibitive, not to mention the change in scenery since the late 80s. So the filmmakers turned to shooting elsewhere – in Miami – and to Digital Domain to bring the LA scenes, as well as crowds for the film’s rocking concerts, to life.

Original plate (left) shot in Miami and final shot (right).

“The production and art department found an area that they could own and take over,” explains Digital Domain visual effects supervisor Jay Barton. “It was just outside of downtown Miami – a few blocks – that currently weren’t in use at the time. So they could go to town changing the facades, putting things inside the store fronts. They could really art direct what they wanted. In one case, they built out an area and put fake glass on the front of it and made it like a store window for a strip club. The set was really impressive. They had a huge cast of extras and populated the area to make it feel really fun.”

Another side-by-side comparison showing before and after shots for Sunset Boulevard.

Production shot street level scenes that then required removal and replacement of backgrounds, including Miami’s skyscrapers. Plates were supervised by Marc Dominic Rienzo, DD’s on-set VFX supervisor based in Florida. Digital Domain used photographic reference taken in Los Angeles and a matte painting/geometry approach to build various angles of Sunset Boulevard and surrounding landmarks, like the famous Hollywood sign, and also matched the more hazy atmosphere of LA which was much different to Miami’s clear air. “We created a library of all these assets,” says Barton. “It’s authentic to what we have in our heads, although maybe not the geography of that time.”

For two concert scenes, one the notable Dead or Alive sequence, production filmed inside the Hard Rock Arena in Hollywood, Florida. Around 1,000 extras came dressed in 80s wear. Originally, Digital Domain was only to fill the stands with CG crowds, but they soon realized to make the concerts feel as big and imposing as they needed that most of the structure would require augmentation. “Other than the crowd at the front of the stage, everything else is digital architecture and crowds,” says Barton. “This was an extension of our crowd work for Real Steel and some other recent shows.”

A view from the Hollywood sign.

DD’s crowd system involved shooting around 50 extras in different outfits from various angles and multiple cameras, and then using a Nuke script to populate them in the arena. “We set up a 20 foot greenscreen cyc out in the lobby of the arena and we kept shuttling out the best of the extras,” explains Barton. “We’d do them in groups or one or two or five of the time. We’d have them do a specifically timed set of actions and rotate them all the way around to get them from multiple angles.”

“We shot with three cameras,” adds Digital Domain CG supervisor Nikos Kalaitzidis, “at six feet, eight feet and ten feet from the center of the greenscreen. We’d end up with five minutes of each person’s performance from three different cameras and angles. We took that into the Avid, we cut it down, pulled the greenscreens and put them into our library of actions.”

The crowd tool then figured out where the extras should be sitting in relation to the relevant camera position, and picked the correct orientation from the library elements, blending between different actions if the camera moved around. To create variations, compers could isolate individual areas of a person and change colors and hues, for instance, but the main differentiations came from relying on time off-sets and frame ranges. “Within one minute of performance we’d off-set the frame range of what they are doing,” says Kalaitzidis. “We’d have them sit down, relax, bob their head, and as time progressed they’d jump up out of their seat and really rock and roll and bang their heads. At the end of their performance they’d relax again.”

DD created this stadium and the crowds for the final concert.

Dodger Stadium featured in two visual effects shots, a helicopter establishing scene and then a CG pull-back past screaming fans that ends with an aerial view of LA. Digital Domain used Dodger photo reference, and imagery from other stadiums, to create a suitable-looking stadium, and then utilized the crowd tool to populate it. “From our library,” says Barton, “we could populate a 10,000 person arena for the Dead or Alive sequence all the way up to the Dodger Stadium sequence – 50,000 people – for the final song.”

Seamless limb removal: Rust & Bone
Original plate.
Final shot.

In Rust & Bone (De rouille et d’os), a whale trainer played by Marion Cotillard loses her legs in an horrific accident. Director Jacques Audiard turned to Mikros Image to realize the leg replacements and limb removal from the able-bodied actress as seamless effects.

“Jacques Audiard is a director who’s not used to working with visual effects,” says Mikros visual effects supervisor Cedric Fayolle. “We had to come up with a solution that would not slow down the shooting and be as discreet as possible on set. We chose a totally free camera that allowed us to get the VFX footage the same way it was shot. After each shot, we would double or triple the shot. I was constantly on the set to decide on the position of the legs so that it would be OK to remove and to check out that backplates could be rebuilt easily.”

Mikros worked closely with the art director, costume department, props and lighting teams to find a solution for each shot requiring limb removal. “Decoration made holes in furniture so that Marion could hide her legs and the wheelchair was customized so that Marion could sit crouched down,” says Fayolle. “Clothes were selected to be loose-fitting so that it would hide forms as much as possible. We had created latex stumps molded from Marion’s thighs. We modelized them in CG. Once the legs where removed, we replaced them with the CG thighs.”

Cottilard is also shown in the film with metal prosthetics, thinner than her real legs. For these Mikros adopted a number of different methods to create the desired effect. “On set, we had several prosthesis styles,” explains Fayolle. “For shots say with the child who touches the prosthesis, a real one was used. Marion was standing just behind, all we had to do was remove the leg. But the contact when you see the child touching the prosthesis was a live shot. For other shots, Marion had to wear green stockings that we would remove and replace with CG prosthesis. We always took light references and HDRs with the real prosthesis on set to have an idea of how we would have to re-create things to bring as much visible detail.”

Floods and other whimsy: Moonrise Kingdom

Wes Anderson’s whimsical 1965 journey into the lives of young New England Khaki Scouts in Moonrise Kingdom relied on subtle digital touches, environment enhancements and small scale miniatures to tell its story. For a flood sequence at the end of the film, Look Effects integrated a miniature water shoot by Fantasy II Film Effects with matte painted elements to create the final shots.

Look Effects created various environment enhancements for Moonrise Kingdom.

“Also, during the big rain storm,” says Look visual effects supervisor Dan Schrecker, “there’s a sequence on the roof of the church which had a Mary Poppins vibe. They did miniatures of distant buildings, basically shooting stills of cardboard flats. Wes always wanted to keep things as practical as he could. We did a bunch of sky replacements and lighting tweaks.”

Look Effects also comp’d in a miniature tent element – that explodes – into a live action plate, and tweaked environments for the film’s island and 1960s setting.

Goat duplication, and other effects: The Dictator
Original plate.
Final shot.

You just never know what you might be required to do in visual effects, and for one effects company on The Dictator – Sacha Baron Cohen’s latest comedy – goat duplication was just one of the tasks they had to handle. That work was completed by Level 256, one of several VFX shops on the film, which delivered almost 70 shots for the film ranging from greenscreen comps, split screens, sign replacements, wire removals and set extensions.

“The goat replication shot was a pretty standard 2.5D ‘crowd’ replication,” says Level 256 owner Scott Davids. “They had a bunch of takes and pretty much just let the goats run around. We scoured the dailies and found some hero goats, stabilized them, put them on cards in Nuke, and then staggered them in space. The most challenging thing about the shot was maintaining the ground interaction, the shadows, and making sure that the luminance was correct for all of our goats. There were clouds blowing very quickly over the location so all of a sudden some of the goats in the crowd would get darker – we just dialed them in to make sure that they were all consistent. Actually, Sacha’s goat herder character is from another take as well, so I guess it was a split screen and goat duplication shot.”

Original plate for set extension.
Final shot.

Greenscreen plates proved a more significant challenge, partly from the various formats (Alexa, RED and even Canon 5D footage). “The (overall) VFX Supervisor, Eric Robertson, did a great job lighting the greenscreen plates, but because they had to seamlessly intercut with material which was not greenscreen, we had to do a lot of sculpting to get the color densities to match precisely,” says Davids. “For some of our greenscreen work we did not have specifically shot backgrounds to use, so we ended up having to make them from alt takes of the scene from principal photography. This usually involved stabilizing 2 or 3 takes and effectively removing the actors that were to be replaced from the backgrounds, but keeping the movement of the extras. We would build out ‘hero’ backgrounds for all of our angles that were 1000+ frames long and then use that to keep the continuity of the movement in the BG intact.”

“The 5D material was shot as a test with the intention of reshooting it later,” adds Davids, “but the improv was so funny that the filmmakers did not want to take the chance of reshooting. Our 5D plates had a ton of channel damage from the compression as well as moire patterns, so we spent most of our time fixing the 5D footage before we were even able to begin integration with our fabricated backplates. Eric helped us meticulously dial in the color to the rest of the scene that was simultaneously being colored in the DI. In the end we feel our shots really worked well, we considered it an invisible victory.”

How Jason Statham rides the subway: Safe
Original plate.
Final shot.

Invisible effects were used to piece together multiple shooting locations for a dramatic NYC subway chase in Safe, the Russian mobster Jason Statham-starrer directed by Boaz Yakin. The sequence, in which Statham’s character jumps on the back of a subway car, shows angles looking down at the track, the bridge and water below, a helicopter view of the train crossing Manhattan Bridge – all pieced together by DIVE from footage shot in Pennsylvania and New York.

“It was quite a puzzle,” recalls DIVE visual effects supervisor Mark Forker, who also oversaw other visual effects ranging from muzzle flashes, car accident comps, breaking glass and knife and stab wounds for the film. “Production didn’t want to build the inside of a subway car when there were subway cars they could shoot inside in New York, but there were cost and safety restrictions in shooting inside a ‘moving’ subway car. In the end we had to shoot at three different locations!”

Scenes of Statham atop the train were filmed against greenscreen at the Fern Rock facility in Philadelphia. “They shot that on a train in a yard where they repair the cars,” outlines Forker. “The top of the cars look like the top of any train, so anything where we were just looking down to the top of the car we could do there. But it was just a stationary car with green behind it. While we were on that greenscreen set we also shot Jason running along – a profile view of the train moving across the bridge for a helicopter shot of us following a train moving across the bridge.”

That helicopter shot featured a plate filmed in New York of a subway car going across a bridge. Then, plates of the PATCO train system were used to construct textures of track and bridge details added to the scenes as 3D models in Maya. For inside the train, plates were shot against greenscreen and composited with plates filmed from a truck. “Most of the bridges the subway went across also had car traffic,” notes Forker, “so we could fake the exterior footage by shooting it that way.”

Watch a promo for ‘Safe’ featuring some of DIVE’s effects for the film.

There were also shots where DIVE needed to show the tracks. “Here, the camera’s above him looking down on the top of the car as he climbs up over the edge,” says Forker. “The New York subway system would not allow us to shoot with a camera hanging off of the back of the subway and we needed the environment of the bridge and the tracks below behind us, so we shot those in New Jersey on the Franklin Bridge, doubling for the real one.”

To marry the different locations together, Forker looked to the New York helicopter shot as a source of the lighting for each angle. “The sun was practically coming right us,” he says. “So on the side-on he’s almost a silhouette for all intents and purposes, and then when we got on top of the subway we had to suggest that strong light coming in from the right hand side. But all of the pieces we had were not lit like that – so we tried to take any of the lighting out by flattening all the plates down. We would mimic going past big posts so that the lighting wouldn’t stay consistent – it would flood and stutter as we passed different elements of the bridge.”

DIVE artists completed the comp work in Nuke. “We also used grading techniques and mattes to make it look like there was dappling and animation of the light as it passes parts of the bridge,” says Forker. “Then some selective hold-outs on him so he’s getting hit with more light on the right than he is on the left.”

Check out our previous invisible effects articles via the links on the top right hand side of this article.

12 thoughts on “Effects you never knew were there”

  1. Gotta love how mike says that FXGuide would warn if there were to be nudity in their content… I didn’t get any warning here on that spot with the girl with the missing half of the legs.


    “We would 100% warn if there was nudity.. but there is none we saw… in another post with a real model – we blurred things that some people might find offensive. As a father of younger children – I respect anyone’s request for a warning but these were digital mannequins. They are – in our minds – the same as undressed shop front mannequins. There is no explicit nudity.”

  2. Hi
    Yes we would warn of nudity, the image you are referring to in this story has a woman – clearly wearing a bathing suit, but being carried, so even if she was nude – which is absolutely is not, – you would see nothing.
    Not sure Jeff what the issue is, but I stand by my comment. In a world of provocative advertising and semi-naked models, this image is valid and I would have no issue with anyone seeing it, no matter their age.
    Again if I have misunderstood let me know, but perhaps you are joking – if so I missed the joke.


    1. No joke. If you watched the whole thing you would know I was referring to the guy. He touches his dick for crying out loud… how is that not nudity?

    2. Another thing, I do agree with you’re comment “In a world of provocative advertising and semi-naked models”. But that doesn’t make it right, unless morality subjective and not objective.

  3. Apologies, Jeff. I’ve removed the video for the time being. It was an external video not hosted by fxguide.

  4. Sorry just to be clear that video when we posted it – did not have nudity, it was changed and as we were just linking to the video – the version you saw did involve nudity. This is a risk of us linking to third party videos- something we don’t do lightly – and I would say this is the first time we have ever had an incident like this.
    So again sorry. The version we approved in editorial was not that version.


  5. Pingback: Week 7: Planning, plotting and pilfering… | Digital Adventures in Composite Video

  6. Pingback: Rock of Ages | Ryan Coster

  7. Pingback: Effects You Never Knew Were There | Tippett

Comments are closed.