A new wave of brilliant invisible effects

In our continuing series on invisible effects, we go behind the scenes of the seamless face replacement work in surfing movie Chasing Mavericks, building extensions and day-for-night work in Hyde Park on Hudson, creating digital traffic for the TV series Banshee and delivering conjoined twins for the film Maattrraan.

Safer surfing

Surfer Jay Moriarity’s spectacular wipeout at Mavericks – a big-wave location in Northern California – in 1994 is the inspiration for Chasing Mavericks directed by Michael Apted and Curtis Hanson. But to capture the story without placing actors and stuntriders in perilous danger from the immense waves, the filmmakers looked to innovative visual effects solutions from Digital Sandbox and VFX supe Scott E. Anderson.

Face replacements

The ultimate goal was to convincingly show Jonny Weston as Jay Moriarity and Gerard Butler as Frosty Hesson tackling the big-wave conditions. “It was a real nightmare situation for visual effects,” admits Anderson, “shooting on a moving surface with a moving subject, a moving camera – all of which you have no control over. And then really getting the actors onto these amazing feats.”

Watch Digital Sandbox’s breakdown of the face replacement work.

To achieve the shots required, Anderson considered a digital humans and faces approach. “But all the work I’ve done doing human digital faces doesn’t quite cut it,” he says. “The true actor performance is pretty amazing, and my goal in almost all the work I’ve done is, how do we hold onto the actor as much as possible? Sometimes we’re replacing them with completely CG, but I’m still going back to that actor’s performance.”

That philosophy guided an early test in 2010, in which a couple of surfers were filmed on the water for face and paddling shots. Anderson’s planned approach was an accompanying bluescreen shoot using sync’d cameras around the principal actor that would allow multiple angles, re-lighting and subtle changes to the final shots. This successful test was realized on 35mm, but the final film and bluescreen work was shot on RED EPICs.

Ultimately, the face replacement shots followed this methodology:

1. Stand-in surfers performed the principal photography wearing tracking hoods. This helped with tracking since often ten cameras would capture the action on the water. Sometimes face replacement still took place even if a tracking hood wasn’t being worn (such as one shot of Frosty passing by camera in extreme close-up).

2. While principal photography was occurring, the usual set measurements and photographic reference still took place.

3. The principal actors (whose heads would replace those of the stuntriders) were 3D scanned in preparation of them being used for tracking models and re-projections.

Gerard Butler and Jonny Weston star in Chasing Mavericks.

4. The original surf photography was edited, and the performance of the principal actor was planned out. “I had broken down literally every shot we were going to do facial replacement for with a cue sheet,” explains Anderson. “These include frosty glances down the nose, a bump here, comes off the lip, turn down. I did an almost every few frame beat sheet that we walked the actors through.”

5. The principal actors were then filmed by a 4-camera RED setup performing on a teeter totter hydraulic rig devised by Anderson and special effects coordinator J.D. Streett. Behind them was a large bluescreen for each angle to be captured. Digital Sandbox had carried out some initial tracking work to show the required performance to the actors. “We pre-tracked and pre-stablized the surfer’s body performance so that the actor wasn’t necessarily distracted by the travel in the frame,” says Anderson.

6. The bluescreen footage was captured often at higher frame rates, with Anderson calculating these as multiples of the principal photography speeds (which were shot from 24 to 96 fps).

7. Aware that the original photography on waves went through quite dramatic lighting changes, the bluescreen setup incorporated a sliding rig so that the keylight could be moved around to match the changing position of the sun. “Knowing that I would be doing re-lighting in post we intentionally over-filled a little bit,” says Anderson. “That meant we were working with a little bit flatter footage than the footage we would get on the wave, but we got a good approximation of the lighting on set in order to avoid CG lighting artifacts down the track.”

8. Once the main actors had been filmed, the shots were realized in comp, a process that involved oversampling in Nuke via frame rate and time. Says Anderson: “We wanted the ability to be able to time slice and time shift. It was almost like a simplified frozen moment rig, where I’ve got oversampling where I can take out performance gaps and move up a performance a little, stretch it a little, slide it a little later. And then by multi-sampling with the multiple cameras for position, we can also either interpolate from one camera to another, or just jump from one camera to another.”

9. Digital Sandbox also looked for hidden cut points in heavy duty scenes. “When Jay does his turn, for example, he comes down and banks and leaps literally out of frame,” notes Anderson. “So I didn’t have to do one contiguous performance, I could let the actor focus on the leap out of frame knowing I had a natural cut point and then focus on the landing point and let him focus on the 360 turn. I could pick up each piece that we would then splice between the multiple cameras, the multiple frame rates and performances.”

10. The final shots came together with additional background fixes and removals, and interactive splashes placed in front of the actors.


The signature Jay Moriarity wipeout was achieved with a combination of real photography and a CG body and board replacement. It began with a real wave plate (later scaled up) that happened to have a master shot and a number of extra camera angles shot by second unit.

See how the major wipeout shot was achieved.

Digital Sandbox then previs’d the wipeout based on photos of the original from 1994 and video that had been shot from a nearby cliff. “We used the two references to get the board flip up, the famous iron cross moment and then flip him back in the wave and tell the story of how much he got slammed,” says Anderson. “We did take some creative license – in the real footage he gets buried back in the lip and you don’t really see him again. But I had seen some stuff at other times where people fall back into the lip and end up riding over the lip.”

The previs was completed for four different camera angles, then provided to editorial who cut the scene as if it had been filmed on the day. Ultimately, two hero angles were chosen for the final shots – a wide angle and a punch in for the flipping and slamming section of the wipeout. At this point, Digital Sandbox refined the digi-double and surfboard animation. “His legs even kick out as he smashes against the water,” notes Anderson. “So even though he’s a dot against a 40 foot wave, you really want it to feel like he’s trapped in there. The first time we showed the previs, people gasped and said, ‘That’s gotta be in the movie.'”

The house at Hyde Park on Hudson

When director Roger Michell needed to show, but not necessarily build, President Franklin D. Roosevelt’s country house in Hyde Park on Hudson, he brought in production designer Simon Bowles and visual effects supervisor Adam Gascoyne from Union VFX. Together, they devised a plan to build only part of the house, and extend the rest in CG.

Watch a breakdown of Union VFX’s house extension work for Hyde Park on Hudson.

The pre-planning also meant that the originally planned locked-off shots turned into more dynamic ones. “The DOP Lol Crawley wanted to get into the back of the car and shoot an approach shot from behind Bill Murray’s head,” explains Gascoyne. “We were very confident so we went for it.”

Union solved the track in Nuke, thankful for Roosevelt’s (Bill Murray) hat in the roto process. The house was then built as individual panels and textured in Mari based on photographic reference. “We built the roof tiles procedurally and textured,” adds Gascoyne. “Having the real textures, paint flecks, glass and curtains gave us a good palette to work from.”

Original plate.
Final shot.

At the shoot in London (which stood in for Hyde Park, New York), Gascoyne oversaw HDR photography, plotted the chart of the sun on the day, and also took various ‘running’ stills – 12 frames of trees moving to provide an accurate representation of shadows and reflections. “The house was never supposed to be a ‘look at me’ house, it was never meant to be featured like that,” says Gascoyne. “So we really were able to feature it a lot more but without it standing out as a visual effect.”

And that went for Union’s other work for Hudson, too. The studio was responsible for several day for night shots, modern artifact removals and the alteration of Bill Murray’s legs to represent Roosevelt’s paralysis from polio.

Original plate.
Final shot.

The day for night shots include scenes at the end of the film, as a car leaves the house. It was shot in the middle of the afternoon and featured, of course, no headlamps. Union added CG lights and light interaction on the road and roadside objects such as hedges. “A good day for night is considering the light when you’re shooting as well,” offers Gascoyne. “It’s not just the case of shooting and grading it down. There’s one shot in particular where you’ve got very bright sunlight coming from behind the clouds that are creating shadows on the floor, which we used as moonlight”.

The leg augmentation involved a brief view of Bill Murray swimming from below. As he passes over camera, it is clear that his legs are more emaciated than the rest of Roosevelt’s body. “The key to that shot was to use the real textures on the day with Bill Murray swimming,” says Gascoyne. “We cyberscanned Murray’s legs, tracked the legs from the plate, made the alterations in CG and re-projected them onto the legs.”

Another of Union’s shots is also likely to go unnoticed. In a long track through and over various people, the camera finds Roosevelt on the radio. Union had to stablize the move to make the shot seamless. “We worked quite a lot to cut everyone out, they were all re-projected onto layers and the camera was then stabilized,” explains Gascoyne. “Nobody is ever going to know that was so much work.”

Dodging cars, and a double decker bus

In the pilot episode of your new TV series, it’s probably not a good idea to have your main actor run over by scores of cars. That’s why the producers of Cineplex’s new TV series Banshee – the story of an ex-con posing as a murdered sheriff – looked to Encore VFX to create CG vehicles for one of the episode’s key scenes. In it, Lucas Hood (Antony Starr) runs through traffic on Fifth Avenue, dodging cars – and bullets – and then avoiding a double decker tourist bus that flips over.

Original plate.
Final shot.

CG cars were determined to be the best approach to the scene for two main reasons, according to Encore visual effects supervisor Armen Kevorkian. Firstly, there was only a limited amount of time to shoot on Fifth Avenue. “We could close it down from 5am to 10am,” says Kevorkian. So to reset all those cars, even with a stunt guy, would have taken forever because it’s a one-way street and only part could be closed.” It was also important that the shots featured Starr’s face, especially when the cars were narrowly missing him, so to be completely safe, digital cars were considered the appropriate route.

To film the sequence, various actions were laid out for Starr, and cones and chalk marks made on the street as to which direction he would be running. Kevorkian also collaborated with the stunt co-ordinator using toy cars to line up the traffic before the empty plates with just the actor and no moving cars were filmed.

Original plate.
Final shot.

Kevorkian took HDRs and panoramic photographs up and down the street to reference building reflections and the surrounding environment. Images of cars that were used on set for other shots were also taken. Encore then tracked the plates in SynthEyes and built car models in 3ds Max or relied on existing ones, reflecting the HDR maps onto the CG vehicles and, using Nuke, also cheated some 2D reflections of Starr as cars rush by. “In terms of animation,” says Kevorkian, “the challenge was to sell movement – braking wasn’t just stopping it was also making the suspension go down, and tweaking those to make it feel just right.”

The double decker bus was created from scratch. “After the shoot I actually went on a tour in a double decker bus to get some reference images and video footage to see what it felt like,” says Kevorkian. “When it flips over there are cups and paper that fly out – it was based on them being quite dirty since they get hundreds of tourists.”

Twice the effect

When Indian actor Suriya was cast to play conjoined twins in K.V Anand’s Maattrraan, visual effects became crucial to realizing the story. VFX supe Srinivas Mohan from Indian Artists, looked to several solutions, including digital head replacement and multi-camera re-projections, to craft shots of the twins dancing, fighting and acting together.

On the Light Stage. IBL lighting pioneer Paul Debevec assisted Srinivas Mohan with the production and scanning for Maattrraan.
Face render.

Principal photography relied on Suriya acting with a body double whose head would later be replaced. Action and dance shots that required fast movements were generally achieved with a digital version of Suriya. Facial scans were taken of the actor on a Light Stage in Los Angeles (with extra assistance provided by Paul Debevec). Wider shots made use of Image Metrics facial animation tech. Suriya’s on-set facial performance was recorded via a helmet rig made up of LED strips, a CCD cam and an iPhone 4s. The digital head model was modeled in Maya and Jupiter Jazz’s Jupiter Skin.

Watch a breakdown of the digital head replacement work.

More static shots of the conjoined twins were achieved by replacing the body double’s head with Suriya’s performance captured separately on greenscreen. But to solve perspective issues, the visual effects team recorded the greenscreen performance from multiple angles using five cameras. A 3D mesh of Suriya was tracked onto the body double that matched the greenscreen performance, before the five camera textures were projected onto the 3D head using Nuke. To calibrate the five-camera setup to the live cameras, an Xbox Kinect sensor helped capture camera placements on location.

See how the head re-projection shots were accomplished.

An interesting approach was also used to properly time Suriya’s performance with the body double’s head and body movements. The body double’s performance was recorded with numbered beats (1,2,3,4 etc) – which Suriya observed and remembered as reference.

3 thoughts on “A new wave of brilliant invisible effects”

  1. Pingback: A new wave of brilliant invisible effects | CGNCollect

Comments are closed.