Engine Room makes a splash with Soul Surfer

In 2003, Hawaiian teen surfing star Bethany Hamilton lost her left arm to a shark attack. The story of her return to professional competition is now the subject of Soul Surfer, with actress AnnaSophia Robb playing Hamilton. To create around 500 shots for Robb’s missing arm, and other effects in the film (totaling 750), director Sean McNamara turned to LA-based visual effects house Engine Room. We talk to the studio’s founder and the film’s vfx supe Dan Schmit.

fxg: Five hundred arm removal shots seems pretty daunting. What were some of the ways you thought you would approach these, at least initially?

Schmit: When we first went into it, the assumption I think from the producers was that it was going to be all digital. It’d be essentially putting the stump on and creating that for every shot. It didn’t take long for us to play around and discover that a really good option was to work with an actual prosthetic in combination with the digital work. So AnnaSophia would often hold her arm behind her back and her shoulder would be pushed back, so we could put a little prosthetic in the front that looked like it was sitting in a normal position.

Before and after images of Engine Room's arm removal VFX

Schmit: Well first we actually did a screen test. We worked with a great make-up and effects guy, Mark Garbarino, to develop the prosthetic. We did a 35mm film shoot with John Leonetti, the DP, and with Sean the director of a body double with a green covered arm running around on the beach and jumping in the water. We brought that back to Engine Room to start working with it and develop our techniques, for (a) getting the green off the background and (b) how do we reconstruct the parts of her body once the stump is taken away? In the end we took those methods into our production.

fxg: So once you got to the shoot, can you describe a typical shoot set-up?

Schmit: The day would begin for AnnaSophia in the make-up chair having the prosthetic applied, which was about an hour and a half process. Mark would be there on set and would essentially cast a new silicone prosthetic for her every day. We had three different looks for the prosthetic because the stump changed through the movie as time goes on. At first, it’s got a bandage on it, then that comes off and the scar is really fresh and the arm swollen. Then for the later scenes in the movie, which are six months down the track, the stump is more toned and tanned. So we had that kind of continuity to keep track of. AnnaSophia wore either a green neoprene sock or we would paint on chromakey waterproof green paint that Mark developed.

fxg: How did you shoot the green arm scenes?

Schmit: We went in with a plan of saying, ‘We’re going to have, say, just three green arm shots in this particular scene,’ but as soon as we got into the movie we realized that the visual effects needs were never going to be the deciding factor in any scene. Whatever was shot was shot and we were going to have to deal with it. We talked about lots of close-ups and lots of lock-offs, but of course every shot was moving and lots of shots showing different angles. For every set-up we would have her perform and either myself or Michael Caplan, our visual effects producer, would be on set in Hawaii doing on-set supervision.

fxg: So what kind of specific challenges did having to remove the green arm and prosthetic bring?

Schmit: Well, the prosthetic was designed for AnnaSophia’s arm to be behind her back, but that was not always practical. We had to decide on a shot by shot basis what made the most sense. We couldn’t have where she was putting her arm inhibit her performance in any way. So often her arm was hanging right there in front of her and we would have to resolve that by reconstructing the whole side of her body.

One of the excellent techniques we developed to be able to do that was shooting an additional plate of the side of AnnaSophia’s body. After the end of every set-up we would have her raise her arm up and just rotate her torso around a little bit while the camera was still rolling, so we could see what her body looked like without her arm covering it. We just needed a few frames of that. We would also shoot a clean background.

fxg: How did split the work up back at Engine Room?

Schmit: The first thing was that we decided to split the work up amongst artists here and around the world. We abandoned the artist-based facility model several years ago at Engine Room. In our early years, we’d have 20 or 30 artists in-house. But we’ve realized it’s better for us to have a very high-end small team in-house and work with a lot of off-site people. So we had six in-house ‘rock star’ generalists, and about 18 off-site who were anywhere from across town to across the world. Most shots were done by a single artist or a very small team, and that lent itself well for being easily divisible.

fxg: Can you take me through how a typical green arm replacement shot was done, say for a surfing scene?

Schmit: We had some nice techniques for the water work. We discovered that if we were moving a green arm over a moving water background, rather than trying to patch water in from another part of the shot or a plate, we found we could do a lot of with stretching. We’d take the water that was next to the green arm and horizontally stretch it to create a clean background and fill in the areas behind the arm. We found that technique worked well for the sand beaches behind her as well. There was just something about the sand texture that meant you couldn’t tell it was being stretched as well.

We had figured that out in testing, so when we were shooting, we asked the water camera teams – some of the best surfing cinematographers in the world – that whenever they were framing a shot that had the surfer’s green arm on, we would ask them to please frame the shot so there would be a little extra space, so there would be something there for us to stretch.

fxg: How did you co-ordinate that work, given that it was going on in so many different places?

Schmit: We came up with an interesting analogue way of unifying the look. Initially we were getting a lot of good shots back, and each looked great, but they were all slightly different in terms of resolving some of the areas that the stump looked like from behind, for instance. We realized we needed to define for everybody what it needed to look like by coming up with a clear reference.

For that we started with a 3D model from a body scan of AnnaSophia – which was taken with her arm behind her back and the prosthetic on – and basically removed her stump and making a clean stump model in 3D in Maya. Then we had a 3D object print made as a polymer cast. The print was of Sofia’s shoulder and the stump and her arm gone, in life size. We then took that piece and gave it to Mark Garbarino and he cast it and made it out of silicone. It was painted in the same way he made it up on the set every day, but now it was complete. It was the stump from all sides.

So now we had this prop, and what we ended up doing was setting up a little photo studio at our place and then for any particular shot where we needed to see what the stump was going to look like from a particular angle, we did a little series of stills on a turntable. We’d light it to look like the lighting of the scene and then shoot a series of stills for the compositor. It could be 50 stills of the stump from all different angles, but as a photographic element. Then a lot of the work was done using those stills – morphing, time re-mapping them altogether into a seamless composite. It worked really well and allowed us to control the look of it and answer questions very quickly.

fxg: What was your toolset for the arm replacement work?

Schmit: All of our compositing and painting work was done in CS4 at Engine Room, but offsite it was a real combination of Flame, Nuke, Fusion. We also did some 3D work in Maya.

fxg: Can you talk about some of the other effects work you had to do for the film? Were there some face replacements for the surfing scenes?

Schmit: Yes, so there were three competition scenes in the movie, so three times you have these scenes where you’ve got six girls in the water, each with their own rashguards, surfboards, and the water unit – all the underwater camera guys with hydroflex camera housings, sound guys in inner tubes. And ADs with walkie talkies in zip-lock bags, floating around on little dingys.

And a lot of the surfing scenes were rewritten in the editing room with the material that had been shot. Part of that meant stealing shots from different competition scenes. The only problem was in say that scene the surfer was going left to right instead of right to left, and the surfboard’s blue instead of red, and the rashguard’s green instead of blue and it’s the wrong girl!

So we found ourselves doing a lot of shots like that where we were painting surfers out of the water or painting people in the water that needed to be there for continuity. It involved changing just about every aspect of the surfers from their bathing suit, to whether they were goofy foot or regular, and their faces. We had the most success using photographic elements we had shot of AnnaSophia against greenscreen, rather than anything from the 3D body scan.

fxg: What would have been the most challenging shot or shots in the film?

Schmit: Some of the arm replacement shots were very difficult, because AnnaSophia’s green arm was facing camera for a long, long time. But there were also some tricky shots that were surfing face replacement shots, and what was particularly interesting with these was that the real Bethany Hamilton did a lot of surfing in the film. Well, Bethany is now 21 or 22 years old and is basically an Olympic athlete, and AnnaSophia is 15.

So early on we were a bit weary of doing a face replacement on Bethany’s body because the scales of their bodies were very different. That was a real concern, but as the movie progressed and all the surfing footage was analyzed it was determined that the surfing that Bethany had done was the best, so it was just assumed we would figure it out! And we able to do some pretty impressive scaling in compositing to make those shots work.