Here’s how Atomic Fiction, led by visual effects supervisor Kevin Baillie, reproduced Philippe Petit’s daring high-wire act between the Twin Towers of the World Trade Center on August 7th, 1974 for Robert Zemeckis’ The Walk. The solution involved many parts – high-wire training undertaken by the film’s star Joseph Gordon-Levitt, lengthy previs, a greenscreen studio in Montreal, simulcam setups, stunt doubles, face replacements, meticulous replication of 1974 New York and the Towers (including hand-animated traffic!), innovative cloud rendering, some neat compositing solutions and well-planned out stereo – at a “very responsible” budget.
The Walk was originally intended to be achieved via performance capture, back when Zemeckis was producing such films at ImageMovers Digital. Baillie was at ImageMovers then and continued the dialogue about how the film would be made after starting Atomic Fiction with Ryan Tudhope. “We always knew it was going to be a set build of the rooftop of the World Trade Center Towers in some scale and then a sea of greenscreen,” says Baillie. “The debates were really, how much of this set piece do we build, and do we shoot this outside or inside on a stage?”
Some of those questions were solved using previs, a process that began several years ago. “We used previs so we could tell where the characters would be running around on the roof,” explains Baillie. “We ended up building a 40 foot x 60 foot x 12 foot tall section of the corner of one of the rooftops, which was the south tower where most of the action happens. We would build right to the extent of where people would be walking and not an inch more. Everything else was visual effects set extension. The height was really a factor to make sure there was enough height for the camera to get underneath Joe or his stunt double as they were walking on the wire, enough to get any up angles that were necessary.”
The VFX supe was happy that the shoot was on a stage (at the MELS film stages in Montreal) to avoid the problems of changing light and weather. DOP Dariusz Wolski rigged an area of space lights in the ceiling to replicate the sky. “Dariusz and his team were able to give a very broad, soft illumination from a cloudy sky,” says Baillie, “or in the case of sunrise they could shut off half the ceiling of these lights and leave the other half on and then kick on a light that was coming from the sun. He could really work with the lighting and the ceiling system to create the bounce light of the sun off the clouds to give it very natural light. It would have been incredibly hard for us to do amazing visuals had the lighting lighting on set not been so good.”
Back to the top.
Walk the line
As the project morphed into a live-action film, Petit’s death-defying walk would need to be portrayed convincingly, and for that several methods were adopted. Gordon-Levitt was trained by the actual wire-walker himself just prior to shooting, and continued training during photography. For scenes on the wire, production utilized the actor, or his stunt double, on a real wire, on special planks, or on what became known as the Canadian Bar, as Baillie sets out below.
“The wire was rested inside this Canadian Bar. It was a 20 foot long steel beam with a slot in the middle that we were able to fit the wire into. So we’d move it below the wire and raise it on these tripods so that it sat in the beam. Joe’s feet would still be on the wire, and he’d still be able to feel the wire, and his foot would curve around the wire, but then he would have the beam on either side of it to support him if he was getting off-balance. Occasionally in comp, we’d use NUKE’s warping tools to warp the wire very subtlety with his weight. We used the shots of either Joe really walking on the wire or his stunt double walking on the wire for how the cable moved and how it kind of breathed during the walk experience to make sure we were emulating that same look. And if his feet were touching the Canadian Bar and flattening out and not curving around the wire as much as they should have been, our compositors would go in and curve Joe’s feet a little more and even paint in a little rounding to make it look 100 per cent believable.”
“The key to all that succeeding,” suggests Baillie, “was actually that there were so many different methods that we used. Sometimes Joe was walking without the Canadian Bar, sometimes it was the stunt performer, Jade Kindar-Martin, who’s a Cirque du Soleil trained performer, and then sometimes it was the Canadian Bar. If we’d used the same method again and again it would have become obvious that we were faking something, but because we were only faking it sometimes, you buy it more.”
Walking on the wire was one thing, but the stunt of course takes place 1,400 feet above the ground with a 1974 New York, including the towers, in the near and far background. Since the plates would be filmed against green, a solution was found for all to preview what would be featured in the final shots. This process began with the early previs. “The ImageMovers Digital previs team, who were from The Third Floor, did the original previs for the movie,” describes Baillie, “so we had an idea of what we were going to be doing there from the previs.”
On the set, the filmmakers relied on techvis from The Third Floor and a simulcam approach feeding in the previs assets using technology from SolidTrack for real-time camera tracking. Says Baillie, “the real camera always had a simulcam going on it so that our operator working the crane and could design some insane moves. The simulcam would really allow us to really see vanishing lines and making sure the top of the tower wasn’t cutting through the character in an awkward way, for example. After we got back from the shoot and editorial was putting together a rough cut of the film we had the The Third Floor do postvis as well.
One challenge faced during shooting was the cross-over between previs, the actual partial set-build and assets that Atomic Fiction had begun digital construction on. “We had previs assets from the original previs that was done,” notes Baillie. “But then we realized they weren’t the same as the set build that had been done off of blueprints of the towers. We had to figure out where the differences were, take the previs assets, tweak it to match the simulcam just enough so that we could rely on it on set. Then in the postvis there was another round that went into it, but Atomic Fiction had started building the hero assets by then and could contribute some of those dimensions to the postvis team. By the end of the process we had a model that was a dead nuts match of the set build and the entire building.”
Since a stunt double was used for some wire work, and with occasional views requiring pull backs or push ins further than the greenscreen set would allow, Atomic Fiction did produce a couple of dozen face replacement and digi-double shots. “For those,” says Baillie, “we worked with a company called Pixelgun which does 3D scanning with photogrammetry. They have a rig of over 100 cameras that they surround the actor with, and all those 100 cameras fire simultaneously and then they use photogrammetry – using the minute perspective differences between each camera – to not only reconstruct the geometry of the face and the body but also since it’s using photographs, we get fully detailed pore-level texture of the face. This is really important not just for the face in general, but also if he’s grimacing or really furrowing his brow when he’s concentrating, the skinning system when – you might have blood going into your forehead making it go a little more read, but on the sides it gets more pale. It captures that subtle blood flow and the peach fuzz hair and the changes in the pupils.”
“We also worked with Faceware to bring a head-mounted camera out on set on the last day of the shoot,” adds Baillie. “We said to Joe, here’s the 50 takes we might need to do face replacement on. We rolled those for him and he acted those out walking on a piece of tape on the ground. We were able to take his actual facial performance and put them on the digital face that then went on Jade, the stunt performer’s body. So when you see a face replacement it’s not just sticking Joe’s face on there, it’s sticking his performance on there as well. It’s Joe’s acting on Joe’s face on the stunt performer.”
Above: Faceware has more on its role in the film.
Back to the top.
The towers rise
An iconic figure of the New York skyline for more than 25 years – both physically and emotionally – the Twin Towers were crucial, but challenging, assets that had to be built for the sequence. “You look at them and they’re basically just a bunch of parallel lines going 1400 feet down to the ground,” says Baillie. “If you try to build that in the most literal of senses, it looks CG so fast. There’s nothing like even geometry to make something look fake. So we couldn’t just follow blueprints. We had to put in a lot of artistry – it wasn’t even a mathematical process to make it look real – it was about tweaking the panel gaps or making sure straight lines aren’t quite straight, or it was about how to make it look like the construction workers hadn’t necessarily put the panels on straight and didn’t quite line up right. You couldn’t mess it up too much otherwise it looked crappy and it really blew the scale.”
Reference here was key, of course, and the art department and Atomic Fiction team scoured thousands of photos and pieces of footage – most of which were not in digital form. “People tend to forget that when the towers were around up until 2001, there was very little digital photography of them,” states Baillie. “Everything was shot on film and was grainy and not really directly usable, but of course we could draw a lot of inspiration from it. We also had blueprints for the entire towers. We contacted the Port Authority in New York for information about the whole area.”
One aspect the team noticed quickly on the towers was the quality of their exterior which was made from anodized aluminum. “This looks matte,” says Baillie, “but it’s actually very broadly reflective. What that means it takes on the qualities of the environment around it. So on a bright sunny day it might look white and then on a dark cloudy day it’s going to look almost charcoal. So we had to figure out how to replicate that exactly, otherwise the towers wouldn’t feel that they were the towers people had seen.”
Since the towers would be affected so much by their environment, it became clear that the entire surrounding world had to be built in 360 degrees. “We couldn’t just focus on what’s in the camera but we had to be mindful of what’s also behind the camera at all times,” comments Baillie. “If there was a dark spot behind camera it would taint the look of the towers.”
The supervisor says that a large part of the success of implementing the world and also the anodized aluminum look came from Chaos Group’s new V-Ray GGX shading model. “This is a micro-facet shading model that simulates all the little bumps that are in a surface. That’s exactly what anodized aluminum is, just thousands of little bumps in every square inch that make it look matte whenever you’re looking straight on but then when you’re looking at a glancing angle it almost looks like chrome. So when we started experimenting with the shading model it was the thing that made those towers go from looking like pretty good CG to looking real.”
An early test for Atomic Fiction would be a teaser trailer shot that zooms up the side of one of the towers and ends with Petit scanning the rooftop for his first view of what is to come. “That shot was particularly challenging,” says Baillie. “We only had 12 feet of height off our floor before we reached the top of our rooftop set. So we had to take our camera crane and put it on the deck and tell our operator just to lean on the crane as hard as he could to accelerate it up and over the edge.”
“Then in post we took the camera move that was real and 3D tracked it. Then through that virtual camera we actually re-projected the set on every frame onto 3D geometry. So that way what we could do was take a digital camera that was starting way, way down – 1,000 feet down below – and zoom it up the tower, and film the plate re-projected onto 3D geometry for a good 120 frames until the momentum of our digital camera caught up with the momentum of the real camera, and then over a single frame we were able to switch to the actual production footage. Because it was for the teaser trailer, we were still developing the asset and trying to figure out how to build the city for the latter part of the shot and building the rooftop. There were a lot of moving parts to work on.”
Although it was a dramatic virtual camera move, the building zoom still included a ‘wobbling camera’ to input some level of realism to the shot. “We’ll often seek out similar camera moves from elsewhere in the movie or take inspiration from other films that had been done, and really study them and try to match them,” explains Baillie. “It’s not just about applying a bunch of mathematical noise to a camera and making it a little jittery, is it more roll it’s doing, or the path up the building that’s different? To add imperfection is great, but the right imperfection is the important thing to make it feel like a real camera.”
Atomic Fiction had also garnered from various reference that views of inside the towers could be seen through the windows at that extreme angle. “You could actually see the ceiling lights and some blinds and desks and bookshelves in the windows,” notes Baillie. “So we detailed the interiors of the WTC tower building for about 30 floors and could replicate those. Having a visual depth and parallax inside the building helped to make it a lot more believable. It also gave the 3D conversion guys even more depth cue to play with.”
Once on the roof, there was one further challenge remaining (apart from the surrounding world, detailed below). This was a collection of protective plastic sheets flapping in the wind, signs of the building still under construction. “Only one of them is real,” acknowledges Baillie. “That’s the one that was on the welding tank that’s behind Petit. We had to roto that out and apply different levels of blur to the background through it. The great thing about having that there was it gave us a great reference to ground every other piece of blowing plastic in. The rest of them were all cloth simulations with thin plastic shaders.”
Back to the top.
During this first roof-level view and during Petit’s delicate walk, the New York surrounds are on show, often for lengthy shots and in constantly changing weather and lighting. “He steps out on the wire when it’s sunrise,” notes Baillie, “and by the time he ends the walk it’s cloudy and blustery. We couldn’t just make one big matte painting and re-use it throughout the whole movie. We really needed to build it out in 3D and render it in excruciating detail.”
Again, heavy book and footage reference was utilized here, while Baillie also filmed plates of current day New York to get a sense of “how does the city feel from that high? How fast do cars and people move, what do things ‘look’ like? It would be very easy to put too much activity in the city and all of a sudden it looks like a miniature, then not enough and it looks dead.”
Atomic Fiction then began building a 1974 New York. “I can tell you,” laughs Baillie, “that every single building in New York looks different – there’s very little re-use from building to building. So for the most part those are all hand-built. We did have some procedural weathering systems, but they were mostly built and textured bespoke-ly. Then we could take all those assets and bring them into Katana which is our primary lighting package and have Katana manage the immense amount of data we generated. When V-Ray went to render it all, the renderer working in concert with Katana gave us a level of efficiency from an IO and memory management stand-point that we needed in order to get it done. It really spoke volumes to how great Katana is as a tool to manage that and how good V-Ray is in handling the tons of stuff that we threw at it.”
That process was also assisted by custom development work Atomic Fiction had invested into its own cloud rendering tool, Conductor. “At the end of the show,” recounts Baillie, “we ended up pushing 9.1 million hours worth of rendering through Conductor – that’s over a thousand years on a single processor. In the course of doing so, we calculated that we saved about 50 per cent versus what it would have cost us to build and maintain local infrastructure over the course of the show to do that. If it hadn’t been for cloud rendering, we wouldn’t have been able to do this level of work and we would’ve had to drop the quality level significantly or we would have had to ask Bob to cut four minutes of the hero sequence out of the film – neither of which are really acceptable compromises. So it’s a good example of technology helping to hit the economic target that we needed.”
Since there was not a great deal of time to complete the work, Baillie rejected an R&D effort to build a traffic system to fill out the world. “However,” he says, “we had a really talented and driven animator go and animate all the cars in the streets below doing all the right things. We created a 2,000 frame long traffic cycle that we used. It was the same with the layout of the props and traffic lights and hot dog stands in the streets – they were all for the most part set dressed by hand. So there was a lot of hand work gone into building that stuff.”
Some elements were not rendered in 3D, such as smoke plumes. “Instead,” describes Baillie, “we had a NUKE 3D system where we placed smoke stacks as cards down below. Same with the crowds and birds – they were all developed as NUKE 3D particles systems that were driven off of curves drawn by artists on the ground.”
Selling the height and the vast expanse of the New York area was also an important part of the city build – for this, Baillie looked to bringing in just the right level of haze from shot to shot, and found an elegant solution from comp lead Jed Smith. “He developed this really cool system that had a 3D plane on the ground that had a color ramp in it and then there was an expression that took whatever depth the haze was, it would take the color from the 3D card on the ground and propagate that up in Y from the ground so we could not only change the amount of haze as it went into depth but we could also change the color of the haze at any depth along the way.”
Back to the top.
The walk in 3D
Amongst all of this visual effects work was another central aspect to the film – stereo. Zemeckis leaned heavily on the 3D side of the final image to help immerse the audience in Petit’s world, both between the towers and in many other moments during the film. To do this, the director actually shot in mono but, says Baillie, ‘for’ stereo. “I think that’s a very important distinction to make, and we had several meetings early on to make sure that everybody from Bob through to set construction were clear on what the rules were on 3D for this film. Bob really respects 3D as a tool and wanted to make sure we got it right, even though we were doing a conversion, that he got the filmmaking and lighting and focus, and what compositions worked. He also had to consider how do I cut the movie, he was thinking about editing while shooting.”
Legened3D handled the conversion work, taking packages of final comps from the VFX vendors and layering them together within the desired stereo budget. “Interestingly,” notes Baillie, “there are only 826 shots in the movie in total. Some blockbuster films have 2200 visual effects shots. But Bob wanted to to make sure that for the 3D to work he would give the audience time to take in the environment and feel it, which works better in stereo.”
“After this project,” adds Baillie, “I am really convinced 3D conversion is the best way to go over native. The reason is, it makes shooting on set a lot easier, as long as you’re keeping rules of 3D in mind. Also, tools for 3D conversion are so amazing now that we could sit in a theater and make decisions in realtime about how deep a scene should be. You could say, this scene feels too deep, the guy in foreground is bugging me – and then you could dial that back in realtime. But in native 3D you are sometimes stuck a little to what the camera has shot.”
Back to the top.
The fin(al) line
Although this article concentrates on the main wire sequence, The Walk also included significant visual effects contributions from two other vendors – Rodeo FX and UPP. Rodeo delivered effects for several Twin Tower ground level scenes that had been filmed only on partial sets, as well as the shots of Petit narrating the film upon the torch of the Statue of Liberty. UPP handled Paris sequences that were filmed in old Montreal, Petit’s early wire performance at a park in France (filmed almost entirely on greenscreen), and his walk between the towers of the Notre Dame cathedral (again a greenscreen stunt).
For Baillie, the show was an important one for his facility Atomic Fiction which was founded in 2010 and which opened a Montreal office during production (to add to the Oakland and LA locations). It also brought the supervisor a surprising new appreciation of the legacy of the Twin Towers and of Petit’s accomplishment in 1974, which the supervisor found he could directly apply to the visual effects work – something Baillie realized only after the helicopter plate shoot above New York.
“During that shoot we had special permission to hover 1,400 feet right above where the WTC memorials are. We hovered into location and I looked down out of the helicopter and I was just absolutely overcome with this amazing feeling of complete terror and inspirational awe. That this guy Philippe had been here in this exact place, without any safety gear walking on a wire half an inch thick. It was the most intense emotional experience that was completely unexpected. I’ll never forget that feeling and I could take that back to the team and make sure every shot we were doing for the film I could evaluate that against that feeling – does this shot make me feel that way that I felt back in the helicopter? If I hadn’t had that experience I don’t think I would have been able to make it as effective as it was.”
Back to the top.
All images and clips copyright 2015 Sony Pictures.