Knight & Day

For James Mangold’s Knight & Day, vfx supervisor Eric Durst oversaw more than 700 effects shots to help tell the story of a secret agent and his unwilling partner pursued around the world. We delve into the digital Running of the Bulls work by Rhythm & Hues and the European train effects by Weta Digital.

Rhythm’s running of the bulls

Perhaps the most memorable sequence in the film features the two lead characters, Roy Miller (Tom Cruise) and June Havens (Cameron Diaz), on a motorcycle being pursued on Pablona streets and in a bull ring by bad guys in Smart cars and by running bulls. Production shot the scenes in both Seville and Cadiz, using real bulls and young runners, with several pickups done at Universal Studios in Los Angeles. Rhythm & Hues provided set extensions, composites and digital bulls under the supervision of Greg Steele.

Plate Photography
Final shot

For one bull run shot, Tom Cruise and Cameron Diaz performed much of the stunt themselves. “On the day Tom and Cameron came,” explained Durst, “we thought Tom was just going to be on the bike and go down the street with the runners and not the bulls and we would add CG bulls. Tom really wanted to do the scene live with real bulls. Everyone said ‘Oh, OK’. So they had two chutes – one where they’re keeping the bulls and one with the motorbike and the cars behind them. We opened the gates and the bulls headed off and then the next gate opens and here comes Tom and Cameron, no helmets, and then Smart cars behind them.”

Rhythm augmented the bull running shots and later shots in the bull ring with digital bulls. For reference, artists spent time in a test bull ring in Seville videotaping the animals. “The look is one thing you can actually nail from stills,” said Steele. “But I think the thing that always destracts and looks funny with CG animals is the motion. So we had to get that right. We couldn’t mocap these guys – the trainers wouldn’t put markers on them either because they’re so crazy and wild.” In order to acquire suitable motion reference, then, Rhythm set up and triangulated three HD cameras around the test ring and had four or five bulls do different actions. Similar reference footage was shot of bulls going back and forth specifically for the bull run shots.

Plate Photography
Digital Bull

Final shot

The footage was tracked and roto-ed by animators from the different cameras to produce vignette pieces of the bulls in motion. “When you’re stepping through the animation, it actually looks a little odd,” noted Steele. “You’d probably do things differently if you were hand-keying the material, in terms of the speed it goes from one position to another and the angle the legs are moving. When it all comes together it all looks correct. A lot of times you introduce what you think is right and it can become this odd-looking thing with the weight being wrong. We also noticed these bulls just can’t move on a dime, especially on these cobblestone streets.”

“However, it wasn’t just us tracking dots and doing that roto-mation mechanically,” continued Steele. “We really needed animators to go in there because there was so much muscle and meat in there under the bones and muscle structure. To get the right pose we needed to move the right bones around, so it wasn’t just the surface of the skin moving, it was the structure of the bones moving as well.”

Artists produced a Zbrush macquette of a bull, relying on one particular reference animal that had significant musculature, and then continued modelling in Maya. This model was brought into Rhythm’s proprietary Voodoo software for animation, grooming and skin creation. “We animated with a low-res model but we ended up using a hi-res mesh as the final thing that got simulated in the end, from the Zbrush macquette that was built,” said Steele. “Even though we put fur on these things, the fur was really short, so it was important to have a lot of detail with the undulations of the bone structure in the head and face and the way the ribs poked through – just all the details. Having all the extra things in there helped with the skin too which would fold and wrinkle in on itself.”

The reference footage also helped with simulating bull muscles, a system set up by CG supervisor Matt Derksen. “We actually built bones, built all the proper muscles, set up drive pose-based deformations on each of these muscles as the bones would move,” recalled Steele. “On top of that we would move masses of muscles around as the beast was moving. As it comes to a stop, the belly swings and you’re getting all this shoulder movement separate and on top of the muscles firing, which added a lot of weight to it.”

Plate Photography
Digital additions

Final shot

Even further motion on the bulls, brought about by the appearance of loose skin on the beasts seen in slow motion replays of the reference footage, was important to maintain a realistic look. “We were able to get this in as a high resolution mesh on the final simulation which gives it a really nice slide and extra detail,” said Steele. “The skin would literally bunch up in areas because it was such a hi-res mesh, but it looked really accurate.”

From stills taken of the bulls, Steele also noticed a strange waddle under the bulls’ necks that needed to be replicated, as well as drapey skin and fur. “The fur had interesting patterns in it and swirls across the body,” said Steele. “The director also wanted them quite dirty and messy. Any kind of cleanliness on the bulls we had to remove. We came up with some techniques for adding mud into the fur, so we put mud into the texture maps. We also used our hair tool to create mud chunks for their ankles and backs. It took a little of the CG off them.”

“Bull fur is actually really reflective,” added Steele. “Normally we would add CG lights to give a broader specular type of light, but what we were noticing was that in the photos of the bulls in shadowy areas, you could literally see the the bull ring reflected in their fur and even people. So we ended up using ray-tracing on all of our renders for the bulls to get reflections on the bulls and environments and other bulls. It really got the sheen right.” The bulls were rendered using Rhythm’s Wren system and composited with their in-house node-based compositing software called Icy.

The unfinished Stadium
Final shot from the film

For plates featuring Rhythm’s digital bulls or requiring set extensions or augmentations, Steele and his crew took numerous reference DSLR stills. “We used that in conjunction with our HDRIs to create environments that the bulls would run,” said Steele. “We have a little box called a HDReye, a little cube that has a camera on each side that we developed about 5 years ago. It takes bracketed photography in about 30 seconds to give out a whole HDRI. Because the environments were so diffuse on this show, it worked out well in terms of not having a 4K HDRI. It was more about 2K.” Standard gray and mirror balls were also used, as well a mock-up of a bull with horns. “And of course we had the real bulls in the shots,” added Steele. “For about 3 shots, we augmented the shots with CG bulls so they were right next to real ones. It was really huge just to have that. How dark the bull skin gets in the shadows, the brightness on the horns, how much spec you get on the tip of the nose.”

In the bull ring, Roy and June are chased by the bulls and Smart cars before heading towards an exit. On set, a setup with real motorbike with stunt riders and real bulls almost led to calamity. “In one of the takes,” explained Durst, “the idea was that the motorbike is meant to exit and the doors shut and the bulls are meant to leave via another exit. This one time, the stunt doubles go to their exit and the bulls start to follow them. The guys are shutting the gate but they don’t get them closed in time and the bulls charge through the gate and jump over it and go into this very small tunnel where the motorbikers are. And we think they are going to be killed! They all went in and all of a sudden it’s complete silence.”

The car filmed for real
The Bulls needed to be fully digital

To get accurate Bulls digital muscles were modeled
The digital muscles

The hair pass
The final shot

“What happened was they thought they were doing a clean exit and then right behind them were the bulls with no real exit. So they ended up putting the bike down and jumping up on a bunch of scaffolding and then the bulls trying to get after them. Luckily, the safety guys came in and cleared the bulls out. What’s cool about it is that on film, you actually see that happen – you see the bike goes through and the bulls go through. It leads very well to the next shot which is the motorbikers coming out of the bull ring and one of the other bad guys coming up perpendicular to the entrance to the bull ring.”

Only limited green screen was used
The final shot

That next shot shows Roy and June zipping past one of the bad guys when suddenly the bulls crash into a Smart car and kill the guy. “We shot this at Universal studios,” said Steele. “They yanked the car with a special effects rig and the first time they did it it went flying through the air and was a bit over the top. On the second take it was about half the force but still really wild. Our animation lead took the raw footage and the vignettes Rhythm had built up, adding additional motion over the top. The final shot was augmented with practical breaking glass elements.”

Rhythm also worked on recreating the bull ring stadium – which had been under construction at the time of shooting – with 3D crowds and added stories to Seville sets shot at Universal. Additional effects included a shot of Roy and June narrowly avoiding an electric train, face replacements, bullet hits, CG bullwhips, effects for the breaking of a huge wooden door, and a distant airplane explosion.

Eric Durst noted that Rhythm’s digital bulls were characteristic of the rest of the effects in the film. “In terms of the bulls and other shots, I think people recognise more and more the abilities of visual effects tools,” noted Durst. “It can work for you, because you know you can do CG bulls. But there’s always doubt until you see it. Once Rhythm showed an initial bull, people were blown away. And then at that point they want to add more bulls!”

Plate Photography
Train added for drama

Weta takes the train

Earlier shots of a train travelling through the Austrian Alps featured visual effects from Weta Digital, under the supervision of Charlie Tait. The sequence called for an alpine snowy landscape to be seen through a dining car window as Roy and June talk and then through a kitchen car window as a fight ensues, ending as one of the bad guys gets thrown outside. “It was shot in a warehouse in Spain with train cars and greenscreen,” said Durst. “You didn’t really have any sense of how elegant the sequence could be when you’re doing the photography. It could have been just normal trees going by or a pedestrian environment. But what Weta did was take it and make the snowy landscape come to life.”

Weta’s approach to the long shots – 139 in total – was to building a simple CG environment for the alpine setting and use projected matte paintings on various planes, with a sprite-based application that allowed artists to adjust tree locations and the speed at which they zoomed past the train. “We actually dug up some CG pine trees which were shader test renders, or turntables,” explained Tait. “We used the different frames of those as variants and we put them on cards in Nuke. We made a white snowy terrain and we put all the cards on there and then we had an access in Nuke that told us how fast the train was going past the trees and we had simple camera tracks inside the train that determined the rotation of the camera.”

Artists spread trees out along the tracks, either further in depth or spread further apart. “You could do about 30 trees per node,” said Tait. “So you’d put more and more of these nodes in and spread them around. We’d put more and more of them or spread them out more for depth. Behind all of that we had a matte painting of snow covered mountains. Essentially it was really simple, but it worked so well.”

Death by train
The Austrian environment

The card tree generator node made sure that the tree cards were always oriented towards camera so that a side of a card would not be seen as it went by. This effect made it look like the trees were rushing past the train. “For compositing we pulled very thin keys,” continued Tait. “We preserved all of the window detail and reflection. The shots didn’t really look like they’d been keyed, like some greenscreen shots can with harsh edges and obvious separation between foreground and background.”

Weta’s simple approach also meant that they could complete the shots in just five weeks with about 40 artists. “It came down to how much time we had,” said Tait. “We thought we’d need a 3D terrain with rolling hills, which would have been a much more involved task. And we could have done that, but we thought maybe the result would be so similar to what we ended up with.”

Other shots

A number of other facilities also contributed to Knight & Day’s visual effects. Soho vfx, under the supervision of Allan Magled, contributed crashing cars and effects for the drone attack on Jamaica. Hydraulx delivered the early plane crash sequence in the cornfield and some Salzburg hotel composites, supervised by Erik Liles. Other contributors included Eight VFX, Spin, FilmWorks FX, Pixomondo, Wildfire Studios, With a Twist and Big American, with Halon providing previs for the entire show.