Nike World Cup: The Mill makes the future

In ‘Write the Future’, a World Cup themed 3-minute long Nike spot, director Alejandro González Iñárritu called on The Mill to produce 236 VFX shots featuring a CG stadium, Massive crowds and other effects, as a bunch of famous football and sporting stars do their thing. We talk to The Mill’s 3D lead Tom Bussell.

Click here to download this amazing World Cup TVC

10Jun/nike/nike3fxg: The stadium shots are pretty remarkable. How were they shot?

Bussell: From day one, we were never able to shoot in the right stadium, so it was always going to be a big 3D production. We were either shooting at Millwall or a small stadium in Madrid, but that was cancelled due to weather. So we had to shoot at an indoor tennis court, which made it harder to judge distances, as 10 feet away from the players was a big grey wall. The stadium was designed by us as a generic stadium, because we couldn’t use any famous ones – they didn’t have the rights. So we had to generate our own design that didn’t look anything like you’ll see during the World Cup. Apart from the stadium itself, we had to replace a lot of the grass too. At Millwall the grass was very fine and didn’t look like how a pitch should be. So 2D had to do a lot of clean-up on that. In sections featuring Drogba and especially Ronaldinho, that was tricky because it was all shot on astroturf and it looked like some kind of massive carpet, so we had to replace a lot of the ground as well. They were running along it and you could see seams.

The shoot was delayed and delayed because of weather and player availability. So it was getting pushed and pushed which meant that the edit was getting pushed. Instead of having seven clear weeks with the edit, we only had, in the end, four. And we were always adding more shots in. We predicted in the beginning that there may be 80 shots of the CG stadium, but in the end there were 110. I lost count at one point! The Mill as a company has never tried to do something like this in this amount of time, but what made it possible was the amount of people we had on it and that it was all in one stadium. We’ve done previous Nike spots where there were five or six different stadiums, and I just don’t see how we could have done it in time. It was one model with one lighting setup which made it possible.

fxg: What were some of the things done while shooting that helped you later on?

Bussell: Well, we had one of the best DPs around – Emmanuel Lubezki. He was the DP on Children of Men. I think they did everything I wouldn’t have wanted them to do – zooms, hand-held, four cameras at one time. But if you try and tell a DP of that level, ‘Oh, can you not shoot like that, can you do this?’ they will just not listen to you. It was just a total logistical nightmare, even though it ends up looking great. Usually you’ve got at max two cameras to worry about and sometimes just one, if you’re lucky. I’d usually have someone as an assistant on set and we’d stick to a camera each and take all the data, but that just wasn’t possible here because they were zooming in and out on the four cameras. So we had to get a script supervisor, which you only tend to do on a Hollywood production to take all the lens details down.

We always had about a two day prep before each shoot. I just absolutely littered the set with tracking markers everywhere so that we would always know the distance between two points. Then we took a Spheron with us, which is basically a camera that does a full 360 degree photo of the inside of wherever you’re at. It takes an image from every pixel, and it’s HDR as well so that allows you to correctly re-light things back in 3D. It’s great for us to model with as you end up with a great undistorted hi-res image wherever you were. We could even use that for texturing if we were re-creating that actual environment.

Most of the time we were able to move our lights around in the stadium, but one of the trickiest parts was that the indoor lights for the shoot were at a certain height because of space limitations. So some of them were quite close. And we had to keep those lights in because if there’s a big flare behind a player’s head, you can’t just paint that out. So we had to make sure there was a 3D light wherever a real player crossed the light. It was tricky because obviously our stadium is one design and the practical lights were at a different height. We had to move the stadiums around and cheat a couple of times, but hopefully you don’t really notice in the final spot. That’s one of the things that sells the spot, the fact that they’re crossing these lights. A lot of the lens flares are taken from the live action itself.

10Jun/nike/nike2 fxg: Was any of the action shot against bluescreen or greenscreen?

Bussell: We actually asked for grey if we could. The reason why we ask for grey is that it’s just neutral. Usually it’s easier for the art department to achieve a decent grey, rather than green. You have to light a green in the right way and make sure the distance is right. We couldn’t have that, really, and we didn’t want to slow the production down. It did mean roto’ing everybody, but that was going to always be the case anyway, since we were replacing the ground. And 2D had to key the shadows back in. So all those things that really are a nightmare to do, we had to do on every shot. In a lot of the shots, the background players were very out of focus, so they didn’t have a lot to roto with. It’s quite tricky to roto things like that without a defined line and it brings a lot of the background into it. All those kinds of invisible things were days and days of work.

fxg: Did you have a stadium designed by the time the shooting took place that you could show to the director and DP?

Bussell: We took a stadium from a previous job, ‘The Next Level’, because the stadium in that spot was also a generic one. We changed the roof and a few details. We moved the exits and the lights. We re-lit it and re-textured it to fit the live action. It is quite different in the end. Then we built a 3D lighting rig to match the live action. There were a few tricky shots where the players had to be, say, on the sideline because that’s where the action took place, but then after putting our stadium in the lights didn’t line up with our lights so there’s a few tricky things we had to do by changing the height. I took some of the video assist/playback shots back to the hotel room and mocked them up for the agency and the director just to show them what it might look like. It was especially tricky for the DP because he was framing for the characters but not necessarily the backgrounds. I’m really happy with the result because it doesn’t necessarily look like it was shot in a studio, which can so often be the case. During the shoot, there was actually a lot of nervous laughing. The DP would say to me, ‘Every time I look around you’re either laughing or looking nervous,’ which I was!

fxg: I think the spot benefits from some pretty frenetic camera moves and fast editing. How did that affect your effects shots?

Bussell: At the time, when I saw them zooming in and out and going hand-held, I was so scared. They were trying some crazy things. They made this makeshift stabilizer that consisted of two pipes and a man hanging from a swing and four people running along and someone with a camera dangling in the middle. They were trying everything to make it look as interesting as possible, and I was at that point pretty much freaking out. But I eventually conceded that it was going to be like this for every shot. As a result, I think it is really interesting looking. It looks very dynamic and the edit is very good.

fxg: Let’s talk about the CG crowds which I think were done in Massive. How did you approach that part of the effects work?

Bussell: On the set, I was a bit nervous about the number of Massive shots planned. So I took lots of video of extras they employed as crowds. In the end we didn’t use one single crowd plate – every crowd shot is CG. However, I did shoot some people against greenscreen, because I thought if we fail to get the Massive shots done we can do it in the old-school Gladiator way and put people on cards. In the end we really pushed up our Massive capabilities. When we first came back to The Mill, we realised that the shots were so much closer to the crowd than we’ve ever done, and the Massive setup we had wasn’t holding up. It just looked CG and on previous jobs it was a wide shot and you were far away. But we had to ramp it up here. I decided we needed to update the mesh in Massive, which hadn’t been updated since we started using it seven years ago. We re-sculpted the generic Massive person, re-rigged it and then added lots of variation. We realised at that point that there had never been a female supporter. We ended up doing CG women with long hair as well, which was quite tricky.

The mesh had been quite low-res, so when we put the improved texture on we actually got better shadows. We got the Photoshop guys and the Massive guys to re-texture all of the characters, maybe 100 different variations of people based on photos I’d taken from the set. Instead of painting the shadows out of the textures, we left them in. Even if technically it wasn’t correct, they all had more detail in them. The lighting might have been from the wrong angle, but it just look better and more realistic. Before we’d do normals passes and all that sort of thing, but over 110 shots I realised we were not going to finish this thing, so we did a beauty and a shadow pass only. So we had to improve the look and simplify things.

We had two weeks to do that before having to start rendering all the shots. We didn’t actually have anything to show for two weeks – not one single shot – and everyone was getting a bit nervous. The clients were happy to let us get on with it, but the 2D operators wanted to get into their stuff. So in parallel with doing the stadium render, we actually put people on cards and populated the whole stadium. It meant the compers could do a pre-comp with a 2D crowd. They were on flat grids and were alright for some shots although in others they looked like cut-out people. From a compositors’ point of view, it was enough for them to get a feel for what the final shots would look like. I didn’t know whether we’d be able to use Massive renders for every shot because of the timing. So it was good we did both, so the compositors had a good feel for it and it meant they could just replace that layer when the Massive renders were done. It meant they could start working straight away without 3D holding them up.

fxg: The crowds also have a bunch of banners and other things moving around. Can you talk about that?

Bussell: We didn’t get to that actually until about two weeks before the deadline. We’d been so preoccupied with the crowd and the stadium, but something wasn’t quite right. Something was missing. We’d be putting an extreme amount of movement into the crowd, almost exaggerating it. If you look at crowds on TV they look like a flat texture, and you don’t see hardly any movement. But for this spot, that just wasn’t good enough. It just felt like no matter how excited the crowd were – some of them were jumping around like lunatics instead of standing around still like a real football match – it still didn’t look right. So then it just occurred to us to add the extra banners. Obviously normally we do flags but usually they are something you shouldn’t feel too much and they’re just an extra detail. But we went for it on this because it gives the shots so much more life.

Usually we would do banners and flags in Massive. You’d get a Massive agent waving a flag, but we didn’t want to put that extra workload on the artists. So in Maya we did some cycles of banners and flags, designing them with the agency to avoid any legal issues like using names or brands. Actually, we couldn’t even use the St George’s Cross flag, so we designed a flag that kind of looked like the St George’s Cross but wasn’t. It took a long time to design something that was legally acceptable but also looked good. We did all the flags in 3D, simulated in Maya and then rendered through XSI. We’ve got proprietary tools in order for XSI and Maya to work together.

fxg: You mentioned the lights in the stadium and I liked how the player’s faces seem to be often enveloped by the lights. How much work was involved there?

Bussell: We tried a few things in 2D to re-create the look of the lights. One of the edicts from the DP was that he hates blue sky and hates shooting things apart from backlit. He worked on Sleepy Hollow so he certainly shoots things in a particular way. Whatever we did, we had to make it look like what he shot. We couldn’t make it look like all of sudden it was a nice sunny place. The spot wasn’t actually lit like a World Cup final, it’s actually a dark, moody place. The Rooney section is kind of a dark place but it’s a style that gets you away from feeling it was an ad for a soft drink commercial. It feels quite gritty and intimidating.

For me, it was important knowing this and matching the lights as best we could. The lens flares from the real camera were such that you wouldn’t be able to make them up. In the end, none of them are real except for what appears over a player’s face or body. We couldn’t key them off the background because they’re so subtle – the gradient and the bloom. What we had to do was re-create all the lights in 2D and 3D. There’s no easy way to do that. It’s quite a manual job. Lens flares are almost mistakes of the lens and it’s not something that software can re-create, so we did them all by hand. We had great reference of course, but we had to re-do them. Rather than give the compositors a matte and then apply a 2D effect, which we would usually do, I actually went through the whole edit before any effects were added and painted some of the lens flares out. So we had a lens flare as it should look, went into Photoshop and put it on black and then, using a particle system, put an image of a lens flare where the light should be in our 3D space. Then we animated them and scaled them. So 2D would have something that wasn’t just a square matte and actually looked quite decent. They added some 2D effects on top like blur or a few glints. It worked great in the end, but it was another thing that we had to invent to match the DP’s vision. He shot it into the light with those lens flares for a reason. He wouldn’t have been happy if we had just put some cheap 2D effects on top. It’s one of the things that helps bed it in.

One of the biggest ones was when Rooney stands up after the tackle and his whole head goes in front of the lights. The flares over his head are real and we just had to make sure that the ones in the background looked as real as the ones that were shot. Even if the client wanted us too, we wouldn’t have been able to paint them out because they go over his face. Then we also added camera flashes and tickertape and a few things like toilet rolls being thrown around for good measure.

10Jun/nike/nike1fxg: At one point Ronaldo is seen in front of his own stadium. How was that shot accomplished?

Bussell: For that, we took a generic stadium we’d done in the past, and did a render. We took our existing 3D and gave it to one of our matte painters to design a more modern look by adding different shapes and glass. We did the text in 3D. It was a fair bit of work for 20 frames out of a three minute spot. And then before that stadium was finalised as a shot – it had actually originally been the outside of the stadium that was going to feature – it got changed. We showed the director and he just wasn’t happy. It wasn’t the work we had done, but he said something along the lines of, ‘It feels like a shopping centre and I just wouldn’t have shot it like this. I would shoot it from the inside of the stadium, because then you know it’s a stadium.’ That was about four days before the deadline. We were like, ‘Oh, OK’. To be fair, the agency said, ‘Look, you don’t need to do this.’ But everyone wanted it to look great. So we had a crisis meeting to work out what the hell we could do. We dug out a stadium we’d done before and slightly bastardised it to what is now Christiano Ronaldo Stadium.

fxg: What kind of effects make up the rest of the spot?

Bussell: We outsourced to our New York office some of the other shots, like the satellite view. We ended up comping the satellite over here because the clients were in and it was easier to manage that. We had some European Space Agency footage but we couldn’t use it, so we had to re-create the shot. We used a satellite we’d done for Enemy of the State, improved it and then did a matte painting for Africa. There’s some matte painting work to show where Rooney has been banished and is doing things like line-marking. There were some trees in the background plate and, well, the director doesn’t like trees. We created a bleak, urban factory setting, somewhere you wouldn’t want to end up. I never quite got to the bottom of why he didn’t like trees, but he thought they were unlucky. He’s only ever had one tree in his previous films and he said it was unlucky. I didn’t really question him.

The shots where Rooney is being knighted by the Queen involved a greenscreen shoot. For the baby section there were only four cots and the rest is plated up so it looks like there is an infinite corridor. For the chat show footage featuring Cannavaro, that set happened to be the most reflective I have ever been on, so it was a bit of nightmare when I saw the footage with all of the greenscreen reflections in every single panel. The ping pong match between Rooney and Federer – obviously we couldn’t get them in the same room at the same time. So Rooney was on a greenscreen and Federer was on that location. It’s things like that that you don’t even notice because you’re just looking at the football section. We had to change some ridiculous things, too, because it went out to so many different countries. Every country had a particular legal issue like the writing on a sign or what-not. We managed to clock something like 300 versions for specific needs.

Probably the biggest amount of additional work was the statue square. That was shot for real but there were probably only 100 extras in the square. That was another Massive crowd job with umbrellas, too. The director was a bit underwhelmed when he saw it in the edit because there were only a few people surrounding the statue. We put additional flags and people around. The only bit we didn’t really touch was The Simpsons bit, so that was 20 frames we didn’t have to worry about!

fxg: What was your compositing pipeline?

Bussell: Our final comps were done in Flame. We really made it as simple as possible for the compers. The lead op was Neil Davies, who was involved in the shoot as well. We would do a matte painting for each section to get a style frame. Then Neil would take that style frame and try and re-create it in Flame until we got something that we were happy with. He would give that setup to the rest of the guys and use that as the base template. It needed to be the same look on every shot. Every time they plugged in the same layers, within 20 minutes you could get something looking fairly decent. This meant that two weeks before the end, we’d actually done every single shot. So we could go back and finesse everything. What we were worried about was getting 20 shots looking really good but then 80 shots not working. They just blasted through every shot spending half a day or a day on each one which is not much time at all considering all the work that went into them. That doesn’t include the grass or the roto – that just includes getting all the layers together and doing a final comp. We just wanted to make sure it was deliverable and then we could worry whether it looked good. It was four senior Flame artists, which you don’t ever get on a job, but it had to be that way. It was great for them because they were able to work with each and other and give each other ideas. Each op took a main section, and they are quite different styles. The Ronaldo section was actually shot by a different director and DP, so it was always going to look different. Each has their own style and character.

fxg: At three minutes, it’s actually a pretty bold spot. What was the hardest part of the work?

Bussell: Well, 3D wise, once you got one shot right, it was pretty much smooth sailing, but for 2D there would be something in every shot that would be really tricky – whether it was a flare behind the head or replacing the whole ground and putting the shadows back on. There were just so many shots, plus we only had the four weeks to finish it, so I’m not sure when a project like this will come around again anytime soon.


Agency: Wieden Kennedy, Amsterdam
Creatives: Mark Bernath, Eric Quennoy, Stuart Harkness, Freddie Powell
Producer: Elissa Singstock, Olivier Klonhammer
Production Company: Anonymous / Independent
Director: Alejandro González Iñárritu
DP: Emmanuel Lubezki aka Chivo
Editing Company: Work Post
Editor: Rich Orrick, Ben Jordan
Post Production: The Mill London, The Mill NY

The Mill London Team:

Exec Producer: Stephen Venning
Producer: Matt Williams
Line Producer: Allison Cain
Shoot Supervisors: Neil Davies, Tom Bussell, Austen Humphries, Vince Baertsoen, Des Anwar
Scheduler: Colette Boyle
Colourists: Seamus O’Kane, James Bamford, Aubrey Woodiwiss
Lead 2D: Neil Davies
Lead Flame: Neil Davies
Flame: Rich Roberts, Grant Connor, Pete Rypstra
Flame Assist: Leon Woods, Danny Morris, Paul Downes, John Price, Ben Turner, Zoe Cassey Hayes
Shake: Darren Christie, Vanessa duQuesnay, Donal Nolan
Smoke: JP
Lead 3D: Tom Bussell
3D: Juan Brockhurst, Edward Hicks, Dan Adams, Daniel Jahnel, Amy Macdonald, Sergio Xisto, Eva Kuehlmann
Tracking: Escape Studios
Rotoscope Assists: Escape Studios
Matt Painting: Dave Gibbons, Yigit Sanalan
After Effects: Pete Mellor

The Mill NY Team:

Producers: Dan Roberts, Boo Wong
Lead 2D: Westley Sarokin
2D: Corey Brown, Suzanne Dyer, Rosalind Paradis, Iwan Zwarts, Jade Kim, Gigi Ng, Robert Bruce, Randy Krueger, Brendan O’Neil, Albert Cook, Alex Lovejoy, Melissa Graff, Tristan Wake, Jeff Robins, Keith Sullivan, Claudia D’Enjoy
Lead 3D: Ben Smith
3D: Wyatt Savarese, Chris Bernier, Ajit Menon