“It was the most tracking and compositing that I’ve ever done in my last 17 years in visual effects.”
– Janelle Croshaw, visual effects supervisor
Brazilian soccer legend Neymar Jr. stars in the VR experience ‘The Neymar Jr. Effect’ for Nike’s line of Hypervenom Phantom II soccer cleats. Watched from the player’s POV as he shoots for a goal, the experience – which even includes a streaker – was brought to life by IM360, a joint venture between Digital Domain and Immersive Media, and directed by Smuggler’s Adam Berg for agency Wieden+Kennedy. Find out how the work was accomplished using a 360 head-rig, wirecam and a ton of digital artistry. And check out our exclusive behind the scenes video.
fxg: How did you get started with a massive project like this?
Chris Fieldhouse (VFX producer): We had lots of things to solve. Is it going to be POV? Is it going to be third person? How do we actually execute this? Adam Berg the director had never done anything in 360 before. So we all collectively jumped in and figured out how to make this work. It was a very collaborative process. We were able to help drive a bit of the creative as well as the technical side of how to do it. So it was good to be involve from an early stage.
fxg: What were the initial challenges in working out what the viewer would see in the experience?
Janelle Croshaw (VFX supervisor): One of the first steps was doing previs. Because we’d done Evolution of Verse which was all-CG and the Bjork VR spot which was all-practical, we had our VR 360 going in both of those directions. We were able to render full 360 previs which was really helpful. Adam had an idea for the choreography and the goal run that he wanted Neymar to take, and the previs helped a lot in the initial envisioning of it.
Fieldhouse: We had to work out was it going to be all-greenscreen, or in a stadium? What kind of cameras were we going to use and were we going to shoot in 2D and covert it to 3D, or shoot it in 3D native? It ended up being a real hybrid of all of those. We also had a football coordinator who helped us plan out all the moves, because obviously Nike wanted to make sure this was as realistic to real football action as possible.
fxg: Can you talk more about the 3D previs?
Croshaw: We had our previs artists use bowling pins to start out to get the rough choreography and then render it out in 3D so Adam could watch in the headset. We realized early on that that was a good way to figure out the planning. The previs team then grew and turned into more of helping in practicing the moves.
Fieldhouse: There were a lot of funny movements of various people in the office going outside and kicking a ball around wearing a 360 head rig to figure out what some of these actual moves might look like! We approached it from every angle.
Croshaw: Even figuring out the head rig in general – it was definitely trial and error. We tried out quite a few different head rigs and would head out to the soccer field and those are some pretty funny tests. We had to figure out how to stitch this stuff. Should the head rig be a unicorn rig, with the six cameras in the front or should it be an actual head-band rig with the cameras all around the head. What’s going to give us the highest resolution and be the easiest to stitch. We still had hot glue going on the day of the shoot!
fxg: What did you settle on in terms of the rigs used?
Fieldhouse: We wanted to make sure the final film was of the highest quality as possible – we didn’t just want to go and shoot this thing on GoPros and call it a day. We spent many many months planning this shoot out and figuring out what the best combinations of cameras setups and rigs were. We ended up with a hybrid between 2D plates shot on an ALEXA rig and a 360 array that we shot on GoPro as well as Codex cameras. We also used a wirecam rig that was hung over the whole stage. We also had multiple witness cameras surrounding the stage that were REDs.
Croshaw: We had 15 cameras in total – the ALEXA, Codexes, three RED EPICs, nine GoPros.
Fieldhouse: There was a 360 camera array was mounted on the primary wirecam rig. There was a 360 camera array mounted on Neymar for his POV. It was a pretty massive shoot – three days of shoot and two days of prep.
fxg: Did you have a way of looking at the footage you were getting on set and doing a quick stitch to see if it was working?
Croshaw: We did. IM360 was awesome. We did Neymar’s run with the head rig because that had to drive everything. We did lots of takes of that. It wasn’t instantaneous, but the IM360 guys stitched within minutes and we’d put it on the headset and right then and there we picked select hero takes of him doing the goal run with all of the hero players, and we were able to get buy-offs from Adam and the agency right there. Then the challenge was to get the wirecam to match what Neymar did as close as we could.
Fieldhouse: That was one of the big technical challenges that we faced, was being able to line up all these various plates we were shooting. We were shooting on 360 rigs, then on 2D cameras. The ability to shoot a 360 take of Neymar and then from the wirecam and go and quickly stitch that in a rough form and look at it with Adam and everyone on set was really invaluable.
IM360 has the VR Toolkit – it’s an instantaneous live preview so you can look at it on a monitor while you’re shooting and while you’re lining things. We wanted to use that but because we had Neymar running around on a football pitch, he couldn’t be wired so we weren’t able to do it realtime, but we were able to pretty quickly – within minutes – look at rough stitched footage.
fxg: Where did you start in terms of co-ordinating the plates and CG work?
Croshaw: It was a massive project. It was the most tracking and compositing that I’ve ever done in my last 17 years in visual effects. It was invaluable having the feature film tracking pipeline here. We ended up having so many passes with the wirecam rig. Once we got the wirecam it was basically like poor-man’s motion control to shoot all of our different passes.
Bringing that into post, it was our first priority to do a rough assembly, so first we had to track that wirecam rig, which we could do pretty easily with all our witness cameras. But then the tricky part was reconciling with the headcam – we needed Neymar’s body. You couldn’t actually use the 360 that was on Neymar’s head for the entire spot because the GoPro’s weren’t high resolution enough and he was just bouncing around too much. That’s where a headcam on a pro soccer player really will make you sick if you’re looking out. It’s fine if you’re looking down at his feet but the minute you try to look anywhere else you’re going to get really bad motion sickness.
So we moved more towards the wirecam which had smoothed out motion. And what we ended up with was a hybrid. So every single one of those players was their own composite that lasted 90 seconds at 60 fps. It was a lot of frames. At one point I had them all calculated but I’ve since put it out of my memory to some extent!
Once we locked down our base assembly, it was all in slow motion so it was important to get that over to editorial for Adam and the editor to get the re-times and the ramps in. It was designed to have this flow where you would go fast and then you would go slow motion into some of these signature moves. We really had to lock those down to be able to add in the effects on top.
Once we had that locked down then we started adding in the CG. During that whole time we were doing asset dev of the stadium. The crowds were shot on a second stage during the shoot. We had 50 extras and on movies like Real Steel and Rock of Ages we would set up a two-and-a-half-D NUKE crowd pipeline system, where you put each person on a card and you populate the stadium with them. That was the plan here, but when you’re doing 360 all of a sudden everything is just exponentially bigger and it just crashes and breaks. So we ended up taking the card system into Houdini and doing the crowds in Houdini but using the crowds we’d shot on the stage.
There’s a lot of details in there – including a streaker! All of the people – the players, the coach, referees, the crowds are all practical live action. The environments, the blimp and all the effects and the grass are CG. The ball is a hybrid – only when we needed to fill in moments where it was too motion blurred it was CG. We even filled in Neymar with a CG Neymar in areas where we thought the GoPro wasn’t holding up very well.
fxg: You mentioned motion sickness, how did you ensure where the viewer would be looking in the right place and then also limiting the amount of movement?
Croshaw: This was a tricky one, we talked about even having a custom app that limits your viewing so you can’t turn around in 360. But in the spot you actually can turn around and look behind you and see your backside running. But also because you’re moving forward, the viewer wants to look forward. It was a conscious decision to let the viewer look around.
Fieldhouse: It was part of our testing phase, we did a lot of previs where you’d see what you’d actually get if it was just a camera on someone’s head and you’re bouncing around playing football. That’s why we went down the road of using a wirecam and picking a smoother motion and combining those two for a smoother path. It’s funny because we all thought it was going to be something that was just on the edge of too much motion, then quite late in the process the client and everyone came back and said hey we should add more shake into it. So we ended up doing a really smooth path for technical reasons and then towards the end we added in some of that natural motion you’d get from an actual soccer player running down the field.
If you’re watching this and are supposed to be the POV of Neymar then obviously your focus is going to be on the ball anyway. But of course this being 360 we wanted to make sure if you weren’t looking at the ball there was fun stuff to look at, which is why we put in the streaker and the blimp in the air.
Croshaw: Even the crowds, in the original treatment they were supposed to be very dark and almost non-existent in the 360 because we knew how hard it was going to be. But it ended up being a little boring, so we turned on the lights and just gave the viewer more things to look at.
fxg: How did you deal with quick views of his hands and legs?
Croshaw: It’s mostly the live action.
Fieldhouse: That was actually the biggest concern on our part going on, knowing that in an ideal world, we’d have a 360 head rig going in that had the same quality as an ALEXA but in the world we live in at the moment we couldn’t put six ALEXAs on someone’s head and expect them to run and play soccer, so we were limited to the cameras we head. We knew that with the fast action and the motion blur and the rolling shutter and all that, there was going to have to be a little bit of give and take on that. Most of what you see is the captured live action footage, augmented to a degree in 2D. We also did build a full CG Neymar body double and we used a fair amount of that to bring back detail where needed, especially with the shoes – it is a shoe commercial at the end of the day.
Croshaw: The shoes ended up being the most CG in the spot, just because obviously the client wouldn’t want the Nike shoes to be blurry.
fxg: What was your comp solution?
Croshaw: Aruna Inversin was our comp supervisor on this and Ross MacKenzie our tracking supervisor – they came up with so many extraordinary solutions to make this project work, as did CG supervisor Ron Herbst, so it could even be rendered. We’ve constantly been, with every VR job that we do, making custom tools and passing that along to the next project. But every project seems to be new and challenging in its own way and so I know that they were coming up with a lot of custom tools to visualize and push it through.
fxg: Anything else you wanted to mention?
Fieldhouse: One of the other challenges was that originally they came to us for our VR knowledge, but originally it was pitched by the agency as a 2D spot and then because of all the market shift to VR, they suggested it now be 360. But Nike and the agency also needed a TV commercial out of it. So we ended up shooting all at one time and created a traditional 2D commercial and short 5 sec teasers out of the VR piece we shot.