Ready Player One is set in 2045, when people have the ability to enter into a digital universe called the Oasis. Once there, you can go wherever you want, do whatever you like, be whomever or whatever you choose to be. With the population beset by unemployment, poverty, overcrowding, and utter hopelessness, “it’s a good time to escape into a virtual world where you can live an extraordinary life through your avatar,” according to Director Steven Spielberg.
Through motion capture and computer animation, the actors not only portrayed their real-life roles, but were also able to bring their avatar counterparts to life. The process involved an intricate working alliance between Spielberg, his cast, production designer Adam Stockhausen, costume designer Kasia Walicka Maimone, and the two main visual effects houses, Industrial Light & Magic (ILM) and Digital Domain (DD).
As a rule of thumb, ILM did everything inside the Oasis and Digital Domain did the motion capture and the visual effects in the real world.
Spielberg employed different techniques to denote a visual difference between the virtual realm of the Oasis and our hero’s stark reality in the real world. As the action moved from the Oasis to the physical world, Spielberg switched from ‘filming’ digitally to shooting on 35mm film, working closely with cinematographer Janusz Kaminski to also mute the colour palette of the real world.
Everything inside the Oasis, from the avatars to the environments, was ultimately created by the ILM team, making ILM’s work not dissimilar to a fully animated mini-feature film. The production VFX supervisor was Roger Guyett and Grady Cofer was ILM’s dedicated VFX supervisor. The DD team which did the effects outside the Oasis, was lead by visual effects supervisor Matthew Butler. DD’s virtual production supervisor and producer was Gary Roberts who was responsible for obtaining the motion capture data and head-mounted camera footage, which was used by ILM and DD.
ILM: Inside the Oasis
ILM handled all the animated Oasis sequences in the film’s ‘VR World’. Close to 90 minutes of the film, (900+) shots) was inside the Oasis and done as 3D animation. The team worked with animation supervisor David Shirk, virtual production concept design supervisor Alex Jaeger and others, to evolve every aspect of the virtual characters, from their appearance and movement style to hairstyles, costumes, and the unique digital textures that are visible on the characters’ skin.
“I think it is amazing for a man, who has done as many films as he has in his career, to be this enthusiastic and energetic, – he loved our reviews – they would go on for hours every week” commented Guyett about working with the legendary Oscar winning director. On average, twice a week, Spielberg would spend hours working through shots at ILM and discussing both the technical and motivational aspects of each sequence. “He just seemed to enjoy the process, which was super creative and very collaborative” added Cofer.
While Spielberg is technical and very film literate, Guyett saw ILM’s job as removing the technology on this highly advanced film. “It was so interesting how keenly he concentrated on the performances of the actors” he adds. Spielberg has made several complex films of this type with digital characters, The Adventures of Tintin, The BFG, etc and so he had a strong plan on how he wanted to work with the actors in capturing their performances. One such aspect was that he wanted to get really strong video coverage of the performances. He wanted to go into editorial with more than just the technical facial or motion camera footage. This allowed Spielberg to pick great performances from more traditional camera footage. “This was driven by personal experience,” Guyett commented, adding somewhat jokingly, “he knew when you’re dropped behind enemy lines – you will need this and this type of material to survive! He was not overly precious about any stages of the process, he understood what he needed and how to achieve it”.
It is worth noting that the visual effects teams needed to be active collaborators with the Director, as Spielberg was also working on the film The Post during this period. Guyett is proud of the way that the team had trust and did not just do exactly what it was told. The artists actively worked to provide the director with the intent of what he asked for, without just “religiously giving back 240 frames of dialogue if that was what was in the edit”, if it didn’t feel right, or animate naturally. ILM felt comfortable returning a slightly different length shot, but one that delivered better animation ,”we were just allowed to get on with it. Of course, we would do what was asked, but we’d also be constantly making suggestions and he seemed to really appreciate that”.
Characters
Cofer commented, “the development of the avatars was quite an effort. ILM got involved early on, working to help prototype the designs, as they evolved. We always had to keep in mind that these avatars are virtual, but they are being driven by real-world actors.” Guyett agrees, noting, “it was very important to Steven, and to us, that we convey the distinct personalities the cast brought to their characters. And the more we got to know them, and the more we saw their work, it couldn’t help but impact some of the design choices we made.”
Art3mis was a complex character to tackle. The character has overly large eyes which gives the character an almost elfin quality. Parzival was hard to balance between the heroic character, and the underdog. The team decided to try and get complete takes from the actors, without complex word by word editing of their performances. The editorial team did do some performance splicing of the motion capture performance data, mainly to add in good reaction shots, rather than editing on individual words as is sometimes done in animated features.
The design of Aech represented the greatest disparity between a human and her alter ego. “Adam Stockhausen came up with some really fascinating looks,” Cofer says. “Aech is a mechanic so he is part machine with hydraulic parts, which was a creative launching point for that character. If you look closely, his skin resembles the tough hide of a rhinoceros and there’s also some manta ray patterning on his head. Those are the kinds of visual choices Helen made to present herself in a certain way in the OASIS. She’s quite a special character.”
The young cast members who were all novices in the realm of performance capture, agreed that the process was demanding but rewarding.
Virtual production
The film was prevized, mainly in the action sequences. When approaching an acting scene, Spielberg would shoot it virtually, much as he would do in real life. For Ready Player One , the director used a cutting-edge VR headset as a tool to direct in a virtual environment. Wearing a VR headset, he could scout an entirely digital set and plan his shots.
Spielberg would scout the virtual set, and he would let the actors be in the environment. With the Digital Domain Zulu kit software for the sets and virtual world, the director would film the motion capture. The scene would be shot with two cameras on the motion capture stage. “Just like any movie, he would be like ‘OK we are on a steadicam, we are pulling back’, etc. Grady or I would often operate a camera” says Guyett. “But of course, you also need more coverage, and that would be stage 2. He could then go back and film further and add to these shots, replaying his favourite performances and filming it from other angles ”
The film would therefore be built in layers:
- Pre-viz to get the beats down and any rough ideas for the scene.
- This would inform how they started the motion capture on soundstage, but the actors were free to explore their acting space.
- Then once after the director had the 2 camera coverage, he could go back and refilm the recorded performances from anywhere to obtain a great wide shot or establishing shot.
This filming of the closeups or establishing shots would often be done on a separate day in a special director’s V-Cam tent, where, viewing the scene on multiple screens—he could frame his shot and do all of the camera coverage that, on a typical film set, would require several takes.
Spielberg commented after filming that, “every single set in the OASIS is virtual, so they created an avatar for me that let me walk through the space and see the actual set. And once I figured out how I was going to shoot each sequence, I asked the actors to put on the goggles so they could get a feeling of what their environment looked like. Otherwise you’re acting in a big white room with a bunch of digital cameras looking down at you. It’s confusing for any actor or director to walk onto a bare-naked set and try to imagine what’s there. With the goggles on, we didn’t have to imagine. All we had to do was remember what it looked like when we were back in the motion capture volume.”
Spielberg also utilized a custom virtual camera rig specially developed for him by motion capture production supervisor Clint Spillers, who had worked with the director on both of his previous mocap films. Having made note of Spielberg’s likes and dislikes on those projects, Spillers consulted with robotic engineer Jim Kundig at Digital Domain to develop and fabricate a lighter, ergonomically friendly handheld virtual camera rig, complete with its own small screen to monitor the real-time game engine render. “Steven could quickly toggle through lenses or adjust his perspective with the push of a button. It’s the slickest camera rig ever designed,” Spillers says with a smile.
All of the motion capture data and head-mounted camera footage was delivered to ILM for the final animation and rendering of each character. ILM’s Oscar-winning Facial Capture System was used to replicate every detail of the actors’ performances. Cofer says, “For all the advanced rendering and effects that we were bringing to bear on this film, if Steven wasn’t making an emotional connection to the character on the screen, it wouldn’t work for him at all. That was a great barometer for us in maintaining the integrity of the original performances. We weren’t interested in redesigning what the actors did; we wanted to translate the essence of it to their avatars.”
For Ready Player One, ILM did just over 1000 key character facial solves. These start with the 4 camera head rigs capturing at 60 fps and fed into the award winning SnapSolving facial solves. The library of expressions was created for each actor using the Disney Zurich invented Medusa rig. ILM would track and solve all the head cam footage and use their solver to drive the FACS rig. ILM would solve to a normal digital version of the actor before re-targeting to the Oasis character. “For Steven it was all about performance” comments Guyett. “It is about making sure the feeling of the performance is there in the digital character,… Steven would say ‘I see Oliva feels suspicious, I am not yet feeling that in her avatar”.
To get the detailed subtle performances, ILM relied heavily, as it always does, on the skill of the character animators to deliver the subtle performances. Uniquely for this film, the animators had to also make sure that the performance of any one character is consistent to their digital character, but also, as the film cuts back and forth to the live action actors outside the Oasis, they also have to be consistent with the live action performance of the actors in the film. Naturally, the actors did not give identical performances on set and later in the capture volume, yet the audience must read this as happening simultaneously.
Assets
The motion capture used a real time render engine. Assets were built so that a lower res version was available for the VR scouting and filming, this was true of both sets and the main High 5 characters. “We made (at ILM) real time versions of all of those assets and gave them to DD. Those were the puppets we would MoCap onto”, explained Cofer. “For example, at The Distracted Globe (the zero gravity club), we’d build it out as much as we could, then we would put a pin in it (pause), reduce everything such as the poly count and texture size and give that to DD. They would then make that set run in real time”. Once the shots were done, the team at ILM could then craft the sequences to their final build and material properties.
ILM and DD worked closely together on the entire project, and Guyett points out that one of the less appreciated aspects of this type of modern filmmaking is the great work the vfx producers do in managing the assets. The thing people can take for granted is the administrative work that goes into a project like this, and the vast organisation that is required”. The film contains literally thousands of assets that are often just used once. “World building was huge in this show” adds Cofer.
Not only did the vfx producers need to manager the huge number of assets but this film had vast licensing issues with a huge array of ‘cameo’ assets that appeared only briefly as part of the plot, but they all required careful management and approvals. It was not just a matter of gaining an approval to use a character, many of these well known character and prop assets came with conditions. For example, it was ok to include a character, but they could not die, or they could not be depicted in certain ways, or only if their character can do something cool, etc.
The game companies were particularly keen to have their franchise characters in the film. “We had all these companies who were excited to hand over their assets, Destiny, Battleborn, – they were all so keen so we ended up with all these great individual game assets in the film”, says Cofer.
One of the cameo assets that was both a fan favourite and ILM treat was the Back to the Future Delorean. “Who wouldn’t want to recreate the Delorean. People here went nuts”, commented Cofer. “The artists went down to the Museum in Beverly Hills where they have the real picture vehicle. They took images, they scanned it, of course they watched all the movies, it is a labour of love for these artists when they are re-creating something that is in our DNA, – we grew up on that culture”.
As much as the film seems like a love letter to 80s pop culture, it is just as much a nod to the great work of ILM. From the Delorean to the Jurassic T-Rex, Ready Player One is a nostalgic nod to ILM’s decades of incredible visual effects work. The time period that has been most harvested for culture references is a period when the majority of the world’s visual effects came out of Industrial Light and Magic and the glory days of Spielberg and Lucas. “And Steven would remember stories of those moments, he is a giant well of film history” says Guyett. Given ILM’s cinematic history, having anyone else produce the VFX for this film would have been like watching a cover band.
Crowds
To populate the OASIS, especially during the climactic battle sequence, ILM developed a state-of-the-art crowd system, by which they had the ability to generate thousands of digital characters.
As the characters were fighting in a game, the film did not just have characters being hit once and dying. Characters had shields and needed to be hit multiple times to deplete their reserves and die. This means the hits change colour each time. “We actually built all of this into our crowd system. It was new for this show and built on Houdini. It is all MoCap based with smart linking between vignettes, it is all rule based to have them collaborate or fight a common foe” outlines Cofer. One of the ideas behind the large crowd fight scene was that like minded gamers in the Oasis band together into guilds or clans. A wizard could shoot spells individually but as part of a Wizarding clan, the group could make a bigger spell. “We build all of this into the system much like an effects layer on top, that allows characters to use their weapons collaboratively”.
The new crowd system, had up to 750,000 agents, all running around leaving footprints in the snow, and moving in intelligent and specific ways so that the vast crowd did not look unorganised or random.
The lighting was integral to achieving that effect. Cofer offers, “one of the most important things to Steven was that we lit the OASIS in a live-action way. So we put on our ‘director of photography hats’ and studied the terrific work that he and Janusz Kaminski have done through the years. We really worked to create cinematic lighting scenarios that underscored the action or emotion of the scenes.”
Guyett adds, “If you watch a Pixar-style movie when a character is speaking, they are nearly always lit, as if a character carried around their own lighting rig. When we started doing dialogue in this film the temptation is to light the characters, but Steven was like ‘no no, let them go dark”. When you watch a live action movie that is what happens, it is astounding how often a character goes dark, and so we embraced that”. Coffer adds, ” and it is additive, – since it is so easy for us in 3D to get perfect lighting everytime. If you experienced that perfect lighting every time a character speaks, then there is a falseness that sets in. Things can’t be perfect all the time, and it was interesting that Steven liked the grit we managed to bring to the Oasis so it wasn’t perfect all the time, and that raised the film up a little”. ILM brought a more sophisticated visual cinematic language to the Oasis, even when telling the story with idealised anime style characters.
Action
The first challenge in Halliday’s three-part competition for ownership of the OASIS is a wild, breakneck race through a virtual New York, with an insane array of hazards—including some inspired by pop culture—popping up along the way. The sequence posed a different but equally tricky challenge to the design and VFX teams. “We were experiencing a lot of these assets at different scales, we see Manhattan for example at the start and then we see the street and then you have these close ups as you are driving down roads,” recalls Cofer. “We had a big effort to build the set realistically at scale but then to also set dress in all the little details. A lot of that was ‘dressing to camera’ – much like traditional filmmaking. You could put on the googles (VR) see what the camera sees and then hand place and set dress the sequence, from this massive kit of props we had”.
The race is seen twice, once from the main race and again from under the road as the first prize is won.
ILM used RenderMan’s RIS for their rendering. “It is all path tracing and we tried to do a lot in the render, we would render volumes, with volumetric lights inside the environment with the characters. We did lots of tracing through volumes, and this meant some pretty hefty renders but we got some pretty nice looking shots, says Cofer.
“Part of it is we were working more like a Pixar film where we were trying to render everything in one pass,” says Guyett, “which in visual effects is mad,.. but we wanted that simplicity of one world”.
As an example of the commitment of the ILM artists to the film, Art3mis rides a version of the Akira Motorbike in the film. The compositor on those shots at ILM hand roto-ed the exact same electrical energy off the tire for his shots as there had been in the original, not because he was asked but because he wanted it to be right.
When Art3mis’s bike is totaled in the race, Parzival brings her to Aech’s garage to have it repaired. For astute fans, the workshop contains a veritable feast of Easter eggs, including a number of iconic vehicles and ships from among the most memorable sci-fi films. Looming in the garage is the title character from the beloved animated feature “The Iron Giant.”
For Cofer, the Iron Giant was a personal treat. “For me personally, I used to make our family watch Iron Giant at least once a year, people made fun of me for it, but for me being able to recreate Iron Giant for the big screen was a joy. I really got into it. I thought about what would it look like, what if you could see inside it, as our version of Iron Giant starts out incomplete, so what is he made of? What is his interior, we put a lot of thought into that” he says proudly.
Located within the OASIS is the entire compendium of Halliday’s expansive journals, which are the main reference source for both the Gunters and IOI’s oologists. They have all been catalogued into an ultra-modern library of sorts, the design of which was partially inspired by the library in “The Breakfast Club.”
The collection is administered by a rather imperious Curator, modeled after the late actor Arthur Treacher, whose name became synonymous with the quintessential English butler. The journals are presented in a composite of the virtual and real worlds as, upon request, moments in Halliday’s life are replayed in live-action, three-dimensional dioramas within the digital universe. The sequences were shot with multiple live action cameras that allowed the ‘live action’ world to appear almost ‘digital’ within the Oasis context, with VCR style controls and a virtual camera.
The Shining
ILM was tasked with the film’s stunning recreation of The Shining (1980). “It is kind of a weird thing, It is so interesting, with The Shining sequence we are saying – part of this world that you can enter into is The Shining. So how would any of that be? In the Oasis, how would you enter that world? And of course at ILM we are doing exactly that, we are re-entering the shining, it was very meta!” comments Guyett thoughtfully. “I may just need to sit with that thought” he laughingly adds. ILM rebuilt the world with stunning accuracy, which is, of course, just what the Oasis would need to do. Everything in The Shining sequence is fabricated by ILM to appear as if it had been fabricated by the Oasis. The sequence has proven one of the most popular sequences and popular culture references in the whole film.
The idea was not to make it seem like the characters were in the actual Stanley Kubrick film but the Oasis’s version of The Shining. “The Oasis is like the real time engine of the future,” explained Cofer. “It was not meant to be perfection, it was meant to be something that was being generated in real time, and maybe you don’t always see a perfect picture…” he adds.
The sequences and shots are both replicas from the original film and new shots with the High 5 characters seamlessly walking the corridors from Stanley Kubrick’s masterpiece. Most of The Shining sequence footage in Ready Player One is digital, it is however combined with some original footage as well as a few new practical shots. Real actors were used for the bathtub lady, for example, and this was then combined with digital elements ILM created.
“As Spielberg had visited Kubrick on the set of The Shining, it was fascinating to hear his stories about the film. It was just so interesting to hear his stories of those moments”, recalls Guyett.
While some things such as The Shining were not in the book, “But Ernie was there all the way. You have the guy who wrote the book there to make sure it is in the spirit of what it should be”, comments Guyett referring to author Ernest Cline.
All four ILM studios worked on the film: California, Singapore, Canada and London, for just over a year.
Digital Domain: Outside the Game
Digital Domain (DD) handled principle motion capture and over 300 visual effects shots outside the Oasis. Their team was headed by Digital Domain VFX supervisor Matthew Butler. In addition, Scott Meadows, Head of Visualization at DD was the previsualization supervisor on the whole film and Gary Roberts handled the motion capture.
DD not only worked with 35mm film plates, scanned by the production at Warner Bros., but the live action was shot anamorphically, making tracking and vfx decidedly different from the clean computer generated world inside the Oasis. The lenses were all mapped individually (down to the their serial numbers). DD built a complex model of each lens across the range of f-stop and focus, and this helped enormously with tracking and matching the renders for compositing. A small number of elements were shot digitally, such as the destruction of the stacks was shot on the Alexa, as it had a taller aspect ratio which would allow for later reframing. DD matched the Alexa footage to match the film footage. With a film camera, DD got no meta-data from the camera, so Butler made sure everything was meticulously recorded on set.
In a pivotal sequence converging the real and virtual worlds, Sorrento invites Parzival to a meeting, and then transports him instantly to his office in the form of a hologram. Butler explains, “So you have Wade’s avatar manifested in the real world. We had to be consistent with ILM’s design of the character while giving it a holographic appearance. We’ve all seen holograms and there is an expectation of what that’s going to look like. As a visual effects person, you don’t want to be derivative, but at the same time, you want the audience to recognize what it is. Coming up with something fresh was a tricky thing to solve, but I was very happy with how it came out.” . As was the director. “Digital Domain accomplished one of the greatest effects in the entire picture,” Spielberg attests. “The work they did with the hologram was amazing.”
The character while shared with ILM, the character was re-rigged at DD and appears as a voxelised particle based object. “We used a lot of projection techniques to utilize renders that we requested from ILM, but ultimately everything was released in this holographic way using a completely different system”. The character was built in Houdini, and rendered in Mantra and V-Ray.
The notion of a hologram in a film is not new. Here the DD team had to communicate that the character was not the same as he appeared in the Oasis, to avoid confusion. DD wanted to be innovative and fresh, while still making the hologram easy to understand for the audience. They initially pitched the character as voxelized rather than scanline technology. “We thought maybe these corpuscles of energy that make him up, inherit some of the inertial characteristic of the underlying person (Parzival)” explains Butler. Additionally, voxels were thought to be something that needed to be refreshed, “the computer is constantly playing catch up. So much computational power is needed to maintain the hologram, that he is never fully complete, it is always going back to fill in some gap”. The combination of voxels, delayed and dragged patches or cubes of light, combined with some laser effects (light projectors) produced the stunning final result.
Capture
Gary Roberts headed up the team that did all the motion capture. Roberts was the virtual production supervisor and is now founder of Fierce Hair Studios. Roberts ran virtual production at DD for seven years and was at ImageMovers Digital before that. He and his team built 3 stages in the UK and the production capture there in addition to some capture at DD’s own stage in Playa Vista. Roberts was also in charge of making the special tools used by Spielberg in the V-Cam tent. Roberts’ approach for many years has been to “remain flexible, to enable us to customize workflow on and off set to meet the individual needs of clients and their productions”.
The role of VR was much more than just virtual cinematography. “It is not just motion capture it is also virtual design. Gary and his team created an environment that allowed Steven to design with the goggles on. He could put the gear on and immerse himself in the world ILM would later finalise”, explains Butler. The ability to work in this way in real time at Digital Domain is very much at the forefront of cutting edge visual effects worldwide.
Parzival and Art3mis have their first date at The Distracted Globe nightclub in the OASIS, where the gravity-defying dance floor is not a floor at all. To enable their characters to fly, Sheridan and Cooke were lifted on harnesses during filming on the volume stage. However, for the more intricate spinning and flipping moves, they were doubled by harnessed professional dancers and a trapeze artist.
For the club sequence, Kasia Walicka Maimone dressed Art3mis in a red dress, the texture of which she says, “is meant to suggest an aquatic creature. When we first meet her, she is in a sleek racer outfit, but her look for the club is her most feminine. We created an original vocabulary for her costumes, influenced by the `80s punk rock scene and current street trends. When you have the right elements, new things emerge, and that’s how Art3mis’s look came to life.”
The suit
One of the most effective pieces of VFX done by DD was the suit that Wade buys with his new found riches. DD needed to show how Art3mis touching Parzival in the Oasis was producing a tactile sensation on Wade in the real world. Tye Sheridan wore a very elaborate suit on set, “unfortunately the only way to show that was to fully replace the suit with a digital one” explains Butler. “Everyone thinks you can tweak the plate, but it is impossible, the only way to do it properly is to fully replace the suit”. Butler and the team worked hard to honor the design of the original suit but once they made it digital they could have sections light up and internally illuminate to indicate Art3mis’s touch. Butler comments “you get to see the activity as somehow sub surface to the suit, .. and what we hope the audience gets is that she is really touching him up, and he can feel this properly in the real world”, all of this without having to have verbal exposition to explain this to the audience. This was again done in Mantra and V-Ray.
Sixers
Sorrento’s office set was raised on stilts and had a floor-to-ceiling glass wall so he could watch over his enormous ‘war room’, manned by his corps of uniformed Sixers. Each Sixer was jacked into an individual rig, working in shifts, 24/7, trying to win the challenges. The scenes were shot on sound stages at Leavesden, in the UK.
The Sixer rigs were designed by SFX supervisor Neil Corbould , who said, “A lot of development went into the initial design and construction. Once everyone was happy with the model, it was just a matter of multiplying it 80 times.”
Digital Domain multiplied the rig out. Butler confirms, “The war room only existed as a partial build, but it’s supposed to be massive. To give it the proper scope, we created a full synthetic (digital) version of it, but you can’t just have a room; you have to fill it with people. That is one instance where we did motion capture for the real world, as opposed to motion capture for the OASIS.” While there was some hand animation the majority was motion capture from one of three sessions that DD did. The CG people spanned not only the sixers in the rigs, but drill sergeants and even people walking in the background.
Practical Explosions
Digital Domain handled all of the other visual effects for scenes in the physical world, including the stack explosion. While some live action Pyro was done, it was more reference, and for contact lighting. There was no actual destruction from the pyro. The explosion was rendered in Mantra. When Wade’s stack digitally falls, the entire fall is a CG element placed into plate photography of actor Tye Sheridan (Parzival / Wade) running towards the moving camera (below).
The Stack set was digitally extended by DD, using V-Ray via Maya and composited in Nuke.
Digital Domain also did the drone sequences, and the close up shots of Wade transitioning into the OASIS. Additionally, in the bowels of IOI headquarters is the Loyalty Center, which consists of endless rows of small metal cells, each holding a person wearing IOI-issue gear that is locked on so they can work off their debt doing menial labor in the OASIS, here DD did set extension and additional effects.
We asked Matthew Butler, what retro reference most delighted him personally , he did not pick a visual. Bulter confessed it was the music of the film that made him smile. “For me, it was them walking into the club and hearing ‘Blue Monday”!