Reallusion, a maker of 2D and 3D animation software, helped power the real-time mocap and previz for Replicas, the new Keanu Reeves film.
In the film, after a car accident kills the family of William “Will” Foster, played by Keanu Reeves, he will stop at nothing to bring them back, even if it means pitting himself against a government-controlled laboratory, a police task force and the physical laws of science.
On the verge of successfully transferring human consciousness into a computer, synthetic biologist and neuroscientist Will believes he can essentially resurrect his family. Will recruits fellow scientist Ed Whittle to help him secretly clone the bodies of his family and create replicas. Will eventually faces a “Sophie’s choice” when it turns out that he can only bring three of his four deceased family members back to life. The film features what has been described by one critic as “cinema’s weirdest not-to-be-missed robots, as it scarily, giddily threatens revenge, with a mere turn of its head.”
The story and screenplay for Replicas was developed at Keanu Reeves and Stephen Hamel’s production company Company Films.
Executive producer James Dodson got involved at the very beginning of the Replicas project two and half years ago, around the time they decided to shoot the film in Puerto Rico.
The film was shot on the Alexa and ended up with over 400 visual effects shots. Keanu Reeves was heavily involved as both producer and actor. “He was interested in the whole movie. It was his movie. He developed the script. He was on it for two years before we shot it…and he was particularly interested in the AV stuff,” comments Dodson, referring to the minority report style sequences, which were complex to act and animate. “Basically we were trying to tell the story that Keanu’s character, William Foster, was going into the human brain and analyzing the data in the brain and then transferring the data elegantly into a machine facsimile of a human – into a synthetic human brain, if you will,” he adds. “And it’s not like you can just go and buy that on Turbo Squid.”
The team worked with multiple vendors, spending months to try and find the right design solution. “Every piece of user interface was crafted, there was nothing off the shelf that you could use. Everything had to be designed specifically for this because it was such a specific story. Keanu was keenly interested in every step.”
The user interface designs of the brain in AR were started by Chris Keifer, who does much of the screen graphics for Westworld. “He finally cracked how to represent it. We had 9 or 10 weeks with him, before Pacific Rim 2 stole him away from us!” jokes Dodson. “Then Josh Zacharias took over, I talked to him almost every day for a year on this project. He really elevated the work and nailed it. It had to be believable and it had to tell the story but not be too on the nose…it was just so important that the audience would buy it.”
The animation was choreographed to the motion data gained from Reeves hand movements, animated in Maya and rendered in RenderMan.
The team used iClone software in the production of Replicas in three different ways:
- James Martin and John Martin led the previz team in visualising some of the action sequences such as the car crash in iClone.
- There was also conceptual work around the AR infographics that Keanu’s character manipulates, that needed the software for the interaction to work onscreen.
- Finally in post-production, iClone was invaluable in animating the robot when it gains consciousness.
Reallusion makes the iClone software the team used. They are headquartered in Silicon Valley, with R&D centres in Taiwan, and offices in Germany and Japan. Reallusion is focused on the development of real-time cinematic virtual production and motion capture tools.
Reallusion provides users with character animation, facial and body mocap, and voice lipsync solutions for real-time filmmaking production. The company first launched iClone v1.0 at the end of 2005 and it found popularity within the Machinima community. The current version 7 was released in 2017. In addition to having its own real time engine, an easy-to-use avatar and facial morphing system with a voice lip-sync solution, the software interfaces with most mocap systems.
More recently the company has developed Motion LIVE, which is a body motion capture platform that connects motion data streams from multiple industry mocap devices, such as Xsens and Rokoko, to animate 3D character faces, hands and bodies.
Motion LIVE did not exist when Replicas was in production however. “That program didn’t exist 18 months ago when we were doing Replicas, but we used a similar thing. It was just Perception Neurons, motion capture software (by Noitom), which then fed directly into iClone. Replicas did not need facial capture of the actors’ expressions, only their movements.”
In the robot sequence, the robot was 100% CGI. On set there was a mocap artist in a Perception Neuron suit. This live data was fed into iClone and this allowed for a ‘slap comp’ version of the robot to be seen on set for framing and blocking. The edit used this footage as post-viz. “The great thing was that once that sequence was cut, we could send those exact MoCap files down to Argentina where the artists could take the data and attach it to the final full resolution robot,” explains Dodson. The export from iClone to Maya was done as an FBX file.
For the crash sequence, the team used iClone to work out how to film with the stunt team and physical effects crew. The stunt car was on “a crane cable with a gymbal and we worked out which shots we could chuck it in the air and pull it back like a pendulum to drop it, and which ones would need visual effects,” Dodson recalls. One of the features the team used in the previz is the ability to use a single jpeg to create a representation of the actual actors on the 3D models used for the layout and previz. Each of the family members was modelled this way for the previz.
Dodson fully acknowledges that major big budget films use expensive and often proprietary technology for simulation and previz. However for him, the benefit is in how inexpensive and accessible tools, such as iClone, are able to, “democratize the entire process of motion capture and real time rendering – I am just addicted to previsualisation, and the level of quality and detail we can do for a few hundred dollars and still have studio quality results is incredible.”
