Snoke: Deep Dive into the latest ILM face Tech

In Star Wars: The Last Jedi, Industrial Light & Magic provided the digital Supreme Leader Snoke, as the latest in a long line of digital character animation work from ILM.

Andy Serkis provided the base motion-capture performance that was re-targeted to Snoke.  Audiences had seen Snoke in the previous Star Wars: The Force Awakens as a hologram, in the new film he appears as a hologram but also very much ‘in the flesh”. ILM’s creative director Ben Morris comments “There was something almost gelatinous and zombie about Snoke’s Hologram-look before and Rian (Director Rian Johnson) felt he definitely wanted to step away from that. He also needed the character to interact”. Ben Morris is the overall VFX Supervisor on the film, Creative Director at ILM and a founding member of the London studio. Morris began his career as a Model Maker at Jim Henson’s Creature Shop, where he designed performance-control systems for various films, including Babe. Morris earned an Academy Award for Best Visual Effects on The Golden Compass. 

The updated Snoke model.

Neal Scanlan, Creature And Special Make-up Effects Creative Supervisor, was given an opportunity to make a new maquette of Snoke for the new Star Wars film, and that took the character towards having more human skin, withered with aging. For the Snoke’s previous appearance in The Force Awakens, that film’s director J.J. Abrams and Neal Scanlan didn’t want the character to look old and decrepit like Darth Sidious. Ivan Manzella sculpted at least two busts for the original look of the character, and his skin was far less human. The new Snoke required a technical reworking by the ILM team. “That really pushed us into a different world”, comments Morris. “Snoke is more damaged but in a far more subtle and refined way”.

Snoke is about 7 foot tall in this film. The hologram in the previous film had Snoke seated and isolated so Johnson was free to make this Snoke as tall as he liked. “We actually got Chewie’s stand in and another tall actor in and decided – ‘yeah that could work”, recalls Morris.

The director and Costume Designer Michael Kaplan, dressed Snoke in, what Morris describes lovingly as, the ‘Insane Gold Dressing Gown’. Due to all of these changes, Snoke became a completely new character technically and was rebuilt from the ground up at ILM, based on lessons learnt from the previous film.

Steve Aplin, is ILM’s global Animation Supervisor and led the team of ILM animators that brought Snoke to life.  Steve was formerly at Double Negative where he had been Creative Head of Animation. Steve worked on the original Snoke in The Force Awakens. Snoke required complex acting.  The character has scenes where he is at once friendly and sinister. This layered sub-text really allowed the ILM animators a chance to provide nuanced acting. “Andy’s performance was the backbone for everything, it was so strong, …once we saw the on set footage cut together as an edit you just knew, even looking at Andy in his MoCap suit, it was going to be a super intense set of scenes”, comments Aplin. “We just wanted to make sure we held on to what he brought us and built on that”. The audience had not seen Snoke close up in The Force Awakens, “so we knew we had an opportunity to really explore micro expressions in the closeups. In those moments when you are really in close, you can tell what his emotion really is, and it is a different story to what the broader action was of his body”. This layering of seemingly welcoming arm and body movements compared to underlying threat was conveyed with tiny facial adjustments, visible in the closeups. “Just using tightening around the eyes, little creasing on the brow, the way he was swallowing – all those elements are little key points we paid attention to” explains Aplin.

Inspiration: Robert Helpmann’s performance in Chitty Chitty Bang Bang.

While Serkis provided an extremely strong performance, it is incorrect to think the technical motion capture provides these subtle facial aspects automatically. The face is tracked but it is at a fairly coarse level (see the sparse nature of the dots below), and all the details such as Serkis’s eye moisture, blood distribution and coloring etc is simply not automatically captured. The smallest amount of eye tearing manifests as a tiny change in specular pings on the bottom of the eye and this can change an audience’ interpretation of a character. As with much great character writing, it is often the case that what a character says and what they mean are often very different, if not diametrically opposed.  “We kind of joked, his nastiness is almost as twisted as the child catcher in Chitty Chitty Bang Bang, he is almost like a cat playing with his prey, with his emotion moving between humor and anger, which I loved”, commented Morris.

“We definitely did a first pass solve, but I would never understate how much animation goes into something like Snoke, there is a huge amount of animation, we do a pre-solve that we give to the team and that is very much the basis of the performance, but the guys work on top of that significantly“, explains Morris

Serkis was motion captured by his Imaginarium Studios team using the four camera Vicon Cara head rig. ILM also animated some sequences where Serkis did not wear a motion capture helmet. These shots were key framed and hand animated by eye.  When Serkis was wearing the Head Mounted Camera (HMC) gear, the production team placed markers on the actors face. On the glancing angles of an actor’s face, details can be lost, so ILM had small hemispheres attached to Serkis face so these parts of his face would still be accurately captured in 3 space.

Serkis was captured in a four day period as he was still finishing the last Apes film, “Andy had to sort of fly in and fly out” explained Morris. Importantly, even with a tight schedule, both the director and Morris felt it was vital to allow the actors to still find the ‘space on set’, walking and physically exploring the set without interruption or disturbance. “The words I gave to the onset team were you have to let Rian and the actors have their moments. For Andy Serkis, Daisy Ridley.. and in particular Adam Driver – who is a particularly intense actor, Rian would clear the set and it would literally be me in the shadows, the first AD, Rian and just the actors,… and they would go through the entire performance. At that moment the tech is irrelevant. “I think this is the future, I think all our tech needs to seamless and not be a burden. And the brilliant thing about Andy (Serkis) is firstly he is nice amazing guy and second he never complains about the tech”.

In these images the side dots can be seen as actual hemispheres so that the face side detail was not lost.                     (As published in WIRED).



 

Johnson was very happy with Serkis’s performance but Snoke is physically very different in head and jaw shape from the actor. “Andy is naturally fleshy and his face moves so much, and when you are trying to pull that back to someone who is far hollower, someone who is more bone with skin tightly stretched over, – we found on our initial solves we were getting just far too much skin movement”, explains Aplin. The animation team had to tone down the movement and then work hard to get back the essence of Serkis’s performance. “I think Rian (Johnson) was incredibly astute, we could show him a fresh solve, as we started a new shot, and he’d immediately say – “wait hang on-  this is way too elastic, this guy has enormous amounts of scar tissue which has much more stiffness”. He was spot on with honoring Andy’s performance with Snoke’s anatomy” adds Morris.



 

ILM used the HMC, but they also had a Medusa Rig set up outside the stage. This meant that the team could do an additional near perfect fine detail facial capture scan not only of Serkis’s face but also the actual marker positions, so there was a detailed positional scan of the dots done each day.  The Medusa was also used for updating the expression space in ILM’s FEZ library poses (FACS poses) for Andy Serkis. “We had Thabo (Beeler), Derek (Bradley) and the guys come over from Disney Zurich Research and they were up at Pinewood with us, which meant we could do a refresh on the data sets of Andy (Serkis), which is great as people naturally change year to year. And that is something cool, we now have the ‘uber-Medusa’ rig in ILM London permanently staged with 6 times the resolution I think of the original”.  The new  uber rig was only tested on this Star Wars film but “we have already upgraded that and it is something we will continue to use” explains Morris. The new Medusa allows slightly more head movement given its higher resolution which allows for a more natural capture.

On the film Warcraft, ILM heavily used their Award winning SnapSolve software to provide the volume of quality facial animation required on that fill.  While on The Last Jedi, the volume of Snoke’s CG performance was significantly lower in screen time, the ILM team still used their proprietary SnapSolver. “We used SnapSolve to a broad extent… we did spend a lot of time on the initial solve stage compared to what happened on Warcraft.  We had more time on each shot to get the best possible solve”, explains Aplin. “We did find Snoke could go off model with some solves and that is why it is import to hold as much of the initial data as possible, …before it got into the animators who could rework it by adjusting a few things including some ‘shape work’,  to just bring everything back to Snoke’s face, before finalising the SnapSolve”.

The ILM pipeline started with the match solve from Andy Serkis to a digital version of Serkis and then from there, once the performance was matching, to re-target this to Snoke. “The truth is when you are doing a SnapSolve you want to pull through as much data as you can. You want to decompose at least a few of the key channels such as jaw movement, eyes, blinks and lips first. We do that with a spline track,… but we weren’t doing a full decomposition of every muscle in the face”. This allowed the animators to be passed a face that was fully rigged and could be re-animated, as much as was needed. The solved face provided a full keyframe animation rig. From the SnapSolve the animators get a set of deltas which are divided into small areas on the face which the animators could then decide to tone down or fully swap out for keyframe animation. “We always thought there was some great underlying surface movement that we could get from any of the retargeting, so we never wanted to lose that too much”, says Aplin.

For Morris the holy grail would be to have no HMC and do the solving and animation by using witness cameras. On this film the team had three production cameras, three Red cameras as well as the four HMC cameras on most shots.  This allowed not only the facial capture with the HMC but the matching head movement (the head mounted HMC camera’s don’t record head movement, as they are ‘glued’ to the head). As Morris points out, the head movement is key to performance, “if you stabilize and zero out the head movement for a technical look, you often look at the face animation and say ‘that’s never going to work at all’, but of course once you put the head movement back in, it works perfectly”. Aplin is also very keen to not combine separate parts of separate takes, for example, mixing the facial performance of one take with the body motion of another. “In my mind it is very much face, body and head as a package” he remarks.

Snoke was rendered in RenderMan RIS v21. It represented a whole new set of shaders and a full upgrade to the previous render pipeline. The team were really enthusiastic about the results. “Rian (Johnson) was very worried about Snoke, he did say to me,” recalls Morris, “‘is this just nuts Ben? Is this going to work? Should we just get an actor in with makeup on them?’ And I said no. You will never get Snoke if you do that.  The whole process of LookDev-ing the character was very informative for Rian, and he finally got it …and he saw we could zoom right in on Snoke’s face. Having him share in the (LookDev) process was key, I believe, to dealing with his concerns about a CG character”.

The renders of Snoke took place in a very red set, with a lot of reddish bounce light. ILM had reference of the maquette on the set, but the production also hired an older actor to provide reference. “We gave him a version of the gold dressing gown and he would jump in between takes and go through a couple of key words or lines on every set up, after Andy. In addition, we had a younger 7’2″ stand-in who would wear the larger flowing robes and he would do any of the larger movements for lighting reference”, explained Morris.

ILM played with having one of Snoke’s eyes as a damaged clouded eye, but this was rejected. Snoke’s eyes are essentially human, as the Director always saw the character as a ‘messed up human’. But Morris recalls when testing the eyes there was one version when Snoke had very piercing blue eyes “I think we went too far with that one – we called him the Steve McQueen version” he laughingly recalled.

Snoke was always a “messed up human” for director Rian Johnson.

While Andy Serkis was also body captured, ILM did have to also rework the body capture data, since the Director wanted a final look that had Snoke more threatening physically than Serkis had initially portrayed. The Andy Serkis Snoke body was more doubled over and broken, and about half way through post production the Director decided to have ILM straighten Snoke up and have him physically taller. This translated into body adjustments and extensive eye line adjustments between Snoke and the other hero actors in the scene. “We un-did some of the twisted work in his body and of course all the eye lines had to be adjusted and some of the shots reframed, especially the fully CG closeups” said Morris. This also worked well with the actual sound mix of Andy Serkis’ voice. “Andy has a very resolute voice, especially after it had been through the sound mix, but the initial Snoke animation with a withered throat and sunken chest just looked like he should have had a wiry and wisy “StepToe and Son” grandfather* voice not this booming voice, so that was another reason for resculpting him half way through production” comments Morris.

Snoke’s death scene required detailed subtle facial animation. “We definitely went around a bit on that, how much should his look be surprise or pain, how much is he just questioning everything. He had this grand plan in this huge game of chess which he was the master of, and suddenly he is outdone by his student. All those elements had to be juggled on the close up” Morris explains. The team also had to produce the wider shot showing the audience how he had been cut in half. The ILM animators needed to judge how much Snoke’s body would collapse and the torso would fall “less like a person and more just like a slab of meat falling to the floor’, commented Aplin. The team did have a target pose to aim for as Snoke’s final resting pose had been established and previously approved. “Rian is into his comedy, he liked it, and the large chunk of meat falling on the floor got a rise and chuckle out of Rian… – that was the counterpoint to someone being sliced in half in a Star Wars movie” commented Morris. On set there was a practical Snoke torso in shot on the floor for the later scenes. ILM ended up replacing this practical Snoke torso to make sure the dead Snoke’s skin and detail matched the digital alive version the audience had seen up until that point.

Other character work

Snoke was not the only character work that ILM did in the film.  While Snoke was the most challenging and complex, another briefly returning character is Maz Kanata played by actress Lupita Nyong’o.  For the Maz cameo, the ILM team gave the actress “a real time render of herself and the character, we set up a stage for her to act out her sequence”. As the actress has different proportions to her on screen character the digital ‘monster mirror’ helped her refine her acting. She had also previously spent time with Andy Serkis learning more about how to translate her performance in a MoCap environment.

In a generous move, actors Anthony Daniels (C3PO) John Boyega(Finn) and Oscar Isaac(Poe) all turned up and were there at the end of the stage so they could provide Lupita Nyong’o with off screen dialogue to act against. It is common for such pickup shots to just have a spare crew member feed the lines, but the three actors attended even though they were not on screen and thus technically not needed. “They delivered their lines to her so she wasn’t just listening to pre-records or anything” recalls Morris.

Maz chats to Han Solo in the previous The Force Awakens.

Given the full CGI nature of her character, Maz required a full motion capture stage. Audiomotion provided the Vicon cameras and capture setup at Pinewood.  They set up, rehearsed, filmed Nyong’o, and then packed up the whole capture volume all in a matter of a few days.

There was also the casino toad guy, Dobbu Scay, “who is a cameo by Mark Hamill,… I knew he had done some voice over work, but he complained bitterly when he saw Andy. He stated ‘I have never been asked to put these balls on and do the whole bit!’. Rian was very gracious and said “alright, we will work something out!” explains Morris. Thus Mark Hamill spent a day at ILM’s motion capture stage in London after the shoot playing around recording that character.

A Fathier physical prop in a stall from The Last Jedi behind the scenes footage shown at the D23 Expo.

The team also provided the casino escape sequence with the Fathier, a horse/lion style quadruped.  This was extremely complex and required a lot of interaction with other characters and props. There was also normal crowd extensions and digital double work that was required just for cleaning up characters and reflection etc as seen below.