The Visual Effects Society (VES) held the 20th Annual VES Awards that recognizes outstanding visual effects artistry and innovation in film, animation, television, commercials, video games, and special venues. This marks the Society’s 20th VES Awards program. Dune was named the photoreal feature winner, garnering four awards. Encanto was named the top animated film.
VES Winner of Outstanding Character in an Episode or Real-time Project:
The Witcher; Nivellen the Cursed Man awarded to Marko Chulev, Rasely Ma, Mike Beaulieu, and Robin Witzsche.
The Witcher follows the story of Geralt of Rivia, a solitary monster hunter, who struggles to find his place in a world where people often prove more wicked than monsters and beasts.
Nivellen is a beast in ep01- who is part boar, part bear, part man. Nivellen was played by actor Kristofer Hivju. ILM provided the digital face of the character, who is one of the most iconic creatures in Andrzej Sapkowski’s Witcher books. The character first appeared in the short story “A Grain of Truth.”
We spoke to the ILM team about their approach to the character which was almost entirely produced during the most difficult of COVID restrictions. ILM did a full Cg Head replacement for the character of Nivellen.
Kristofer Hivju was transformed to Nivellen using facial Capture with a single Head Mounted Camera (HMC). Hugo Debat-burkarth, Look-Dev on Nivellen co-ordinated with modelling, texturing, and the groom team to get the final look of the character. The look-dev had several challenges in translating the human traits of the actor to the boar-like face of Nivellanto. “One of the main challenges was that we did not have any prosthetic reference to work to as the production was building the prosthetics in parallel with the CG,” Hugo Debat-burkarth comments. There was a physical maquette made but it was not finished and available to ILM until quite late in the production. Unable to match a specific physical sculpt or prosthetic, the team started with a fairly human digital face with the intent of adding and converting it to look more like a boar, once the look was finalized. “The main idea was to make him look dirty, but not too much as he still had to be a bit classy, but with his body exhibiting some disease,” he adds. “His skin, in particular, is not healthy.” The team worked back and forth to find the right skin texture.
Once the design was further along the team experimented with making the face look more like a boar. Too much and they lost the expression, not enough and the character still looked human. There was some early consideration of keeping the actor’s actual eyes, but they decided against this quite early. One of the ideas that the team came up with at this point was replacing the eyes of the character with gorilla eyes. Thus, the Nivellen would not look human but still, have eyes closer to that of the actor than of an actual boar. Ultimately it just became too hard to faithfully match Kristofer Hivju performance this way and the team ended up with digital copies of the actual actor’s eyes.
Simon Herden, Lead Compositor, “We started off with eyes that were more bloodshot and then we moved away from that,” he explains. On the very first close-up, Simon’s composite did steal some reflections and pings from the actor’s eyes in the plate photography, but “immediately after that first shot, we figured out the formula for dialing it the look from lighting to comp. The plate photography was always king, so we would always go back and look at how Kristofer’s eyes looked and then get very specific in comp, in dialing it right in to match – so that we could always maintain Kristofer’s performance.”
The problem the team had was that Kristofer’s face was sitting back and recessed into the prosthetic head wig. His face was therefore well back from where Nivellen face physically would be in terms of lighting and shadow, plus it would often have the shadow of the HMC and capture camera falling over the actor’s face. “However, Nivellen’s larger brow gave us a little bit of license to play his eyes into shadow in the same way that Christopher’s were on set,” Simon explains. “But it was always taking the plate photography as a starting point and then if sometimes the eyes or face were glowing a little too much, we’d bring them down globally.”
The HMC was only a mono camera due to production constraints. “We went with the approach of single-camera computer vision camera, binocular HMC set up and with only very sparse markers, so in terms of a facial expression solving solution we pushed the markerless tech that we built for The Irishman, the adapted our flux system,” comments Adam Ferrall-Nunge, senior R&D engineer at ILM who worked on development and production support for the face capture pipeline. The team combined the Flux work with newer research in virtual face landmarks from the Disney Research team in Zurich. “That was a core part of our expression solving approach, which gave us a relatively decent fidelity of Kris’s performance. Understanding of course that we’re not targeting a skin type perfect solve, because it’s going to be retargeted to a boar creature and then touched by animation after that.” This is one of the main reasons the ILM team could make their handoff work so well between capture and the animators. It was always key that the animators would be able to provide substantial direction on top of the technical solution.
“We worked in tandem with animation so everything we produced the animators were able to control but solving Kris’s face for his performance to get the right expression was a crucial part,” recalls Rasely Ma, the layout lead and face comp supervisor in Witcher for Nivellen. “It was key to get that exact expression with fidelity to the animation team so that animation can bring it to the finish line.” On some shows the captured actors’ performance would have been just reference and animation would have ‘done their own thing’, but Rasely Ma pointed out that on Witcher, “we had very tight communication. No work was ever left unused. We fully utilized every single piece of information we were given from the facial solve.”
The pipeline approach for high-end work is to take the data from the HMC and use it to animate a digital double of the actor, only once this interim digi-double looks perfect, does the team then retarget that data to the character face. This is standard for high-end work. Of course, this requires a model of the actor’s face, which under normal conditions is an easy thing to request and scan with a sensible range of expressions (or ROM – range of motion). But with COVID the ILM team were not able to be on set, not able to scan the actor or have anyone else scan his face. They had to model Kristofer Hivju from just still images they could gather from the internet, and in many of Kristofer Hivju previous roles, he had a very strong and dominant beard. “There was no actor scanning – it was at the height of the pandemic – it was all just copied. The production had just started in March 2020, so suddenly we were all working from home,” recalls Bruno Baron, ILM’s VFX Supervisor. “Kris, the production, us – everyone’s suddenly working from home, with no way to organize a scan. So, it just never happened.” The only option the team had was to request that the actor get his iPhone and with ILM’s instructions, he filmed his own face and head by just moving his iPhone around at his own home and then emailed the video to the ILM team, – not exactly the structured Light, Light Stage or Medusa data the team would normally get, but better than nothing.
Marko Chulev, Creature Model Supervisor at ILM, and one of the VES winners, set up a rapid prototype almost alongside the established ILM pipeline that iterated on the character and took any data they could to improve and constantly refine Nivellen. “He’s brilliant,” comment Bruno Barons. “He managed to get these really fast quick iterations. But then because this work lives outside our normal pipeline, you’re a bit limited. So to make sure we could take advantage of this work, he would constantly convert that into our pipe structure. This way we still maintained about 90% of that work and yet we didn’t have the kind of data that we’d get from nice scans normally.” The team credited Marko Chulev in doing an amazing job of creating the blend shapes for the shapes library of the face and in making sure that the animation team, led by Animation Supervisor Mike Beaulieu, still had a lot of access and control beyond the face capture target solve to push further with the Boar performance.
The next issue that animation faced was dealing with the tusks of Nivellen. As animated as Kristofer Hivju was in giving an honest performance, he was not acting with two huge tusks – which completely changes the mouth expressions and naturally his spoken dialogue. The face rig was still using the canonical animation style ILM rig, but one of the pieces The R&D team had to address was adding a capture expansion toolset that was able to solve the tusk issue within an animator-centric face rig. This allowed the process to solve on a similar control structure of the digi-double Kris – that would be relevant and present on the Nivellanto rig. Without that correspondence, one could not do a sematic transfer, but still the process had to allow for the structure difference and behavior the tusks presented.
But even with the vast technical strength of the technical ILM pipeline, it must be noted that there were some shots where the HMC camera did not work or record, it either broke or had technical issues. “So, we’d get these really complex shots back with no HMC footage at all. But I think with all the cues the animators had from all the surrounding shots, plus the HMC footage they did have, – that the team got the gist of what Kris’s performances and nuances were at that stage, and they were able to kind of really nail it straight up in keyframe animation. In fact, it is to the point where you wouldn’t be able to tell what’s face captured and what’s keyframe animation,” explains Shaun Roth, CG Supervisor.
The hair or fur was a major issue. The ILM team had to blend the digital hair into the physical head wig. But again, due to COVID, the wig and weave of the actual hair was unable to be sent to the team for reference, as they were not in the office. Thus, matching the digital hair had to be based on just the plate photography. On the top of Nivellen head the hairs ended up being like bristles – with a thickness three times that of a human hair.
The integration of the hair changed on a shot-by-shot basis. If Nivellen turned to face away from camera, such as in the hallway when he is turning on the lights, then the team would replace the whole head because it would minimize the challenges in blending between CG and prosthetic. For scenes where Nivellen is at the dinner table, and thus primarily viewed from the front, it was almost like a mask that would stop just before where say Kris’s ears would be under the wig. “We were able to get a really nice kind of clean kind of mask of all of those fine hairs, and then, the CG would just sit straight on top,” commented Simon Herden.
The final render CG of the Nivellen was in RenderMan. All the renders were produced with a Deep pipeline, mostly to help with isolating the hairs for facial integration. This was coupled with a robust set of AOVs which were used to adjust specular in particular, to avoid Nivellen looking too ‘sweaty’ and also to help with flickering candlelight interactions and contact lighting adjustments as part of the Comp.
While the Nivellen shot count was relatively low by VFX standards (~120), the work is outstanding and now award-winning, – and all done under complex and challenging COVID conditions.