Warcraft is a fantasy epic adventure which pits orcs against man in one of the year’s more ambitious productions. The film that took almost three years to make was actually shot and into post-production before Star Wars Force Awakens. It was meant to have hit cinemas months earlier but was moved back to avoid release clashes.
On the up side this allowed editorial and the effects team a bit more time for post-production. This means, however, that the film’s pipeline actually preceded some of the technologies that ILM developed for films already released, such as The Force Awakens. Bill Westenhofer, Oscar winner for Life of Pi and Golden Compass, was the production supervisor reporting to Director Duncan Jones. ILM’s Visual Effects Supervisor was Jeff White.
Warcraft represents a vast undertaking in terms of character, environment, battles and scale. Half the film’s emotional narrative is told by the film’s CG orcs and in addition there are immense battle, huge fights and CG dwarfs, griffins and magical creatures.
The film is based on both the game Warcraft and the immersive World of Warcraft or WOW. The highly influential Massively Multiplayer Online Role-Playing Game (MMORPG) WOW was actually launched in North America 11 years ago and still boasts some 7 million players, which is even more remarkable when one acknowledges there is a monthly subscription.
The WOW game mechanics rely largely on animosity between the major factions that players are allowed to join. Both main factions, The Horde and The Alliance, were initially comprised of four playable races each though since the expansions several more have been added. The Horde was initially made up of Orcs, Trolls, Undead, and Tauren, whilst the Alliance were Humans, Dwarves, Gnomes and Night Elves.
To fully appreciate ILM’s work, one has to understand how faithfully the filmmakers embraced the game world. This is very much both a tribute to the fans and the key to the film’s entire approach. Director Duncan Jones treated the game with the reverence and respect normally only afforded a much loved literary piece. Of course, there is a vast difference between a narrative feature film and an immersive game. But it is not too hard to draw parallels between the first three Lord of the Rings films and what Jones was aiming to do with Warcraft.
Hellfire Peninsula, Draenor, Lakeshire, Redridge mountain, Wetfall, Blackrock pass- are all well known to the WOW community. Jeff White felt that right from the start that Director Duncan Jones did not want to water down the WOW world, and this be disrespectful to the fans. It is perhaps then no accident then that the physical effects and prop team included a major contribution from the Weta Workshop, whose skill in manufacturing swords, armor and practical weapons is legendary.
Logistically in the MMORPG, one chooses a faction (such as Mage, Warlock, Priest, Rogue, Druid, Hunter, Shaman, Warrior, Paladin) and a race. For the film the story mainly centres around the Humans and Orcs but with a variety of other races in various key roles and/or making cameo appearances.
On the whole, the film stays true to the MMORPG with one major exception, Garona Halforcen in the game world was actually half Orc and half Draenei. In the film she is half Orc and half human. Garona may still have some Draenei blood, but she is presented as half human – but more importantly she is green not blue, as in the game world. The actress, Paula Patton acted on set without any specialist makeup and her green hue is from her performance being entirely rotoscoped for every shot in the movie.
Orcing around
The film starts with Gul’dan uniting the orc clans and forming the Horde. He creates a portal to the world of Azeroth by using fel magic to drain the life out of their captives. Gul’dan leads a small warband to capture prisoners on Azeroth and sacrifice them to bring the rest of the Horde through the portal. For much of the film we follow Durotan, the chieftain of the Frostwolf Clan, and his initially pregnant wife Draka.
Starting the story with the orcs and requiring the audience to connect with Durotan and Draka was a major challenge for ILM. While embracing the source material works for many things, the characters in the WOW world are heavily exaggerated with tusks and vast muscle bound torsos. “How do you make something look real when the character’s hands are as big as their head?” questioned White when he first joined the film.
It is one thing to have a field of rushing army of invading orcs look cinematic. it is a lot harder to have a pregnant mother – with tusks – joke and lovingly tease her husband by a warm firelight and not look stupid. White and the team started with this intimate scene as the basis of perfecting ILM’s facial performance pipeline. “It was one of those shows that was a bit overwhelming to take in everything that had to be done at the start, – but by the same token so much fun in terms of the variety of things we got to do,” explains White.
ILM got not only production artwork, but they got to work with a production artists from Blizzard Entertainment directly. Jeff White was particularly impressed with the work of concept artist at Wei Wang (Principal Artist at Blizzard since 2005) or Glowei (also an artist member of creative collective the Sons of Storm). “Wei did nearly all the hero character designs and they are just phenomenal – I think he is working on an art of Warcraft book and I hope he is, because his work is so beautiful”
Director Duncan Jones was similarly impressed. “One of our concept artists is this amazing guy called Wei Wang who was actually a fan of Warcraft,” Jones stated recently. “He had done such amazing artwork and he submitted it to Blizzard, and they hired him to come onboard for them. He was the one who realized the sort of true dimensions of an Orc and how to realize them in a sort of live action environment.”
Wang’s initial highly detailed drawn images were the blueprint for the work that ILM had to do, “but I remember looking at them and saying to Jackie Lopez (ILM Visual Effects Producer): ‘Do they have to have quite so much hair?'” laughingly recalls White.
Facial capture
With so much dialogue, the team realised that they would need to do more than just get reference facial footage or even normal facial capture that required vast amounts of clean up. With 13 speaking orcs in the film White and the team set about producing a facial lip and lip sync pipeline that would capture much higher fidelity and require much less manual intervention or the film would never meet its targets and deadlines.
Key in the new approach was a much higher on face dot count and a new gaze tracking system for much more subtle eye movement tracking. The team started with a test. The shoot for the initial test was actually done before principle photography “so that a few weeks into principal production we could show everyone here is a solved performance of Rob Kazinsky as Orgrim, and even though it was grey shaded it didn’t matter…the actors could see on the left is the performance on the right is the solve onto the character’s face, – that got the ball rolling and gave the actors a lot of confidence”.
The second shot the team tackled was the opening intimate performances of Durotan together with Draka. The actual shot is the close up on Durotan “and what was so great was being able to show Toby Kebbell the shot and say ‘you can play this character the way you want to..you can do the acting just with your eyes and the performance will come through’ and as a result everyone got very excited at that point,” recalls White. Kebbell is no stranger to motion capture having given a remarkable performance as Koba in Dawn of the Planet of the Apes.
In that remarkable closeup of Durotan, perhaps the strongest element is the eyes. The design of the eyes is very human in construction. “We did a lot of work to get the eye’s looking right,” says White, “making sure we had all the nice little tear line highlights and the reflection of the eyelid into the eye, and the right feeling for the iris with a little bit of that rainbow effect that one can see… but the number one thing that makes you buy them is the movement.” The attention to detail extended to the muscles around the eyes and the way the muscles move as the orcs looks and blink. There is a close correlation between the eyes and other muscles around the face.
The gaze tracking was significantly advanced from ILM’s previous work, Hal Hickel Animation Supervisor, ILM explained. “Honestly every time we’ve done a project like this – be that VFX or something like Rango, I remember all these discussions about eyeline. ‘The eyeline looks off’ or ‘he is looking right’ or the eyes are not converged enough,” relates Hickel. “Even in simple dialogue scenes when characters are just meant to be looking at each other, the eyeline seemed remarkably easy to break. But on this film we almost never had that discussion, the eye tracking was really good, really spot on and it tracked to the orcs really well.” This improved automation of the accurate eye gaze tracking meant that the animators could focus on other things, especially integrating the face track into the rest of the head and integrating it with the body capture.
White also complimented Jones for not piecing together performances. “There was no Frankensteining a performance together from different takes,” he says, “using say this take for the eyes and another for a smile- Duncan committed to the performances, and tried to stay really faithful to the performer.”
Hickel, agreed that when you have an actor “giving a performance there are a million interconnected things between the brain of the actor and what you see in a performance and so if you try and pull them apart into components and rearrange them you get a weird result.”
The team did a Lightstage capture scan of Rob Kazinsky and used his eyes as the basis for the eyes in the film. For all the great reference and scanning work that was done, White particular acknowledges the lighting work of ILM’s Tom Martinek (Sequence Supervisor). “He lit that shot and did a fantastic job of getting the highlights in the right place to bring out that character,” says White.
What the audience sees in Durotan is almost entirely from the facial solve and not from hand animating a performance. The capture was done with a stereo pair of cameras mounted on head rig. Duncan said in the same Hitfix interview that “hanging on that shot as long as we did was really a start of that job of getting the audience to empathize with a father, a husband, his baby that’s on the way, and his people who are really in this critical situation where they need to find a new home.”
The on-set actors wore a helmet camera system developed by Technoprops, consisting of a pair of stereo cameras that delivered two streams of video. “Animatrik’s role in the facial animation was in the post process tracking of the data that came off those cameras,” explains Animatrik’s President and CTO Brett Ineson.
The stereo cameras meant that the team could track the data in 3D as opposed to 2D. Animatrik would track the facial marker set in 3D and then deliver that data over to the ILM who then used their proprietary facial animation solver.
In summary the face character pipeline was:
- Lightstage scan of the actors to build the model and gain texture information (at Lightstage LLC).
- Medusa facial pose scanning to build the FEZ library poses that the system solves into as the ‘face space’. (FEZ library is built from Medusa. The FEZ rig/shape library is ILMs FACS implementation.)
- On set, a stereo camera head rig is worn and gets solved as a 3d point set by Animatrik.
- The point set is then handed over to ILM, who solve those points into the face library space to drive the rig.
- The first rig is the actual actor’s face and not the orc’s face. Retargeting comes later once the performance is verified between the real actor and their exact digital duplicate face.
- ILM’s SnapSolve is then used frame by frame to snap and align the solved performance to the exact performance (per frame shapes . “That gave us a lot of extra subtlety,” commented White.
- Retargeting the performance to the orc is next. The orcs and the actors share the same mesh for easy translation.
- TDs then make adjustments to allow for the retargeting from normal lips to lip sync with tusks. They also add corrections for eye line/ eye focus point (as the orcs eyes are further apart than their actor performers).
- The rigged face sequence is then handed to the animators for “them to add the final 10%”
- The face is integrated beyond the ‘hockey-mask’ as the capture process does not solve for the neck or ears, etc. Sometimes the forehead is restricted by the helmet and, if so, witness cameras are referenced.
- Body motion capture is then refined and adjusted
- Muscle metering is added in Maya. This is the timing of the flex of the muscles and the timing of the firing of the muscles (one flexes a muscle before one moves – muscles lead movement).
- Hair, cloth and fur added.
- Lighting is done.
- Rendering was done in RenderMan using the Reyes algorithm (in part due to committing to the render pipeline 2.5 years ago before RIS was an option).
White commented on the remarkable job that Brian Cantwell, ILM global facial lead, did in designing and implementing the facial pipeline system. This was key as the film ended up with almost two hours of on screen facial solve.
One of the complexities was creating accurate lip sync, given the orcs have tusks. Interestingly, this was solved by both animation fixes and a tusk deformer that was started at PIXAR on Monsters University. ILM took over PIXAR’s work and continued developing it.
“It was essentially a push out deformer,” explains White. “It would take whatever the lip animation was doing and push it to the outside of the tusks, but still allow the lips to slide over the tusk itself.” Central to the problem was being able to make the right phonemes or, more accurately, the right viseme. Often, several phonemes correspond to a single viseme, as several phonemes look the same on the face when produced. “We had discussions about how they will make a P or an M sound.. but you know, what we found is that amazingly, a lot of that could just happen in the middle (of the mouth) and it would still be okay,” White adds.
Even harder than the visemes for consonants was making the orcs smile. After all, a smile has the corners of the mouth go up, and on the orcs that part of the lips and face is under considerable tension from the volumes of the tusks. Animation supervisor Hickel agreed. “It was interesting that the first smiling shot we had was in the opening scene with Durotan and a pregnant Draka… and the first time we saw it we were like, ‘Smiling orcs are kind of scary!'” says Hickel. “But then, interestingly, Duncan’s (Jones) take was that it was okay and that it was fine if they did not look completely cute smiling. And I completely agreed with him….but it was definitely a challenge… the tusks made things much harder – so honestly, our jobs were made a lot easier when they didn’t have dialogue or smiled.”
For Hal Hickel, the role of the animators in dealing with the orcs was not to look for things to change, but to look for things that were not translating to the orcs properly. The other issue for the animators was the body capture. For example, in the same scene at the start, Durotan and Draka have to hold hands, but the physical size difference of their orc hands meant that the real actor’s hands were completely in the wrong place, relative to the huge orc bodies.
So much so that the animators had to almost entirely recreate the actors movements with key frame animation. “The body capture was virtually unusable directly,” relates Hickel. “We had to push their bodies apart to accommodate their bigger size, and rework how they held each other’s hand and even when he reaches over and puts his hand on her belly. Unlike an orc standing or fighting, where the body capture worked, this was all just a ton of hard work by Kevin Lan and Shawn Kelly (ILM animators) to make it look real”.
ILM has dealt with large muscle bound characters before, most specifically Hulk. In comparing the two, there are similarities in the challenges of animating them to look heavy and grounded, and yet move swiftly. Of course, the difference between them is in the battle sequences where the Hulk leaps and virtually flies, compared to the orcs who are much more normal in their range of movement. The Orcs’ thick bodies, compacted solid torsos, and giant muscles translates into limited general movement. If an actor hears something behind them, they can turn their head with a much greater range of motion than either the Hulk or the orcs can – their arms need to be further out from their bodies at all times, etc.
To make sure that the actors incorporated these aspects, all of the actors went to orc school and were trained by Terry Notary. “He was awesome as the movement choreographer,” explained Hickel. “He did the Planet of the Apes films and LOTR with Weta and he put all the actors through this sort of orc bootcamp to show all of the ways of moving that would suggest a much larger bulk… and in lots of interesting ways, for example you might think they need to stomp around to look bigger and heavier but it is the exact opposite. Soft foot falls were key to making them feel really big – and with a sort of conservation of energy.”
Jason Smith, Visual Effects Supervisor: ILM explained to get some of the face rendering looking correctly they took staff members outside the ILM building and did a special shoot to try and dial in just the right level of SSS. “it was really tricky at the beginning were walking the line between latex and silicon.. just trying to get the orcs to look like skin. In the end it just came down to painstaking work to get just the right amount of scatter and when you get the scatter into the shadow it really does bring warmth to the character,” he explains.
Motion Capture
The motion capture and virtual production technology for the production was supplied by performance capture company Animatrik. “Warcraft was the largest virtual production Animatrik has ever been involved with,” says CTO Brett Ineson. “The production had eight different stages over five months of shooting. Some were 300 feet in length, and there were even outdoor stages where the team had to film in the rain!”
Animatrik’s camera tracking, motion capture suit and simulcam technology enabled complicated and realistic on-set performances from the orc performers.
Compositing orcs in real-time
“We supplied tracking cameras, tracking weapons and props, all integrated with live action sets and live action actors, and then integrated those CG sets and actors together,” explains Ineson. “The idea was that as we were filming you could see everything happening through the eyepiece of the movie camera – all the backgrounds, orcs and creatures composited in real-time with our tracking tech.”
“It was just so complicated,” explains Ineson. “It’s one thing to capture performances but when you have nine orcs fighting nine humans with moving cameras and all that action happening, the fact that the cameraman could see this all in his camera in real-time, it was so great.”
Animatrik deployed two kinds of motion capture camera technologies: NaturalPoint and Vicon cameras and a SolidTrack system. The first tech setup was coined ‘outside in’ and relied on capturing the motion of performers in motion capture suits along with the main film camera. This provided a real-time simulcam composite of the characters and their backgrounds.
When there was no motion capture camera equipment on set, or the actors walked inside sets or areas where the cameras could not see them anymore, Animatrik used their second or ‘inside out’ approach.
“We’d mount a small vision survey camera to the movie camera,” explains Ineson. “This camera would integrate with a system called SolidTrack”. The Solidtrack system would lock onto each pixel it saw and calculate its 3D position. The team could then work out the offset between the vision survey camera and the actual movie camera for the simulcam view.
“Having that simulcam view was so valuable for everyone on set,” adds Ineson. Simulcam helped with things like framing, since the orcs, for example, might be up to 12 feet tall. Knowing that on set meant the DOP could re-frame for the best angle.
Outdoor capture – even in a river
Animatrik’s motion capture tech helped solve another challenging aspect of Warcraft‘s outdoor shoots.
Motion capture outdoors has traditionally been much trickier than a controlled stage environment. The team used active marker suits fitted with LED lights to achieve highly accurate motion capture. “Because of the outdoor sets, light pollution and rain, we needed to create LED suits for the actors to wear because we knew we would have trouble bouncing light from the camera to do a traditional motion capture,” says Ineson. “The LEDs were synchronised to pulse in time with the motion capture camera and therefore not impact on the main photography. This approach allowed for precise outdoor performance capture, and a more realistic performance from the actors themselves, who could more realistically place themselves into the scene.”
Capturing the facial expressions on set was very beneficial. For example, at one point Draka jumps into a river to try and facilitate her baby escaping. This sequence was shot in a river in Vancouver in late winter, “so it was cold, and when she jumped in, we found in the facial capture this little wince as the cold hits her legs before she scans the environment,” states ILM’s Jeff White. “That little wince – I don’t know that we would have thought to include it in traditional key frame animation, or as an adr or studio capture – the fact that we are able to do that on location, with her right there reacting to her environment was another big part of the success of the approach.”
Haircraft
All the orcs have nose hairs, ear hairs, peach fuzz and “we used that a lot in lighting, to give little random hits,” says Smith. “If you have peach fuzz on the shoulders, you can get just a few of those hairs to catch the light. It almost looks like a mistake but it just puts a crack in the CG look. This was by far the hardest hair show we had worked on”. ILM had a team of 3 or 4 people working for a few months only on King Magni’s beard “and part of this we were really pushing our pipeline for hair,” relates Smith.
Interestingly, not only did the team need to do hair but a lot of fur. One may think that hair and fur would be done in the same way but, actually, the hair and fur are quite different in approach. The fur pelts that the orcs wear use fluid simulations to drive the movement and look.
“What it comes down to is how we model, create and groom the fur,” explains Smith. “What we did with the hair braids for example, is create polygon cylinders that shape the braids and then we populate the cylinders with hairs. For the fur, we model a fur volume and the modeller would make a surface for what the outside fur surface would look like, with all the tuffs and bumps. Then we would do a fluid simulation emanating pressure from the pelt – moving outward through the surface as a flow simulation and generate hairs that way”.
The ILM artists could paint on the outside fur surface where pressure could escape. They could paint little tuffs where the pressure could escape, then drive the fur to look a certain way at that point on the overall fur pelt.
The feathers on the flying griffins was hairs on hairs. The ILM team would grow a feather spine across the whole length of the body of the griffen. Each feather would get one thick spline grown for it and then that hair would grow hairs on it for the final feathers. The griffin was all rendered as fibres.
Environments
The Third Floor visualized about 30 sequences for Warcraft, collaborating with director Duncan Jones and VFX Supervisor Bill Westenhofer and their teams. The work ranged from previs and postvis, to virtual production prep and techvis, and to support the filmmakers in choreographing action sequences and camera movement, and assist in the human-to-orc-scale approach.
n some cases, the previs was taken to set and used as a guide for the shoot and for virtual production. The Third Floor’s team choreographed their own mo-cap for previs, utilizing a lower-res solve of final CG characters together with actor mocap to create an accurate silhouette for postvis.
The environment work was not rendered in RenderMan like the foreground orc, but instead done in ILM’s standard environment pipeline with 3DS Max and V-Ray. While there was some lighting environments shared between the character lighting team and the environment team, they were primarily quite separate.
One of the challenges the team had was just how many environments were required and how different each environment was, from deserts to hanging sky castles. There were extensive sets but many of these still needed immense set extension.
FX animation
In WOW, magic plays an important role. For Warcraft the film, the idea was to have the magic very much like the MMORPG and not just be a flash or instant effect. To achieve this, the ILM team carefully crafted the spells to look very different from magic that had been done before and we tried to embrace the game magic components. “But it is in our terms fairly low res, so we looked at what colours were used and what are the basic structures”, Jeff White explained.
The team noticed that a lot of the spells in the MMORPG had steps to them so in the film, the magic takes time to perform, so it isn’t like Harry Potter style magic, with wands that instantly fire. Runes need to be drawn out and ILM was careful to make these runic divinations appear to have their own internal logic.
“It was really fun and challenging to work that out, but one of the things that really helped is that in the WOW world they have this idea of all of these symbols and line work that appears in the air – and I think that really helped it look different,” explained White. “And in my brief examination of the WOW discussion boards, people seem pretty pleased with the depiction of magic and those are the magic critics we wanted to please the most.”
To get the contact lighting on set, there were lights set up around the actors for when the spells were big. Prior to those big practical lights, the actual actors had hand lights that were remotely controlled. This allows everyone on set to see when the magic would appear and to provide invaluable contact lighting on not just the actor doing the spell, but any other actors near them.
The FX animation also included smoke, fires, dust and the portal, but White believes that the magic fx animation was the hardest part of the work the fx team had to do.
Crowd simulations for the wider environments were provided by Hybride in Montreal. ILM remained focused on the foreground orcs, which would be integrated with Hybride’s orcs in the vast environment that was laid out by ILM, all co-ordinated via alembic caches. “It was really complicated but Hybride did an amazing job and they built their crowd pipeline from the ground up for this show,” points out White.
Spoiler Alert : The final (animation) word…
For the animation team working on an action film with orcs, there was a surprising amount of subtle performances required. One example that many viewer may have missed is in the tent scene at the beginning. Draka has something between her knees at the beginning of the sequence. In dailies, Duncan Jones asked Hickel if he could take her belt that had been a part of her costume around her waist and move it between her knees. Jones explained that when women are pregnant, a lot of times their hips will get sore and they will put a pillow between their knees and it helps them. “I was like wow that is so sensitive, – so we did it… if you look careful she pushes it aside as she rolls on to her back, … that kind of attention to natural living human detail was terrific and we know one doesn’t get to do much of that normally.” explained Hickel.
One unique aspect of the film was that both Durotan and Draka were killed. It is very rare to have a CG character acting and even rarer to have that CG character get a death scene in a film, yet alone two such scenes.
“Yeah that was pretty shocking to me when I read that on the page.. I was like, what… wait, what? They actually die?” Hal Hickel recalls. “My son is an avid Warcraft player but I was not up with the lore so I wasn’t ‘Oh, of course he died’… yeah that was rare. But you know, the thing that was the most rare for us was just having a character sit on camera for 5 or 10 seconds doing nothing but breathing and being alive… we never get to do that. I mean we get to kill characters from time to time, but we got to have a super closeup on Durotan and he is just sitting there breathing – just his eyes moving – that was super special.”
