One of the biggest challenges faced by Sony Pictures Imageworks’ visual effects and animation team on Sony Pictures Animation’s Hotel Transylvania 2 was the unique style of movement that director Genndy Tartakovsky wanted to impart on the characters. Explored also on the first film, this exaggerated motion gave an incredible level of extra effect to the performances, but also meant that the artists could not always run typical physics sims for things like cloth and FX. Visual effects supervisor Karl Herbst runs through some of the those challenges, and we present two exclusive behind the scenes videos showing how Sony dealt with them.
fxg: What did you learn on the first film about the style of animation that carried through to the sequel?
Herbst: What we learned more than anything on the first film was that we couldn’t build rigs the normal way we wanted. If we put enough effort into every character’s rig to get it there we would basically still be building rigs today to do it. So for this second film we knew that a lot of the posing ended up being sculpted by the animator. The poses that we get from animation are incredibly pushed and kind of destroy anything you would rely on in terms of physics.
So, skulls are changing sizes, limbs are moving longer and shorter. The animators even put in pre-smeared blur in their performances which you can really see in the motion of hands and arms. They’d stretch out the hands so that you get incredibly streaked blur. We actually modified our motion blur on this film to compensate for Genndy’s more 2D aesthetic which is a sharper leading edge and more of a smear behind the object – which in some cases even meant there was a print of a sharper sense of the object being the hand or the face inside that sharper leading edge. He really doesn’t like the physical lens in terms of motion blur and shutter, so we were always trying to compensate for that. We are trying to add smearing inside the animated shapes and then we send that to a simulation software for cloth that is relying on physics, and there’s nothing there to rely on.
fxg: Since you have to do so much hand-sculpting, what’s your pipeline like for doing this?
Herbst: It starts with running through layout into animation. When we get into the simulation, cloth for example, the artists run a first pass to figure out how bad are things going to break and where are they going to break. We developed some tools on top of the normal cloth sims that would re-build the sim garments depending on what was happening in the animation.
An example is that the arms are being stretched out, but we want to keep the cuffs at the right lengths, or say Johnny’s shoulders go from being really wide to really narrow which would turn the shirt into a potato sack on him. But we wanted to keep his default form close to his model, even though his shoulders went away. So we created a way of doing a run-up to these poses that is re-scaling that sim garment along the way to try and get as close to what it needs to be, so that it doesn’t turn into a rubber suit. We still go for flow and drape, but it’s moving correctly from short-arm to long-arm, shoulders-in, shoulders-out, and we don’t have to re-build the sim garment by hand trying to compensate – which is what happened on the first film and why cloth sim sometimes got dropped out of certain shots because we couldn’t handle doing that much hand-tweaking on every shot.
As you get farther down the pipe with dust and other fx, we would approach the same idea – so a character might hit the ground and create a dust plume in a certain way, but we might want to put some counter-weight to that so it feels like it’s got torque or more Wily Coyote to it and gave it a little bit of stylization. It’s the same with FX artists – they do a first pass of how things with our tools, and then they go in and hand-tweak and put forces in in places we need to get that extra shaping into whatever we’re trying to do.
Lighting would also have it’s issues with the style since sub-surface scattering relies on volume to some degree. If your hands are going from being really thick to really thin, all of a sudden you’re going from what you set your lookdev to be to, suddenly having Gumby and everything is a think sheet with light just traveling right through it. So again the lighting artists are going in and hand-tweaking material properties or re-adjusting things in comp to balance those things from pose to pose and shot to shot.
fxg: That seems like an incredible challenge – how did you adapt to doing it like this?
Herbst: You can’t really build a rig or a facial expression and assume those will be the ones used over and over. Genndy re-invents particular poses or facial expressions based on the moment. If you’re picking up from one pose to the next pose, he might interject a pose we haven’t seen before. That’s one of the great things about working with him on film, and he sits behind a Cintiq during the animation review and thinks about how would I have drawn this? And then the animator would try and get as close to that drawing as they can, hoping that they don’t destroy the physicality of the character in getting there, trying the best to get to that mark and get a usable product that can go down-stream to the rest of the departments.
fxg: How would Genndy give you reviews and make suggestions for tweaking?
Herbst: Our facility is split between LA and Vancouver – Genndy would come into one of our screening rooms in LA and be connected to everyone. He has the shot playing back on a Cintiq that’s in front of him – he has a main screen and a Cintiq – and we have multiple cameras on him as well because a lot of the time he’ll act things out along the way. We have a wide camera and a close so we can see his facial expressions. The animators are picking up the feeds for all of that as they’re picking up their dailies.
Genndy would draw pose to pose and re-compose a shot for animation. Those lines would then be saved out in a way that we could actually load the lines up in Maya and have it as an overlay inside the session for that particular frame Genndy drew it on. So the animator can actually try to line up on that particular shot. We’ve even tweaked the tool since so that they can see the line being drawn, actually how he drew them. This line first, this line second, and went over that line again, here, say. Which usually means he’s trying to reinforce that that’s important to him. It then plays back to the animator and they can see, say, oh, Genndy’s really worried about this part of the silhouette and then they can line up on that particular part.
fxg: Can you talk about one of the moments in the film where Genndy would do these kind of draw-overs?
Herbst: There’s a moment in the film, in the battle where our lead characters, Mummy, Werewolf and Frankenstein all yell and scream when they’re about to run in. It’s the exaggerated Mummy scream that Genndy was adding to. Genndy would kick off on a sequence and those shots would get cast out to an animator, the animator would take a rough pass at it and block in what they thought Genndy wanted. They would be the first showing, and he’d see that and start his over-drawings – move this here, stretch this here – he’d show us where he wanted to really push or re-time the actions.
We actually filmed the animator using the rig as far as they could to get to that pose, and then showing how much was done using artiasn tool brushes in Maya to re-sculpt and push the clay around to get the shape he really wanted. It actually took about 3 hours of animator time – that’s how much artistry went in to tweaking the animation that’s on frame for less than one second.
What’s hard about this style of animation is that everybody has to be more of an artist at every step along the way down the pipeline, rather than just heavy artistry at the start and relying on heavy physics and sims in the middle. Everyone along the way has to be in tune with the style and play to it.
The Mummy shot was a really great example of where Genndy really cared about how it looked – he’s only on the screen for a split second, but the amount of artistry that goes into sculpting this one particular pose is extraordinary.