Band of Misfits: Dneg’s pirate adventures

The latest Aardman and Sony Pictures Animation project is The Pirates! Band of Misfits, a Peter Lord film that utilized a mix between traditional animation, CG techniques and visual effects enhancements. Aardman relied on an expert team of designers and stop motion animators, plus an in-house group of visual effects artists and outside vendors to create the film. Helping to realize aspects of the facial animation was Double Negative, also contributing rig-removal and compositing for various shots. We chat to Dneg’s CG supe David Vickery.

fxg: Dneg helped to remove the ‘cut lines’ on the puppets – what was the workflow for this?

Original animation plate.
Final composite by Dneg.

David Vickery: Aardman developed a new technique to create the facial expressions for their characters on Pirates. They constructed their puppets as usual, with heads made from hand moulded plasticine. The lower sections of the head, including the mouth and all the required mouth poses were created with a series of interchangeable 3D printed parts. This approach not only sped up the animation turnover but also allowed the characters more diverse range of facial expressions than if the whole head had been plasticine. Rather than try to blend the join between the two head pieces before shooting, the removal of the seams (or ‘cut lines’ as we called them) was left to post production meaning that every shot in the movie required visual effects.

Dneg’s first task for any shot was to remove these cut lines. Aardman provided us with a Nuke toolset that would sample pixels from either side of a cut line on a frame by frame basis and then use an average colour to patch over it. Artists could define the area the that Nuke tool sampled from with roto shapes, giving them a precise control over the end result. Some characters also had joins where the head attached to the neck. Aardman were always very careful to disguise cut lines and joins behind beards, glasses, and clothing wherever possible but it was still rare to find a character that didn’t have some join or seam that needed a bit of digital help.

fxg: What was one challenging shot, for example, requiring clean-up work such as removal of rigs?

Vickery: Double Negative’s work on The Pirates! required some incredibly complicated rig removal. One of the reasons that Aardman animation is so effective is that the characters aren’t limited in their movements in the way live action puppets often are. They can fly through the air, jump, somersault, stand on one leg and defy gravity. Of course it requires a lot of clever rigging to achieve all this. Rigs ranged from exposed metal armatures supporting character poses to fine lines used to pull on sails or curtains. Shooting stop motion even gives you the freedom to completely swap rigs out from one frame to another.

– Watch an interview with director Peter Lord on the use of rapid prototyping to create the puppet faces in Pirates!, thanks to our media partners at The Daily.

One shot featured eight characters dangling off the side of an airship, strung out hand to foot in a human ladder with every character requiring its own pose-able rig to hold it in place. The shot starts at one end of the human ladder and tracks past each character to the top. There was rigging everywhere and the bright sunlight in the frame threw a perfect crisp shadow of the rigs right across the hull of the boat so all the shadows had to be repaired too. Stereo only served complicated the work further. Pete Jopling’s 2D team in the UK had to create many hundreds of 2D patches and paint nodes just to clean up one of the stereo eyes. All these patches were then copied to the other eye and had to be hand animated to match the ever shifting stereo divergence throughout the shot.

fxg: How were composites handled when placing characters shot on green screen into various plates? What were some of the stereo challenges for this?

Vickery: Shots were often supplied as multiple passes on green screen to allow the Aardman animators to work with smaller numbers of puppets per take. This approach also gave the DOP the space to hero light foreground characters within the tight confines of the stages without having to worry about affecting other elements in the shot. The shots were ‘filmed’ using digital SLR’s mounted on computer controlled motorised dollies. The operator would shoot the right eye and the camera would automatically move and re-frame for the left eye before shifting along to the next frames’ position. Camera moves could be perfectly repeated take after take to create multiple passes (including clean plates) that lined up perfectly. These camera moves had none of the discrepancies or smoothing inherent in full scale motion control set-ups. The small scale of the sets made it hard to light green screens evenly which meant that some shots were very difficult to key.

Original plate.
Final shot.

Great care was taken to make sure all the individual passes were graded correctly when composited together. At this point additional effects like moving candle flames and chimney smoke were added to bring life to backgrounds and any slight distractions that would interfere with the stereo experience could be fixed. Typically these would include glints and twinkles that might only be seen in one of the stereo views – an ever present problem in a story containing pirates and their sparkling treasures. Small errors that are an inevitable part of a stop motion production could also be addressed.

fxg: What were some of the challenges of having to match to a stop-motion and plasticine look for any of your visual effects work?

Vickery: We had to liaise really closely with Aardman to work out the differences in our pipelines. Aardman shared many of their own in house compositing tools, much of which Ian Simpson was able to integrate into Dneg’s own Nuke tool set creating an efficient pipeline for us to comp within. One aspect of this workflow in particular provided a new experience for our 2D artists who are more accustomed to working on live action projects. Aardman’s characters would only be animated on every second frame meaning that any roto shape or animation position needed to replicate their move, hold, move, hold frame to frame pattern. Aardman’s expertise in these matters ensured that we were up to speed remarkably quickly and that our work was consistent with their own.

Details like the animators’ thumb prints on characters, quirky set shifts or random glue spots contaminated every shot. In order to retain the hand crafted soul of the work; Jody Johnson (Dneg’s VFX supervisor) would study each frame and decide which of these errors to selectively leave and which to remove. It was important to Aardman that the work was finished to an incredibly high standard but still retained a hand crafted look.

Original plate with rigging in place for the characters.
Final shot with rigging removed and background inserted.

From a 3D perspective the largest challenge was the initial look development of the CG characters. Aardman provided us with their models and textures along with photographic turntables of the actual puppets against a neutral grey environment. The CG characters needed to be carefully lookdev’d to match the practical puppets. We had an HDRI environment from the photographic turntable and used this to light the CG models. The show provided a perfect scenario to production test Double Negative’s new ‘V4’ Renderman pipeline; a physically plausible shading system that relies almost entirely on ray-tracing. Initial render times were a little longer than we were used to but the results were fantastic. Small subtlety’s in the way the light flooded and bounced around the models provided a near perfect match to Aardman’s practical puppets and we were even able to introduce tiny amounts of subsurface scattering into the plasticine to further perfect the look. We were so confident of the new shading system that we didn’t once refer to Aardman’s own CG turntable renders. We had matched the real thing and we knew that would be more than good enough!

fxg: For the CG character work, such as crowds, can you discuss how the characters were modelled and rigged and then how Dneg assembled the shots?

Vickery: The second major part of our work on Pirates! was the integration of CG characters into live action scenes. The shots range from intimate scenes with a handful of mid-ground CG doubles to entire audiences with hundreds of characters. The Pirate of the year awards and the Scientists convention are great examples of these. To maintain consistency of style across the production all character animation was executed by Aardman, who would then export geometry caches and send them to Double Negative where we could reconstruct their scenes. Aardman provided HDRI maps for each new setup. For wider shots containing many characters they would position static puppets across a set and shoot lighting reference for Dneg 3D artists to match CG characters to. Dneg London set up lighting rigs using a combination of the supplied HDRI maps placing additional area lights to simulate the diffusers and bounce cards used on set. Small point lights were added to replicate lamp or candle positions in the original plates.

Original plate.
Final shot.

3D Artists then used Nuke templates created by 2D supervisor Oli Atherton and Mike Allen to create simple test comps that matched as closely as possible to Aardman’s lighting ref. We had to work hard to match the qualities of light and shadow in the scans. Wherever CG characters needed to connect and interact we had to rebuild the practical sets in 3D to cast shadows and bounce light correctly onto our characters and help integrate them into the plates. The Pirate of the Year sequence provided a particular challenge in trying to create the feel of a large crowd in very dark lighting conditions. Oli’s team did a fantastic job at the compositing stage to craft the final look of these shots. It was great to see such a strong look created, picking out details and movement in the shadows with rim lights, letting the rest fall into the shadows to subtly establish each CG characters presence in the back corner of the tavern.

fxg: Can you talk about how you communicated with both Aardman, and Dneg’s Singapore office on the production?

Vickery: We worked on a total of 393 shots for The Pirates!. Under any circumstances you’d need a really efficient feedback and data handling system to deal with that much work. The majority of the visual effects was carried out by the Singapore office with Jody Johnson remote VFX supervising from London. Jody worked very closely with Aardman supervisor Andrew Morley and would then relay briefs to Oli Atherton in Singapore via Polycom video conferences. It was really important to us that Aardman considered our Singapore team to be part of Dneg as a whole and not a separate facility on the other side of the world. Singapore lead 3D artists Cori Chan, Sonny Sy and Leah Low took charge of hero sequences and gave progress reports to London during cineSync sessions each morning.

Our shot databases between London and Singapore are completely synced so we can see everything they do and vice versa. In the end even the time difference worked to our advantage. When we got in first thing in the morning it would be midday in Singapore and they’d be ready with loads of questions. Then London would pick up the baton and be working well after our Singapore crew had gone home for the night. Producer Clare Tinsley could schedule London crew to fix problems for Singapore whilst they were fast asleep and be ready for them when they returned to work the next day!

Images copyright © 2012 Sony Pictures Animation Inc. All Rights Reserved.