Shaping Water at Mr X Inc.

Guillermo del Toro won best director at the Golden Globes for his latest film, The Shape of Water. Trey Harrell at Mr X Inc., was the Digital Effects Supervisor on The Shape of Water. Harrell and the digital team at Mr X Inc. provided far more of the performance on screen than most people might think.

The design of the creature was personally financed by the Director for several years before the film was green lit and funded. Three years before shooting on sound stages in Toronto, del Toro hired Guy Davis and Vincent Proce to begin design work on the lab. The next year he hired two sculptors, David Meng and Dave Grosso to begin working on the design of his Fish Creature at his Bleak House workshop at his own expense. Del Toro was so committed to getting the creature right, he financed the design of the creature from his own pocket, spending hundreds of thousands of dollars in a process that took some 9 months of gestation. “I knew I wanted the creature to feel real but at the same time for it to be beautiful which is a very hard line to tow,” confesses del Toro. “I knew it was going to take a long time, so I didn’t even put it in the film’s budget. This is truly the hardest creature design I’ve ever done.” Early on, he assembled a crack team of artists who are pros at infusing life into the imaginary, including Shane Mahan of Legacy Effects.

While the work on the suit and physical on set prosthetics are incredibly impressive, so too is the digital work that helped bring the character to life. While del Toro embraced the notion of a man in a suit look for the character, in reality most of the time the audience sees the character’s face it is digital, and additionally many of the water shots are CG.

The film’s visual effects supervisor, was regular del Toro collaborator Dennis Berardi. In 2001, Berardi founded Mr. X Inc. The highly respected studio has grown from a small eight-person shop to one currently employing over four hundred staff in Toronto, New York, Montreal, and Los Angeles. Over the years Berardi and his team have garnered numerous awards for their achievements in Visual Effects, including two 2017 Emmy nominations in the Outstanding Special Visual Effects category for Vikings and American Gods.

When Berardi read The Shape of Water he says he was overcome with excitement about what lay ahead. “I could see how my team could make a real contribution to the film’s core emotional components and I was all in,” he recalls.

Berardi began by creating an exacting digital double of Doug Jones in the prosthetic suit. “Guillermo wanted the creature to not only be able to emote like Doug but to also move underwater in a certain way, so we did a lot of early movement tests with our Animation Team at Mr. X and we got to the point where we could do a digital version of the creature that could match up with Doug’s beautiful performance,” he says.

“Our hope is that the audience can’t distinguish at all between the digital version of the creature or the Doug Jones version. Doug’s performance informs our animation and I think our animation has also informed how Doug was photographed. If the audience can’t tell which is which, we will have succeeded,” says Berardi.

 

Start of the detailed digital work.

The film started principle photography in September of 2016, and the Mr X team “went to camera about three months prior to that,” explained Trey Harrell. “Dennis Berardi (Mr X), Shane Mahan (Legacy Effects), Mike Hill (creature sculptor) and Guillermo del Toro had all been collaborating behind the scenes on how to best bring the amphibian man to the screen over an extended period prior to that. I came on board when we started camera tests prior to principal photography.”  As part of the elaborate plan to get the film made for the budget that it had was to film it on the same sound stages of Guillermo Del Toro’s TV show The Strain.  “I understood The Shape of Water needed to cost under $20 million, because that allows them to market it for what it is,” Guillermo del Toro told Variety.  Part of the solution was to shoot “on the Strain sound stages during hiatus – they tore down all the strain sets that they had reserved, put up the sets for the Shape of Water. Shot it, tore them down, and put back The Strain sets. As it was all in Toronto it was all quite possible, plus Del Toro lives in Toronto for part of the year,” adds Harrell.

In those early camera tests before principal photography the team did black light tests and other experiments to see what would work.

Xscan face capture.

The VFX team used the Mr X Inc, Xscan photogrammerty rig which has about 56 DSLR cameras all on USB remotes in a mobile rig. Sometimes it is used on set, but normally it is in the Mr X studio. The rig allows “us to capture extremely high res geometry better than LiDAR could ever do, and textures as well – for characters and creatures,” explains Harrell. The Xscan can be reconfigured for face only mode, or full body mode.

The system does not use structured light or polarized light “but I can tell you that we have looked into it for the next developments…moving forward it is extremely likely the next version will have some sort of structured light and cross polarisation,” comments Harrell. The rig mainly uses it for photogrammetry modelling but the rig also provides a base albedo / diffuse texture from the rig – but not the specular pass, currently, those are generally painted by hand. The rig has an enormous softbox over the top of it which provides a wide diffuse light.

The team’s aim in doing these scans on The Shape of Water was not to get an animated actor with accurate skin movement etc. They scanned Doug Jones in the creature suit in roughly 5 poses each time at both the beginning and the end of the shoot. Mr X likes around 5 poses since they believe it covers the face well, including areas on top of the head that are often missed in photogrammetry. They also scanned Doug Jones 48 separate FACS poses, out of makeup. These FACS Range of Motion (ROM) poses are the basis of Mr X’s Blendshapes, poses and rigs for the amphibian man.

Animated Face of Amphibian Man.

Harrell estimates that 90% of the tme the audience sees the Amphibian Man in the film, they are seeing a digital face, primarily from the top of his forehead to the top of his upper lip. “It is entirely CG from the top of his head to the top of his upper lip, in every shot in the film”. They also used the ‘talking animals face digital makeup’ technique to allow their compositors to paint where they want the composite to be, and feather in around the seams. The reason this needed to be decided on a shot by shot basis was that the actual suit on set changed slightly every day. “Every day on set the suit changed…it fitted practically on his body on set differently. We had an ideal version of the Amphibian Man… but we had to blend all the different elements together,” Harrell explains.

The team did a full animation of a CGI Doug Jones along side the digital Amphibian Man. The team did what  Harrell called ‘Rig Aerobics’, hitting all the ROM poses. The team did exactly 1 to 1 of the same ranges of movement using re-targeting from one to the other. “That was one of the most illuminating things for us, in terms of keeping a natural performance, on a non-human face,” comments Harrell. The team did a lot of work with tension maps and minor wrinkles that a form latex suit could not, “and a lot of that was to increase the emotional range. You take so much for granted in your forehead folds, in nasolabial folds (smile/laugh lines) and of course our romantic leads in the film are both mute.”  The team made the decision early on to use a pose based tension matte setup. This system of micro wrinkles allowed the character to produce expressions and range of subtle emotion an actor could never deliver from under form latex.

 

Skin or scales?

The next issue was the nature of the creatures ‘skin’. There was no reason for the light to produce strong sub surface scattering (SSS) like human skin, one could even have argued for fish like scale reflective profile. “Our solution was to make his skin react like latex rubber”, comments Harrell. “We deliberately did not go for hyper real, we went for stagecraft, which is a very different approach than many productions would have done, but I am absolutely convinced was the right decision for this show”.

For Harrell, he believes that Mr X completely committed to the practical crafts, “Honestly it is my jam… I love doing suit argumentation more than any other thing in this entire industry and I have gotten to do a lot of it with Guillermo – because that’s his MO, it’s how he approaches film.”

Fully Digital

There is also a few shots such as when he hisses at the cat that the team went further and did full head replacements. There is also entirely digital shots, for example when the Amphibian Man is suffocating in the bath before the salt is added – the creature was “entirely digital, as was the bathtub, – that is all CG,” Harrell adds. Mr X is keen to not give the gag away, today as much as during production, but it has not escaped their notice that as people rightfully discuss the practical suit and makeup in the film, the press have published photos of the digital CG work done by Mr X, assuming it was practical on set work.

Also most of the underwater swimming Amphibian Man is fully CG, since no actor or diver could move at that speed. Crafting the underwater movements was a research-intensive process that involved looking not only at Olympian human swimmers but such aquatic species as sharks, puffins, otters and penguins. “We looked at anything that moves very gracefully through the water in order to base it all in reality,” Berardi explains.

For the CG rendering the team used primarily Mantra, “Our entire lighting and rendering pipeline and sim pipeline was entirely Houdini based,” comments Harrell. Inside Mantra, the team developed and uses their own custom path tracer, with its own SSS, with per light separation Sub surface contribution. This is now standard in Mantra but wasnt when the team developed their own. The Mr X is closer to the newer Pixar SSS model, than the new Mantra. The team did not use backscatter on the Shape of Water, but the team expect to use it on the next show they are starting now. The translucency of the webbing took full advantage of their custom path tracer and SSS. Side Effects visited Mr X around the time of their first turnover of shots and Harrell requested “perfect control over compositing through up to four layers of refracted glass and water condensation – and be able to separate out primary rays – and they worked with our TDs a solution, so we could render out in one pass and still have compositing control.” In addition, the team extended Cryptomatte so that it would work through refraction and give back primary rays. “We doubled down on Cryptomatte on this show, and being able to pull arbitrary mattes through arbitrary refraction depth was absolutely a god send,” adds Harrell.

Dry for Wet

Not all the underwater scenes involved actual water.  For example in the end sequence the majority of the shot is actually filmed dry for wet. While the amphibious man swimming is CG in the wide shots, for the Elisa Esposito (Sally Hawkins) is hanging on wires. The exception is the closeup kiss where the Amphibian Man’s head only is fully CG. This sequence and the original opening dream sequence were both filmed Dry for Wet with heavy smoke and wire rigs for the actors where appropriate. These sequences also had some added digital elements to sell the shots, but they are based on beautifully lit and art directed practical photography. One interesting aspect that could not be done live in either sequences is the hair of actress Sally Hawkins. In both sequences her hair is replaced with digital floating hair.

The team played with the creature’s colors, shifting them with his mood, and often replaced the odd fin or breathing gill. “Anytime you see the fin move – it’s us,” comments Harrell.

 

 

---

This opening sequence was all done with a deep comp pipeline in Nuke, but as soon as the comp was done the team cleared the deep cache as the data overheads were so demanding, and intensive. Also the first reveal in the lab when the tank was rolled in and some parts at the end were done with a deep Nuke pipeline.

In addition to the creature, Mr X did other work such as adding to the bioluminescence on the amphibian man’s body and set extensions and invisible VFX work.

Looking back over the project Trey Harrell comments that the Director Guillermo del Toro “always had the best eye in the room, and we always knows what he wants, – and he is pretty inspirational to work for. He just loves sitting down with the artists and he brings out the best in everyone who works for him.”