ILM is being honored this year in both the technical and artistic sides of the Oscars; after all the Academy is the 'Academy of Arts and Sciences', and ILM regularly features on both sides of that equation. This year they are being recognized for their work with PhysBam, their Shape Sculpting System and for their work on the second Captain America film. The Sci-Tech Awards were handed out last week and the Oscar for Best Visual Effects is being awarded on Sunday night.
fxguide attended the Sci-Tech Awards in LA and spoke to the teams. The event at the Beverly Wilshire Hotel was one of the best in recent times, returning as it was to this hotel, and being hosted by some of the best guest presenters, perhaps ever. In addition to the Academy President, Cheryl Boone Isaacs, and Richard Edlund, the four-time Oscar winner and Chair of the Academy's Sci-Tech Committee, Margot Robbie and Miles Teller hosted the Awards. It was Robbie with her sharp wit and genuine warmth who charmed the 58 recipients who received the 21 awards. It was also Robbie who lead the most uplifting moment of the night when ILM’s Colette Mullenhoff received a standing ovation from the entire gathering for her work in developing ILM’s Shape Sculpting System. Mullenhoff was the only woman on the night to be receiving an award and she was genuinely overcome with the outpouring of support for a woman to be rightfully part of the awards. It is only to be hoped that more can be done to see other companies promote women in the science side of the Oscar awards.
Along with Mullenhoff, Cary Phillips, Nico Popravka and Philip Petersonfor were also award for the architecture, development and creation of the artist-driven interface of the ILM Shape Sculpting System. This comprehensive system allows artists to quickly enhance and modify character animation and simulation performances. It has become a crucial part of ILM’s production workflow over the past decade, and is extremely popular with ILM’s artists.
The ILM Shape Sculpting System
ILM has a long tradition of sculptural artistry - it’s a tradition that began with the original ILM Creature Shop and has stayed with the studio during the CG revolution. The result is that ILM’s CG characters continue to be borne from a strong history in sculpting. Which is why, when the studio looked to develop an efficient sculpting system around 10 years ago, it wanted to include ways for traditional artists to work with the latest digital tools.
The result was the ILM Shape Sculpting System, an artist-driven interface that allows visual effects artists to quickly modify character performances. The System was recognized this year at the SciTechs (the award recipients were Cary Phillips, Nico Popravka, Philip Peterson and Colette Mullenhoff) with a Technical Achievement Award.
The Shape Sculpting System is actually built into ILM’s Zeno framework (itself recognized with a SciTech Award last year). Clearly ILM has a strong set of digital sculpting tools, but the Academy considered the System an important innovation also for the way the studio has integrated sculpting tools into its deformation system.
“The thing that is special about the Shape Sculpting System,” says Cary Phillips, “is that we allow the sculptors who are very non-technical but have a great sense of physiology and anatomy to inject their artistry into the look of the characters.”
Phillips notes that the System is used at ILM in two main ways:
1. It is often used for corrections. “For example,” says Phillips, “the riggers will set up the skeleton of a character, and the skinning algorithms stretch the skin over the skeleton as it moves. Depending on how much time and effort they invest in that process, it works reasonably well or not so well. A crease may be in the wrong place, or the profile not right. The System allows sculptors to go in and fix the geometry on a frame by frame basis so that it looks just like they want it to look.”
2. It’s bread and butter use is in transformation shots. The most well known of these would be the transformation of actor Mark Ruffalo into the Hulk. “Traditionally we use the System for a digital character that starts out as a replica of a human actor and over the course of a few seconds, transforms into a non-human character,” explains Phillips. “That is notoriously difficult to describe using traditional rigging techniques. Our system is one that lets that be a sculptural process.”
The Shape Sculpting System was rolled out in a major way for Pirates of the Caribbean: Dead Man’s Chest, which featured the CG tentacle-faced character Davy Jones. “One of the things that happened at the same time,” outlines Phillips, “was making the transition from modeling using b-spline NURBS patches to sub-division surfaces. Phil Peterson (one of the SciTech award recipients) implemented ILM’s sub-division surface model and one of the key things that happened there was when you model a character with a network of b-spline patches there’s a natural segmentation where the model is divided up into a bunch of little pieces you can operate on independently. When you instead take all of those elements and add them into one big mesh, we’re talking about very large pieces of geometry. So our initial implementation was very slow. Phil did two major things - one was implement the basic representation of the geometry and then optimize the hell out of it. And make it really fast and efficient.”
Phillips says his own contribution to the system centered on the management of shapes - “the notion that these sculptural changes are small units that get organized and combined and stored and manipulated and flow through the pipeline.”
Nico Popravka considers his central contribution to the System to have been as “the main builder of Zeno’s deformation system which is a very generic way of applying deformation to geometry. It has application not only shape sculpting but also for special animation. It’s a general framework but for the case of shape sculpting, we designed and optimized it heavily to treat these shapes.”
Colette Mullenhoff was responsible for the UI and artist experience. “I had the pleasure of being embedded with the artists and working side by side with them early on,” she says. “This allowed me to see what they were doing first hand and what problems they had to solve. I could create prototypes they could try and we would iterate until we had the best tools for their needs.”
And thanks to the Shape Sculpting System, audiences have enjoyed the exploits of characters like Davy Jones, Hulk and so many more.
PhysBam, which was also recognized at the SciTechs, and the Shape Sculpting System have many things in common - one such link is Davy Jones - the incredible character work done for the second Pirates of the Caribbean film. PhysBam uses RBS and deformable (springs) for most common sequences. While it is not a fracture system, Stanford's Ron Fedkiw recalls it as a key starting point for the software, with Davy Jones' tentacles. "We built a rigid body system, we put all the springs in there for the sticking of the tentacles - we had all the attachments to attach different rigid pieces together because the tentacles were articulated and all those tools from Davy Jones made it really easy to build the subsequent building or whatever that would come later - and knock them down."
"We built such a really good system for that movie, and John Knoll (the film's VFX sup)," adds ILM's Brice Criswell, "so it wasn't hard to build a destruction pipeline for the subsequent movies." The destruction rigid system actually has an "equal component of a character dynamic system," he adds. "So the clustering system and the rigid dynamics system live right next to each other, and have the same underlying system - but they are used for entirely different things."
The PhysBam system has been used on nearly all the amazing destruction sequences ILM has produced from Transformers films to Terminator: Salvation, Star Trek and many others.
PhysBam is much more than just a collision and destruction tool, but this year the Academy was honoring work in this area and so along with such other great contributors from things like DMM and Bullet, the ILM pipeline was honored. The ILM-PhysBam system is a combination of the Stanford base research and code and a significant amount of ILM specific code. The collision level set code for the collision model is for example based on Fedkiw's great work in level sets but specific to ILM and is used even today. The system works with particles spread across the geometry colliding against implicit surfaces - "the contact collisions - the stacking, all that stuff came from a 2002 paper (from Stanford)," says Fedkiw.
One of the key contributors from Stanford was Dr Rachel Weinstein Petterson. She completed her PhD in 2007 and her doctoral research focused on simulating human character motion with applications in entertainment and biomechanics. In fact, Davy Jones is included as part of that thesis. Both Criswell and Fedkiw feel that they owe her a lot of recognition for her contribution and perhaps should have even shared with them in the award. If she had been included she would have joined ILM's Colette Mullenhoff as the only other woman this year, showing that at least at ILM some inroads are being made in including women researchers.
Petterson actually worked for a period at ILM, having a credit on Transformers, "and she worked with Brice on Davy Jones," explains Fedkiw. In fact, her not being included is of course due to Davy Jones not being a destruction sequence, and hence while the code laid the path for the PhysBam ILM destruction code, the award was focused on destruction, not PhysBam's contribution in the broader context. "Us as programmers see a much tighter connection between those two things," joked Criswell, "and we would have loved her to be included as the third recipient." Such is the nature of physics engines, things such as creature work share algorithms very closely with destruction tools and even the crowd simulation system at ILM uses the same base collision detection algorithms.
PhysBam is much more than just a collision and destruction tool, but this year the Academy was honoring work in this area and so along with such other great contributors from things like DMM and Bullet, the ILM pipeline was honored.
The two recipients for PhysBam are Brice Criswell and Ron Fedkiw, and they represent the academic research done at Stanford and the implementation and research done at ILM. Ronald Paul Fedkiw or Ron is a full professor at Stanford in the department of computer science and since 2000 he has been a consultant to ILM. As noted, he has already received a Technical Achievement Award from the Academy in 2008 for the fluid simulation work in PhysBam (shared with Nick Rasmussen and Frank Losasso Petterson).
Fedkiw has published over 100 research papers in computational physics, computer graphics and vision, as well as a book on level set methods which are used in PhysBam. And if that is not impressive enough, he used to be a competitive weightlifter with a personal best squat of 800 pounds (!), bench press of 555 pounds (!!) and deadlift of 735 pounds (FYI that is 333 kg!!!), all in the 198 pound weight class.
Brice Criswell is no under achiever either, having an impressive array of credits in R&D at ILM from The Chronicles of Narnia, Pirates of the Caribbean: Dead Man’s Chest to most recently Captain America: The Winter Solder - all from R&D serving the story and the incredible demands of ILM’s impressive effects work both in the USA and Singapore. His work at ILM for almost 16 years has spanned from being a lead developer for rigid body and crowd simulations to animation and modeling systems. The PhysBam system is of course central to the amazing destruction sequences that ILM have done for films like the Transformers franchise. fxguide featured this work in our 2011 Art of Destruction (or Art of Blowing Crap Up) story, which explains in more depth the level set approach pioneered at Stanford and implemented and extended by ILM.
From our Art of Destruction story: “At ILM, the PhysBam approach is similar to the 'standard RBS’ (Rigid Body Simulations) but with some differences and of course a huge range of ILM custom tools and very advanced techniques, all of which has been extremely relevant to ILM's roster of recent films including the huge destruction sequences of the third Transformers film which has some of the most advanced work to date in the industry."
Interestingly, the Academy singled out PhysBam for its early work, stating as part of the award description “the PhysBAM Destruction System was one of the earliest toolsets capable of depicting large-scale destruction with a high degree of design control.”
The first productions it was used on were Indiana Jones and the Kingdom of the Crystal Skull, and then the first Transformers films," notes Criswell. "The first couple of uses were just for fracturing objects and being able to direct fractured objects, being able to take a set of clustered rigid bodies and build an internal strain graph so you could evaluate the strains during the simulation, and also be able to impart forces on the structure so that it could collapse in particular directions."
Also honored at the event in the area of destruction and RBS were Erwin Coumans for the development of the Bullet physics library, and to Nafees Bin Zafar and Stephen Marshall for the separate development of two large-scale destruction simulation systems based on Bullet at DD and SPI. Also Ben Cole for the design of the Kali Destruction System, to Eric Parker for the development of the Digital Molecular Matter toolkit, and to James O’Brien for his influential research on the finite element methods that served as a foundation for these tools. You can hear our podcast with the DMM team winners here.
To quote Brice Criswell (ILM) in respect to the range of approaches to RBS - Bullet, FEA or say PhysBam, he states that: "I don't have an opinion either way that we should only be doing one or the other. I think all the approaches are all valid, and they are nothing more than different development paths or directions that people have taken. I think they are all valid directions and at some point I think that they are going to converge back to solving the same problem which is just trying to model material values on objects at very small scales."
In every industry there are odd metrics that the press grab hold of. In destruction sequences articles it is the number of "collision elements", but as ILM points out - really that is not the most impressive part of a destruction system. "Think about it," says Criswell, "if you have a 2K image - you have say only 2 million pixels - so if you are simulating a million collision objects then each item just gets a pixel or two, at that size those might just as well be particles. For real chunks of buildings exploding, sliding, interacting etc, that are hard core rigid body elements, then they really need to be bigger. Which means there will not be millions of them - practically speaking it is 10s of thousands, maybe a 100,000, so our focus has been getting a tool that can do 10s of thousands of really controlled - really high quality - high fidelity, rigid bodies, as opposed to millions, which would really just be a bunch of dust."
A great example of this is in the Terminator: Salvation sequence when the Terminator is covered in molten material, hardens and then cracks and break free. "I always thought that shot was really great, because it is the energy from the rigid skeleton underneath that is propagating out that is causing pieces to pop, and move and break and fall off...he was breaking out of the shell," recalls Criswell.
The Shape Sculpting tools and the PhysBam tools are all built at ILM with the artist in mind - a point the teams drove home when accepting their awards. Talking just before the event they commented to fxguide, "We want to get a mid level artist be able to use these amazingly complex tools and effects, and generate shots right out of the gate, rather than have just a handful of incredibly talented experts who are the only ones who can do this kind of work. By spending more time and more energy into developing the artist aspects of the tools to a certain level, it means we can have artists who maybe aren't experts in every conceivable aspect of the maths, operating the tools and generating great, great shots. We have a lot of artists using these tools, it is not just 'oh there is one destruction guy or whatever who knows how to do that'. We have 20 or 30 on any show."
All Sci-tech award winners
Front row: Allan Padelford, Robert Nagle, Iain Neil, André de Winter, Margot Robbie, Miles Teller, David Gray, Richard Edlund, Cheryl Boone Isaacs, Dr. Larry Hornbeck, Greg Pettitt, Bill Werner and John Frederick.
Second row: Philip Peterson, Dan Piponi, Kim Libreri, George Borshukov, James O’Brien, Marco Revelant, Shane Cooper, Steven Krycho, Frank Poradish and Magnus Wrenninge.
Third row: Greg Croft, Michael Sechrest, Chris King, Peter Braun, Ron Fedkiw, Brice Criswell, Ben Cole, Eric Parker, Nafees Bin Zafar, Stephen Marshall and Reiner Doetzkies.
Fourth row: Nicolas Popravka, Cary Phillips, Colette Mullenhoff, Alasdair Coull, Jeff Budsberg, Mihai Aldén, Erwin Coumans, Ken Museth, Karl Rasche, Scott Peterson and Mitsuru Asano.
Fifth row: Brad Walker, Michael Fecik, Steven Tiffen, Jeff Cohen, Ken Pearce, Bob Myers, Jonathan Gibbs, Masahiro Take, Mitsuyasu Tamura, Ichiro Tsutsui and Thomas Lianza.
Back row: D. Scott Dewald, Greg LaSalle, Roger van der Laan, Tim Cotter, Peter Cucka and Robert Bridson.
VFX Oscar - this Sunday night!
On Sunday night ILM’s team are in the running to win Best Visual Effects for their work on Captain America: The Winter Soldier. Dan DeLeeuw, Russell Earl (ILM's supe on the show), Bryan Grill and Dan Sudick are nominated for the film. This is the first Academy Award nomination for Dan DeLeeuw who was the overall VFX supervisor on the film. This is the third Academy Award nomination for Russell Earl from ILM. He was previously nominated for Star Trek (2009) and Transformers (2007). This is the second Academy Award nomination for Scanline’s Bryan Grill. He was previously nominated for Hereafter (2010) and perhaps most impressively it is the 7th Academy Award nomination for practical special effects supervisor Dan Sudick. He was previously nominated for Iron Man 3 (2013), The Avengers (2012), Iron Man 2 (2010), Iron Man (2008), War of the Worlds (2005) and Master and Commander (2003).
Watch ILM's Winter Soldier breakdown below:
We've been a free service since 1999 and now rely on the generous contributions of readers like you. If you'd like to help support our work, please join the hundreds of others and become an fxinsider member.