We spoke to two of the groups honored with Academy Awards for their technical achivements. First, ILM who are honored for developing OpenEXR and their image modeling tool. Then, the Foundry who are being honored for their Furnace tools. To win such an honor means your work significantly advances the art and science of film making.
No one will care what these Oscar winners wear, who they are sleeping with or the names of their children. No one will camp outside their houses or try and photograph them on beach holiday. In fact the only people that really sit up and pay attention to their work are the millions of film goers who pay their money, buy their popcorn and enjoy the amazing visual effects they make possible.
While many people are concerned about who wins best actor – it is the technical Oscars that signify the depth of technical excellence and hard core image processing R&D that allows the stories that audience pay most to see. In pure box office terms, the vast majority – almost the entirety of top 20 all time box office pictures have strong visual effects. Time and time again people cry that they like things real and not cgi- like cgi is a bad thing, but this is flawed logic.
Firstly, Tom isn’t really a secret agent, nor is that his car, or his house,.. in fact it isn’t even a house – it is three walls and lighting gantry ! Real is very subjective adjective. And secondly: great effects allow the telling of great stories – stories that could not be told without effects, from Aslan’s kingdom, to Luke’s space piloting these stories could not be told without effects. Audiences love these stories and the vote with their own hard earned cash. And the technicians that solve the complex maths equations and epipolar geometry that allows new and better visual effects each year are the incredibly bright and talented technical artists honored by the Academy in the technical Oscar section.
Unlike some other categories, these winners are honored for work that isn’t just about this year, and to win they needed to go through a long and complex review process. It is therefore a great honor and a technical oscar really signifies a major contribution to the art and science of film making. To win means having panels of people review and discuss your work, supporting letters and testimonials from leading film makers from all over the world and normally it represents work that has taken years to develop and perfect.
The formal awards ceremony and presentation took place on 10th February 2007 in Los Angeles.
We caught up with Bill Collis from the Foundry and in this week’s Podcast we talk to Steve Sullivan, Director Of Research and Development for Industrial Light and Magic. (see below for a detail discussion of the ILM research)
The Foundry
Furnace enjoys an enviable reputation, having been used on a host of high profile feature films including Casino Royale, X-Men 3 The Last Stand, The Da Vinci Code, Charlie and the Chocolate Factory, Batman Begins, King Kong, The Lord of The Rings trilogy, Poseidon and Superman Returns.
The Foundry’s head of research and development Dr. Bill Collis, chief engineer Simon Robinson, and senior software engineer Ben Kent, and Dr. Anil Kokaram, will receive the Academy Award® for their roles in the design and development of The Foundry’s Furnace. Furnace is a suite of effects plug-ins, which robustly utilizes temporal coherence for enhancing visual effects in motion picture sequences. The Academy considered the Furnace toolset’s modularity, flexibility and robustness to have set a high standard of quality for optical flow based image manipulation and so the team will be honoured with a Scientific and Engineering Award from the Academy of Motion Picture Arts and Sciences.
Optical flow or motion estimation is an area in which The Foundry has pioneered a new generation of algorithms which enable the tracking of every pixel in a frame to subsequent and preceding frames. Launched in 2002, The Foundry’s Furnace was the first comprehensive toolset to deploy these new algorithms, with the retiming tool Kronos rapidly becoming a market leading technology for speeding up and slowing down footage.
The latest version of Furnace provides about 30 plug-ins within the Furnace package that take advantage of advanced motion estimation technology, providing artists with a range of tools to tackle everyday compositing issues as diverse as wire and rig removal, grain reduction, dust busting, image stabilisation, auto rotoscoping, image segmentation, flicker removal, image stitching, adding motion blur, tracking and generating depth mattes amongst many others.
The Foundry is the only British company to be honoured with a 2006 Academy Award. There are also 3 other companies being awarded the Scientific and Engineering Award Academy Plaques, the other three being ILM (see below), and then Phillip J. Feiner, Jim Houston, Denis Leconte and Chris Bushman of Pacific Title and Art Studio for digital film restoration, and Howard Preston and Mirko Kovacevic for the design and engineering of the Preston Cinema Systems FI Z wireless remote system.
FXG: You must be pleased with the Academy’s acknowledgment?
Bill Collis (BC): We are absolutely delighted to have our innovation recognized by the Academy. We continuously strive to deliver real solutions and cutting-edge technology for the motion picture industry, and to receive such a high accolade is a great honour. Thanks must also go to our customers; we are privileged to work closely with leading companies around the world such as Weta Digital, Pacific Title and Art Studio, Double Negative and The Moving Picture Company. Their requirements for technology that pushes the boundaries of what is possible, willingness to trial our new motion estimation applications, and eagerness to share their views, have made a huge contribution to Furnace and helped it become the success it is today
FXG: What role did you all have in the development process?
BC:Well broadly speaking Anil and myself look after algorithmic research and Simon and Ben as well as developing algorithms focus on implementing and developing the algorithms to get them working in a real-world environment; ensure that they are performing as we expect them to, accuracy, speed that sort of thing. Although with Furnace it really is a completely collaborative process so at one point or another every member of staff has a hand in some aspects of the development.
FXG: When did Furnace research first start?
BC:We began our research into motion estimation in 2000.
FXG: It has been a long and interesting road since the matrix ‘bullet time” – what were the highlights along the way ?
BC:The Matrix was in 1999 and I was still working with Snell and Wilcox back then, I guess the “bullet time†was the start of it all. Although it’s been tough it’s been a fantastic journey and we have most certainly come a long way since those early days. The first major turning point was in 2002 when all those months of research and development were packaged up and delivered as the first Furnace plug-in suite. When those first sales orders come through that gives one a massive sense of achievement, when all of our research and development into motion estimation became a product that was actually being used on major feature films it gives you a rather different perspective on things. In the first year Furnace was in operation in over 50 post facilities and our customer list featured some really high profile names, many of those companies are still Furnace customers to this day which is a huge testament to both the product and the team. Since then the product development has kept us busy and with our customer list now in the hundreds they keep us pretty busy too.
FXG: Furnace covers a few big areas of image processing – optical flow perhaps being the most widely known but the furnace package is more than just optical flow isn’t it?
BC:Optical flow is definitely the class of algorithm, which is most widely implemented across the Furnace plug-ins. In addition to optical flow, we employ a variety of techniques for example non-parametric methods for texture duplication, wavelet decomposition for noise reduction and blind deconvolution for removing blur.
FXG: Where is the product at today?
BC:Various ‘flavours’ of Furnace are now available across a wide range of hosts including Autodesk Flame, Flint, Fire, Inferno, Smoke and Toxik, After Effects, Avid DS, Shake, Baselight, Film Master and Scratch. We are on version 1 for After Effects, Avid DS, Baselight, Film Master, Scratch and Toxik and version 3 for Autodesk Flame, Flint, Fire, Inferno, Smoke and Shake. We are working on the next generation of Furnace, which we are planning to release in the coming month.
FXG: What are the plans for Furnace’s future?
BC:We want to continue to develop useful solutions that automate or significantly reduce the amount of time involved with solving everyday issues during the creation of digital effects. Developing solutions to enhance workflow, boost productivity and generally make our customers lives that little bit easier is our core focus.
ILM
ILM is being honored in two areas, image-based modeling and for the development of OpenEXR.
Image-based modeling
A Scientific and Engineering Awards, (Academy Plaques) in the same category as the Foundry’s award, is being given to Steve Sullivan, Colin Davidson, Max Chen and Francesco Callari for the design and development of the ILM Image-based Modeling System.
ILM’s image-based modeling system is a set of tools designed to recover detailed geometric information from photographs or film sequences. It exploits texture and contour features to densely reconstruct surfaces with complicated shape. At one extreme, it can create useful approximate models from only a few silhouettes (e.g. Minority Report), while at another extreme, it can integrate many image cues to produce dense high-precision models comparable to laser scans (e.g. Lemony Snicket, Pirates of the Caribbean: Dead Man’s Chest).
The reconstruction process typically starts with two or more photos/frames of a subject, which are then undistorted and matchmoved to establish the projection geometry. Depending on the desired output, an artist will identify regions of texture, silhouette contours, point or image correspondences, and/or 3d curves or surfaces to drive the reconstruction. Models are then either produced directly, or templates are fit to the data to preserve existing topology and parameters. The artist can optionally use the interactive editing tools to correct reconstruction errors, change resolution, filter noise, or even extrapolate solutions as needed. Typical reconstruction time for a quick model from a couple snapshots might be 15 minutes, while a high precision facial model takes around 1-2 hours.
The system has been applied to a wide range of production problems over the past five years, including constructing CG versions of maquettes/models, building an organic virtual environment from an outdoor photo survey, creating set geometry for match-moves and character interaction, creating scene geometry in our 3d-from-2d process for stereo films, and capturing detailed models of actor faces and bodies for digital-double work. It has evolved considerably based on feedback from ILM’s own production teams, and is now preferred over Cyberscans for modeling human subjects by ILM’s artists.
Laser-based scanners address many of the same problems, and can actually produce better or faster results in some situations (such as static objects or sets where there is time to scan). However, they are often impractical or impossible to use under common production circumstances, such as when the set cannot be cleared and held long enough to do a scan (shot setups rolling too quickly), plate or reference footage is available but no scan was taken, the scale of the subject is too large/small, the subject cannot remain still for an extended period, or a scanner is simply unavailable or unable to be brought on location. In fact, the need to scan babies and children was one of the motivations for building our own solution.
Because the ILM system is image-based, it is far more flexible than active-scanning technologies, since it works for subjects of any size, allows instantaneous capture, can be used “after the fact†on unplanned footage, and the source photos provide backup validation for reconstruction results. With common consumer cameras (under controlled conditions), the ILM system can achieve point-wise accuracy comparable to a Cyberscan, and quality can improve in the future as photographic equipment improves. These cameras can also be synchronized to a beauty lighting package and/or extra cameras to provide nearly-simultaneous (1/30th of a second) capture of production-quality texture reference matched precisely to the geometry, something no other system currently provides. Just as important, the interactive tools allow artists to generate useful reconstructions from incomplete or unplanned data, and decide for themselves how much post-processing or cleanup (e.g. smoothing) they care to use.
Regarding image-based approaches, photogrammetric tools are now in common use across the industry, generally reconstructing shape by triangulating isolated points. ILM uses photogrammetry extensively at ILM, but the pointwise approach can be extremely laborious or too sparse for common cases like people, terrain, architectural detail, or other organic forms. It can only serve as dimensional reference for these kinds of subjects, and modelers commonly complain of needing far more dense contours and detail to help construct production-quality models.
OpenEXR
ILM is also being honored with a Technical Achievement Award (Academy Certificate) which is being awarded to Florian Kainz for the design and engineering of OpenEXR, a software package implementing 16-bit, floating-point, high dynamic range image files.
OpenEXR is a set of software libraries and tools that implement an image file format designed specifically for visual effects production. Many of the software packages employed by the visual effects industry use the OpenEXR libraries and file format. A significant number of effects production houses have adopted OpenEXR for their workflow on feature films.
16-bit floating-point pixel data in OpenEXR image files have finer color resolution than the human visual system and a wider dynamic range than any practical image capture device. Mathematically lossless data compression keeps file sizes manageable, without altering the images. In an effects production pipeline, the image file format is not a quality bottleneck. Bits are lost only at the end of the pipeline, when images are output on film or video.
Arbitrary image channels, beyond the usual red, green and blue, allow 3D rendering software to output more than just a pixel’s color. For example, OpenEXR image files can store per-pixel lighting information, surface normals, depth, and motion vectors, which are often required for visual effects compositing and image processing. Because pixels are stored as floating-point numbers they can hold scene-linear data, which simplifies pixel computations; operations do not have to take film or video gamma into account.
The OpenEXR libraries are available free of charge in source code form.
More and more companies are adopting OpenEXR, probably for a combination of reasons: there is no loss of image quality due to the file format; image file sizes are reasonable; floating-point pixels make image processing easier; visual effects often require many image channels; reliable open- source libraries make integration into application software and production pipelines relatively painless; and the libraries ensure compatibility between applications by avoiding competing and mutually incompatible implementations of the image file format
Pixel data in OpenEXR image files are stored as 16-bit or 32-bit floating-point numbers. With 16 bits, the representable dynamic range is much higher than the range of film or digital cameras: 30 f-stops without loss of precision, and an additional 10 f-stops at the low end with some loss of precision. Most 8-bit file formats have a range of around 7 to 10 f-stops.
With 16-bit floating-point numbers, color resolution is 1024 steps per f-stop, as opposed to somewhere around 20 to 70 steps per f-stop for most 8-bit file formats. Even after significant processing (for example, extensive color correction) images tend to show no noticeable color banding.
The 16-bit floating-point data format is fully compatible with the 16-bit frame-buffer data format used in PC graphics cards from Nvidia, ATI and 3Dlabs. Images can be transferred back and forth between an OpenEXR file and a 16-bit floating-point frame buffer without losing data.
While OpenEXR was created to store HDR images, the file format or ‘container’ can be modified to fold a variety of similar data. OpenEXR files can contain an arbitrary number and combination of image channels. For example, red, green, blue, and alpha; luminance and sub-sampled chroma channels; depth, surface normal directions, or motion vectors.
Pixels in an OpenEXR file can be stored either as scan lines or as tiles. Tiled image files allow random-access to rectangular sub-regions of an image. Multiple versions of a tiled image, each with a different resolution, can be stored in a single multi-resolution OpenEXR file. Multi-resolution images, often called “mipmaps” or “ripmaps”, are commonly used as texture maps in 3D rendering programs to accelerate filtering during texture lookup, or for operations like stereo image matching.
You can also add user data, that moves with the shot, software that reads OpenEXR files automatically ignores attributes it does not understand. But for in house tools can use this user data to store additional data; for example, color timing information, process tracking data, or camera position and view direction. OpenEXR allows storing arbitrary extra data, of arbitrary type, in an image file.
For a complete list of winners visit Oscars.org.
