This article has been updated. We keep this version here for historical reference, but a newer version updated for the latest films, techniques and companies is here. We recommend referencing the new article that contains much of this original article but a lot more in addition to this. Thanks.
Stereo conversion, or dimensionalization as it is sometimes called, is the process of making stereo images from non-stereo traditional 2D images.
Many people argue that if you want a film in stereo you should shoot it in stereo. Yet many studios are warning post houses that if they work on stereo projects, even ones shot in stereo, there may well be a need to convert some footage and that high quality conversion is an important tool in the box of any effects house.
Stereo conversion is also needed for converting older films – such as the Star Wars franchise. John Knoll (ILM) is overseeing the stereo conversion of every Star Wars film for director George Lucas and in a recent AWN article he expressed the view that, “You can’t rush it and it’s an iterative process, and if you’ve got a gun to your head and you’ve got eight weeks to convert a 2,000-shot show, it’s not possible to maintain the level of quality control that you need.” In the case of Star Wars conversions for which Lucasfilm “will be (mostly) using outside vendors,” he also pointed out that he had been vocal in saying that past efforts “were victims of a too rushed production schedule and a too low budget.” Few people doubt that Knoll will deliver anything but cutting edge results.
3D animated films, such as Toy Story 3, Tangled and others find it easy to correctly generate stereo imagery from either stereo-RenderMan use or just rendering the entire scene from two similar but offset virtual cameras. For everything else live action, normal stereo production is hard. Even Avatar required tiny amounts of stereo conversion, for example the opening macro eyeball shot was far too close for a stereo camera rig to film, and this first shot of the film was converted stereo from 2D.
One of the leading company in this area today is Prime Focus, who after a battering of critical opinion on the conversion of Clash of the Titans, recently converted the film The Chronicles of Narnia: The Voyage of the Dawn Treader, which stands today as perhaps the best stereo conversion done thus far. For the last Narnia film, the decision to go stereo was made after principal photography but before the visual effects were completed. This allowed a partial hybrid approach of the some 1900 edits in the film. Some 10% or 190 odd shots were fully stereo generated 3D shots. The rest were converted mainly by Prime Focus. Interestingly, for some shots this conversion was not done by Prime to the final shot, but in partnership with the relevant effects house.
The problems with generating a second view or second ‘eye’ for stereo conversion are:
• The parallax effect. That means a second eye will see around things the original eye won’t and thus there is missing background information to be replaced.
• A depth map is needed of the scene to determine the correct distribution of the objects for the second eye. While amazing work has been done with programs such as Ocula by the Foundry (see below), the process is far from automated.
• Cardboard cutouts and the need for roto. Not only is a roto required for the outline of any character in shot, if they are closer than say a wide shot, internal mattes are also required to generate different depths for different parts of their bodies. A character could easily have 7 rotos in addition to their outline for features such as nose, eyes etc and all of these must be conceptually and logically correctly placed based on z depth.
• Projection. While some shots respond well to re-projecting or ‘camera mapping’ the mono footage over 3d models and then filming the stereo by rendering the 3D scene from two virtual cameras, this rarely works well for people in movement as the difficulty of generating accurate 3d models to map onto, renders the approach extremely expensive. The technique works well for building, hall ways and other regular and mostly rigid body solutions but most films are about people – normally with soft edges such as hair etc.
• Shot design. Most stereo films are shot with an understanding and consideration of the stereo nature of the experience. A mono film may be poorly composed from a stereo point of view. For example, most staging of a stereo scene would have a distribution of the objects over the immediate foreground, avoiding the clumping that may seem odd when all the props are at the back of a room. Yet in mono although this shot may be extremely creatively valid, it is only when the shot is converted does it seem empty or oddly distributed.
• Singular depth resolution. One of the trickiest problems is say glasses on a live action character. While the glasses have a depth from the camera, the eyes behind them are further away. If incorrectly dealt with, the eyes of the character would appear to be printed on the glasses, not behind then glasses, but conversely the reflections on the glasses should not appear to be drawn on the actual eyes of the character. Add to this one tends to focus on the eyes of character in a close up and one could guess that this is perhaps a contributing factor as to why there was scheduling concern that led to a film starring a certain famous glasses wearing boy not being stereo converted recently. Similarly, hair is extremely complex and in a single close up may require extremely complex and fine roto work.
Key to good conversion
• A strong working relationship with the overall supervisors.
• A strong relationship with the vendors, so that relevant files such as keys and mattes can be provided.
• Pre-production planning. As with all vfx work, the earlier the conversion company can be involved the better.
• Time. Given the huge volume of work, ensuring the schedule allows time for roto and volumizing is vital.
• Speed of camera moves, framing and staging can all affect conversion. If you can isolate the key stereo moments in the script and shoot those knowing how they will be converted, you can produce a more impactful final conversion.
• Good asset management and asset tracking.
• If Titans taught the industry any overall lesson, it is that stereo conversion should not be a process ‘tacked’ on the end of a film’s production. Sometimes a film is converted years after principal photography, but if the conversion team can be involved before principal photography it can aid the creative, schedule and the budget.
• Make a good film in stereo not a good stereo film.
In The Chronicles of Narnia: The Voyage of the Dawn Treader the character of Lucy finds a magic book and casts a spell that makes it snow.
Additional video available for fxinsider members
The shot was filmed mono as you can see from the behind the scenes clip. The snow sequence was handled by The Senate in London. While a clean plate of the room with no snow was shot, it was of little use as the camera move was not motion control.
The Senate’s job was to add some more snow and also do some work on the ceiling of the room.
Rather than just finish these shots, The Senate did the ceiling repair as normal but rendered out passes of snow as seen by a stereo camera. These various elements, un-composited, were then handed to the team at Prime Focus who converted the mono plate and matched it in with the new stereo foreground snow.
Ed Marsh was the overall show stereo consultant and supervisor, who was brought in by Walden Media when the show was decided to be converted. Visual effects supervisor Angus Bickerton could not be more positive on the role Ed Marsh played in both working with the production and educating the team on the subtle stereo issues involved.
On this scene, Ed Marsh worked out the convergency and disparity and passed this information to Prime Focus. “It was such a team effort this sequence,” says Matthew Bristowe, senior producer at Prime Focus. “It was a team effort to deliver the effect and quality that Ed Marsh wanted, we had eight or nine people working on that sequence to introduce just the right and sincere fall and movement of the snow.”
Chris Brown and Johnathan Banta (3DCG) were hired by Walden Media and were the interface between Walden and Prime Focus. Both Brown and Banta ended up spending considerable time in Mumbai at the end of the project as well as working with Ed Marsh. Richard Baker, Prime Focus’ senior stereographer said of the stereo design of Ed Marsh: “I am a fan of the way Narniawas converted (from a creative stand point)…nothing too extreme, it is comfortable on the eyes, but with moments when things do come out of the screen, and they are effective.”
Bickerton was initially concerned that this multi-facility approach would be problematical, but in the end he felt the stereo conversion worked very well, actually improving the look of the effects, ‘adding’ to them in a way he had not expected.
This snow scene was visually extremely complex, given how much of the frame is moving. “That shot in particular,” says Baker, “is a matter of keying, rotoing, extracting areas, placing things in different places in depth and a lot of that comes down to the skill of the artists, and I think that is where we have come a long way in the last 6 months, the skill and techniques of the artists, the software has advanced – I would put a lot of that down to the skill of the artists.”
Tools of the trade
Roto is the primary tool used for stereo conversion by volume. While the actual roto itself just prepares the material it is some of the most time consuming work in any conversion process.
The Foundry in London, long time image processing and compositing experts, makes Ocula for stereo post production. Ocula produces some of the highest quality disparity maps allowing for a range of high quality solutions. Ocula was not prmarliy developed for stereo conversion. It was actually more developed for stereo production correction and adjustment. Filmed stereo footage can be analyzed and the stereo properties very accurately adjusted to aid matching and camera/rig imperfections, as well as accurate stereo roto. This is roto that object matches left and right eye rotos in each eye, not rotos designed for stereo conversion.
Several major effects houses use Ocula. Recently, ILM decided to invest in a significant number of Ocula seats after a successful evaluation period using it to tackle common stereo problems. MPC, London, has also made a major Ocula purchase for their upcoming work on Disney’s Pirates of the Caribbean: On Stranger Tides. They’ll be employing Ocula to resolve vertical alignment issues and to colour match plates.
India’s Prana Studios are now renting a large number of Ocula seats in response to an increased demand for stereoscopic effects and Australia’s Animal Logic also recently plugged Ocula into their stereo workflow.
Once footage is stereo converted, the missing information due to parallax must be replaced. Several approaches to this are used, including in-painting, often a proprietary tool that effectively smears into the gap the colours from the surrounding gap. This is only needed for one eye and only on one edge as the other side of the same object would move in the opposite way and fractionally cover some part of original image.
This is an approach whereby the image is projected over a depth map calculated point cloud. If the objects in a scene move or the camera moves, it is often over a set of frames to use optical flow style techniques to produce a disparity map. If this process is inaccurate, an edge of a persons hair, for example, can appear to not be where the character’s face is, but rather projected on the wall or building behind the character.
Several vendors use projection techniques to re-project imagery over stand-in geometry. This combination of scene reconstruction and roto has proven very popular in Nuke. The Foundry’s Nuke is now the mainstay of feature film compositing for many companies but it has also been one of the most popular tools for doing short form TV show or commercials stereo conversion. Nuke naturally works closely with Ocula, as it is currently the only visual effects compositing tool that supports the software. Nuke’s latest release will also support RenderMan rendering.
Note there is a Nuke Stereoscopic Course on offer in the January 2011 term of our sister site www.fxphd.com.
Specialist Tool: Dimensionalization by In-Three
In-Three’s similar tool is called Dimensionalization, which was launched in 1997. It uses Imagineer’s Mocha tools to help produce the roto isolation. Ross Shain, CMO of Imagineer points out that not only have they seen a lot of growth in Imagineer’s products for Stereo Conversion, with a huge demand for interchange between say Mocha and Nuke. “What we have seen is that Mocha to Nuke is a really successful pipeline and popular workflow for companies building 2D to 3D pipelines,… and we are working with a lot of companies, In-three of course but also Legend, ICO-VFX – they are doing a big ramp up, also Stereo-D, Pixel Magic and others”. The main reasons he believes Mocha has been popular is the company’s planar tracker, links with Adobe’s After Effects and the ease of learning the software – “as many of these companies are ramping up quickly at the moment, and having to train a lot of artists”.
In 2008, Disney and Bruckheimer films turned to In-Three to handle the live-action 2D to 3D conversion of G-Force, while Sony Pictures Imageworks (SPI) handled the conversion of the CG animation integrated scenes. “This was our opportunity to go full scale with our newly developed workflow. With all the pieces in place with our proprietary solution, and with mocha integrated nicely into our pipeline, we could confidently take on this challenge,” explained In-Three.
The results were a success with the G-Force 3D team recognized by the International 3D Society, winning the 2D to 3D Conversion Project of that year. It was so successful, Disney came back to In-Three for Alice In Wonderland. In-Three was assigned that task of converting scenes from the prologue and the epilogue of the film that were primarily live action. Elements created at Matte World Digital and Café FX were integrated into some shots as well.
In November of 2010, Digital Domain Holdings (mothership of Digital Domain, Vencie) bought the Thousand Oaks, Californian based In-Three Inc. and announced that it was moving it to Port St. Lucie, Florida, where Digital Domain Holdings has its HQ. A small team from In-Three will shift to L.A., where it will work on business development and work with clients. The 11-year-old In-Three had most recently worked with Digital Domain on some sequences on Tron: Legacy. Digital Domain, for which filmmaker Michael Bay is co-chairman, began building a production site in Port St. Lucie last year. The Orlando Sun Sentinel reports the move should bring about 70 jobs to Florida, making the facility relatively small for stereo conversion or the hub to outsource from to perhaps other offices or companies around the world.
“Pretty much all the movies we’re working on have a stereo aspect,” Digital Domain CEO Cliff Plumer told Daily Variety. “Our preference still is shooting in stereo, if you can. But a lot of films are still shot flat, and the post schedules are getting much tighter, which forces visual effects companies and stereo companies to work together.” Plumer and In-Three CEO Neil Feldman cited the concept of “digital production,” including both visual effects and 3D conversion, as central to the union of the two firms.
Specialist Tool: View-D by Prime Focus
View-D 2D to stereo 3D conversion tool is a typical example of the specialist tool used after roto for the actual second camera/eye stereo compositing and repair work. This tool was developed in house at Prime Focus from an initial technology concept by Chris Bond (then President of Prime Focus Film VFX, North America). The tool allows artists to take the roto work that has been done to identify the separate elements and then expand and build from that the second eye and stereo effect.
For example, on Dawn Treader, the comp team using View-D converted the roto files of the mono shots to final stereo shots in about 8 weeks (after roto was done). As with most pipelines, general tools are incorporated in the workflow and Prime Focus’ workflow includes Fusion which has been modified and expanded with custom plugins. While the software is evolving, the process is not an automated one. The techniques are still very manual. It is not an automated 2D-in / 3D-out process, so artist skill is as vital as workflow R&D tools.
On Clash of the Titans, the roto was done in Mumbai but the composting team was small and relatively new to View-D. In fact at the time, View-D was version 1.0. In contrast, on the completion of The Dawn Treader the software had progressed by several major versions and changed and expanded considerably. Instead of a relatively new compositing team, – on Narnia the team included, according to Prime senior producer Matthew Bristowe, “85 experienced artists in the Los Angeles facility, 65 out of the London facility and the Mumbai facility had 40 experienced compositors plus another 100 artist in training, and this is before you get to the roto team.” In regards to the combined software and artist workflow approach, Bristowe explains: “This is version 3.0 and we are already working on version 4, 5 and 6 of this technology, it is not finished being improved.”
The size of the operation you need to mount to handle Stereo Conversion
Case Study: Prime Focus
Following the success of the new Narnia, Prime Focus has signed several new international orders and is now riding high on the expanded opportunities that stereo conversion offers, so much so that it is opening a 50,000 sq. ft. studio facility in Chandigarh for 2D to 3D conversion, creating employment for over 3000 local artists.
Chandigarh, in northern India, is the tenth Prime Focus facility in India. The company also operates studios in Hyderabad, Bangalore, Chennai, Goa and Mumbai, in addition to international facilities in London, Los Angeles, New York, Vancouver and Winnipeg.
The 50,000 square foot facility, will hold 3000 artists and will be dedicated to working on global projects, utilizing Prime Focus’ global digital pipeline which connects India to North America and the UK. Merzin Tavaria, co-founder and chief creative director (India), Prime Focus, commented: “Our new facility in Chandigarh is a major expansion of our View-D 2D to stereo 3D conversion capacity. Prime Focus has always wanted to mark its presence in Northern India, considering its existing strong presence in Western and Southern India. Chandigarh is an ideal location due to its huge student population, many of whom are looking to establish a career in animation and VFX.” The Chandigarh facility will be under the supervision of Prime Focus’ existing expert team of stereographers.
Prime Focus had a great working relationship with the vendors such as MPC and Framestore. In fact, Richard Baker joined Prime from MPC (vfx team) at about the time of the end of Clash of the Titans. Baker worked closely to try and get depth maps, lens distortion grids, and any mattes that may exist for any digital or green screen characters. While edge mattes especially for fur or hair is extremely useful as already mentioned above, these existing mattes only make up part of the set of mattes needed for a stereo conversion for each character in a scene. “The big benefit for us is if we can get say a clean background, and a character matte. But within that there is still a lot of work, when we rotoscope say a character – live action or cg we are not just doing an outline. Say if we were doing an arm, it is not just the arm outline – it is fingers to wrist, wrist to elbow, elbow to shoulder and so on. So when the View-D artist gets the roto they can start to sculpt a depth map,” explains Baker.
Even so, the Prime team found on Narnia that if they could get the various characters with the correct grades, with separate mattes and a clean plate of the background, then it would reduce the conversion process time by about half, so it is still very much worth doing. They worked very closely with the vfx vendors to streamline the process, thus allowing more time for the director to creatively stereoscopically adjust shots and not be rushed. Overall, 20% of the shots that had clean plates, the rest had to be extracted and repaired from the source material.
Key historical timeline
First use of projection mapping for stereo conversion of matte paintings – Sassoon Film Design, Metrolight Studios.
Displacement mapping and projection mixed for stereo conversion of selected scenes – Sassoon Film Design
Conversion of original lunar mission photography as set extension, and as full frame images in a stereo film – Sassoon Film Design, Digital Dimension and others.
First big digital animated 3D stereo film, kicks off the start of the modern digital stereo era.
Stereo conversion of Roar: Lions of the Kalihari to 3D. First full live action feature conversion using hybrid displace/projection methods.
Converted by Industrial Light & Magic.
Sony Pictures Imageworks.
Sea Monsters 3D
Combination of full 3D and conversion shots (by Sassoon Film Design).
In-Three and SPI conversion (plus some original stereo generated animation).
Minor conversion done by companies including Prime Focus.
Alice in Wonderland
In-Three conversion for the start and end of the film. The Wonderland section was SPI.
Prime Focus converted the film in just 10 weeks.
Major conversion by Prime Focus. Stereo consulting by 3D CG.
Stereo-D Stereo conversion
Stereo-D Stereo conversion
Gulliver’s Travels (2010)
Major conversion by Stereo D, Rocket Science 3D and I.E.Effects.
The Phantom Menace
(set for a 2012 theatrical release)
John Knoll supervising conversion. Lucasfilm working with other vendors.