First look: Imagineer’s mocha v3

April 16th will bring the release of Mocha v3, and it’s a big one. The version has a number of new features and a huge amount of work has gone into improving workflow for artists and facilities. It’s shaping up to be the biggest release and the most popular ever from Imagineer, having been in development for over 18 months. New features include:

• New Roto Tools: Transform Tool and Join Layers (point parenting).
• Layer Management: View, organize, color code and group layers
• Project Management: Merge and share projects between artists
• 3D Camera Solve: Camera export for After Effects, Nuke, C4D & Maya
• Dope Sheet: new keyframe editor

In this fxguide exclusive, we’ll take a look at some of the tech behind the new features, a bit of behind the scenes into the development of the software, as well as a written review and hands-on video from fxphd prof Ben Brownlee. Brownlee is teaching a new mocha pro v3 course at fxphd this term, and the new release will also be available on the VPN. But before we get to that, a bit of historical perspective is in order.

History

At IBC 2001, Imagineer showed their planar tracking technology publicly for the first time, launching mokey, a removal product which allowed artists to get rid of objects in a scene with less manual tracking and painting/clone techniques. Planar tracking tech grew out of computer vision research carried out at Surrey University in the late ’90s. It uses techniques from 3D tracking rather than 2D approaches like optical flow, combining several different algorithms to produce a robust result.

Traditionally, most tracking was done by selecting a small region on the screen and having the software track the area from frame to frame. The software uses various ways of analyzing the pixels on screen from frame to frame in order to track the pixels. If an artist wants to map an object onto a plane, one would generally do a four point track, manually placing tracking points at appropriate places in order to define the rectangular plane.

With planar tracking, the process is a bit different. The artist feeds an application information as to what to track by creating simple spline shapes. The planar tracker then analyzes all the pixels in the shape and determines which are moving on the same relative plane. While some users interpret “planar tracking” as only solving “flat” screens, almost all objects can be analyzed as a flat pixel pattern. With this in mind, mocha’s planar tracking is excellent at tracking faces and objects that are not thought of as “planes.” Comparing all aspects of this “planar pixel pattern” over time is what allows an application such as mocha’s tracker to stay locked on through color shifts, luminance changes and blurs.

Additionally the shapes can use multi-layer modes to animate, add or subtract to the search area. Using these methods, mocha’s Planar Tracker is quite robust and easy to use in its ability to track through problematic shots such as tracking obscured objects, motion blurred pixels and objects that go offscreen. Using traditional point-based tracking methods, there is generally considerably more artist intervention.

In 2003, Imagineer worked closely on an R&D basis with London-based Cinesite during post for Harry Potter and the Prisoner of Azkaban. Cinesite was looking or an efficient way for inserting the animated paintings into the picture frames on the hallways at Hogwarts. This R&D development became a product in 2004 when Imagineer introduced monet, a planar tracking and insert compositing station designed to easily insert content into moving shots.

In 2006 mocha was first introduced as an affordable planar tracking and roto utility, designed to export 2D tracking and roto data into apps such as Flame and Smoke, Avid, Final Cut, After Effects, Fusion, NUKE, and others. The following year, the planar tracking tech was introduced to a much wider audience when Imagineer licensed Mocha AE to Adobe and had the product bundled with After Effects CS4. With After Effects in the mix, Imagineer now has over 42,000 users worldwide.

Two years later, mocha and mocha AE v2 were released, featuring an improved UI and stabilization module. The applications reached 64-bit support in 2010 with the release of v2.5 as well as saw significant improvements in speed, including hardware accelerated rendering. In 2010, Imagineer also dramatically simplified their product line. Over the years, the array from products (mokey, monet, mocha, etc) from Imagineer could be seen as somewhat confusing, so they decided to combine products and into one: Mocha Pro. Mocha Pro had a new UI and modules, including Remove, Lens Distortion, Insert Compositing, and Stabilize. It quickly became Imagineer’s best selling product.

Versions 2 and 2.5 were important releases for interface design, 64 bit support and the merger of all of Imagineer’s technology into one code base with mocha Pro. While all of this was necessary to better support their customers and clean up the product line, there were not many “cutting edge” new features in those releases. That changes with the release of Mocha Pro v3 and Mocha AE v3 in 2012.

Mocha v3

The development process for v3 brought some change to the way Imagineer worked with their customers. They have always had a good relationship with facilities (Cinesite is a good example of this), but tried to get even more of their customers involved early on. “One thing that we approached differently for this release was to solicit customer feedback very early in the process,” says Imagineer’s Ross Shain. “In fact, we shared early ideas with some select customers over a year ago to make sure we were going in the right direction and addressing their feature requests.”

It’s easy to focus on the numbers of mocha AE users around the world, due to the ubiquitousness of the After Effects software. However, mocha Pro is rapidly becoming a go-to tool in the feature film arena. One of the main areas Imagineer has seen growth in has been the stereo conversion market as well as large scale roto pipelines. “More facilities have been using mocha to assist with roto and clean plating,” says Shain, “so we worked with artists from Digital Domain, Legend 3D, Identity FX and more to help prioritize features. From this feedback, we focused on layer management, project merging, advanced roto editing improvements and more.”

A telling sign of the adoption of mocha Pro at these high-end facilities is the acknowledgment of one feature that didn’t quite make it into the release: Python scripting support. According to Shain, the groundwork has been started for this, and will be added to a v3 point release in the not-to-distant future.

The workflow changes made in the software aren’t just something that large facilities appreciate, as you’ll read later in Ben Brownlee’s review of the software. While not necessarily sexy, those kind of improvements have a big impact day to day — features such as more keyboard shortcuts, new transform controls, layer management, and a proper dopesheet for keyframe editing will be improvements artists see every day.

As an artist we often look at a piece of software and figure that workflow changes shouldn’t be all that hard to do….after all, they’re generally tweaks. However, many of the changes in this release involved some major changes to the tools. “Adding layer multi-selection was probably the single biggest job because knowledge of which layer is being manipulated is vital to almost every part of mocha,” says Shain. “The layer list has been completely rewritten and shares foundations with the new dope sheet. The new layer backend has the potential for more powerful organizational tools such as nested groups, tagging and search, features we could add in the future if users ask for them.”

As far as single features go, probably the biggest is the addition of the new 3D Camera Solve. It’s important to note, however, that this isn’t meant to be a replacement for a standalone application such as PFTrack or SynthEyes. It’s intended “to add an easy to use, alternative to conventional camera tracking,” says Shain. “There are shots where a full camera track is necessary or advanced track editing is required that we are not exposing to the user.” Instead of exporting point clouds, mocha v3 exports a camera and planar data such as 3D nulls or axis points.

Users of mocha have asked for the ability to export 3D data for years. In fact, according to Shain, many customers use mocha alongside camera trackers to help with difficult 3D solves. “They use mocha to planar track various objects on screen, add 2D grids or patterns, then re-run them through their 3D tracking apps to get the final solve,” says Shain. “So polling our user base and hearing stories of how mocha’s 2D tracks were assisting on difficult 3D solves gave us reason to focus in this area.”

Imagineer’s 3D solver uses planar tracking to its advantage (such as tracks through heavy blur or obscured objects where point trackers break down) to solve a virtual camera. “We believe that once customers get used to this idea and methodology, the feature will find its place in many workflows,” says Shain. “Certainly customer feedback will greatly influence the direction we take with the feature, but in this first instance, we are looking at this as a way to solve the 3D camera and select planes within a scene. We expect it will compliment the other 3D trackers quite well.”

The software doesn’t attempt to create a true physical camera, but provide a camera that works for a specific task. “All our calculations are done in pixels relative to the image dimensions and our camera focal length is expressed as an angular field of view,” according to Shain “For many visual effects tasks this is all you need, our tool is not designed for situations where you need a physically accurate model of the scene.”

This begs the obvious question as to whether mocha should include a 3D environment for making roto easier for artists. After all, rigid objects in a tracked scene can become much easier to roto, track, and isolate, simply by placing shapes at the appropriate location in 3D space. According to Shain, this is definitely a direction where they can see the product going. “Adding proper 3D space in conjunction with planar tracking could make for some very useful tools and improve things like roto or object removal,” says Shain. “Our tools are also being used quite a bit in stereoscopic conversions and there are many areas here that 3D space could be useful.”

There are several algorithms available for use in mocha v3 in order to get an accurate solve. There’s an auto mode, which before analysis does some quick measurements as to how layers in the scene move and then chooses the right algorithm. But if that doesn’t work, users can choose one of the four algorithms manually.

The PTZ (pan/tilt/zoom) solver uses the fact there is no change in perspective on the scene to give an almost instant result, but because there is no parallax, the relative depth can’t be determined and all the objects are placed at the same depth in the scene.

The two moving-camera solvers (small and large parallax change) each make different assumptions about how the camera moves that optimize them for different types of shot. Small Parallax works for shots where the camera moves slowly over a short distance relative the distance to the subject. So an aerial shot where the camera moves a 100 feet at an altitude of 1000 feet is “small” even though the actual distance the camera moves is quite far. The Large Parallax solver assumes the camera is close to the subject and moving fairly quickly. This is the solver to use for fast moving hand-held or tracking shots where the angle of view between the camera and the subject changes a lot.

They also have a fourth solver, which is used for moving objects. Solving the camera needs stationary objects, but once the camera is solved we can convert any planar track in the scene into a 3D solution for that layer.

We’ve asked Ben Brownlee to provide an overview of the new features which, as we mentioned before, include some very significant workflow features.

mocha v3 overivew

by Ben Brownlee

Version 3 of any software is an important milestone. There is an expectation of maturity now. The issues with V1.0 are meant to be long since fixed, the core of V2 is solid, so V3 holds the promise of a fully-featured workhorse. Imagineer Systems mocha has now hit that magic number and it opens itself up to an even wider range of users.

Imagineer Systems’ product family has always been built on the planar tracker. This single piece of technology has been the basis for a whole range of tools over the years, including mokey, monet and, of course mocha. At its core, it offers rock solid tracking where point trackers would fall over. How you used that tracking data was up to you and the product you were using. Whether that was object removal, screen insertion, to aid roto work or just as exported trackers to your chosen compositing app.

Although this release does have some big new headline features like the 3D Camera Solver, for people who use mocha on a daily basis, you’ll probably be more impressed by the less flashy stuff.

This is a huge usability upgrade if you are working on larger projects or complex roto. There are now proper layer management tools. No more working with a huge list of layers. Layers can be grouped, collapsed and both layers and groups can be easily colour coded. On paper this sounds a little tame, but the time savings over the course of a day are huge. It also means that house rules regarding layer structuring can more easily be enforced. So when you pass your shot on to another artist either in mocha, they can see immediately what they are working with. Likewise, when you come back to your own shots after a couple of weeks, you’re not wasting time figuring out what you were thinking that day. Your 20 layer hand roto sits in its own group, hidden away until you want it.

Another problem fixed in V3, was how you managed larger shots that required multiple artists. Now it is simple to merge projects together into a single master. Boom. Solved.

The roto toolset is significantly improved. For a product that was highly reliant on understanding the distortion of planar surfaces with its tracking, when it came to manual shape animation things were not so easy. Again, that has changed. It is now straightforward to do distort, skew and non-uniform scales. What does this mean for your workflow? Simply put, less work, fewer keyframes and better animation. The more natural movement you can capture with the main shape, the less work there is to do on individual control points.

The bounding box that lets us do all of these shape transformations is not limited to a single shape. Select multiple shapes or even just a few control points, and the box automatically resizes to fit the new selection. This opens mocha as a seriously fast rotoscoping tool. For those of you who have taken one of my classes, you’ll have realized how this fits perfectly into my roto ethos; track what you can, animate your total shape and only then hit your control points for the fine-tune. This keeps things, fast, simple and easily fixable!

I’ve been working with a pre-release version mocha Pro V3 for a while now and the productivity improvements with these tools are immense. You know a feature is good when you dread going back to working without it.

In the past, joining layers together hasn’t always been smooth. The Join Layer tool worked for the frame you joined them at. Then stopped working as soon as you animated either control point. It worked. Just with limitations. Those limitations are now gone in a very smart manner. The Join Layer tool, now creates a Parent/Child relationship between the two points (or a Master/Slave one if that’s your particular bent). Connect your two or more points together to create a seamless edge between them. Great not just for clean rotoshapes, but essential for complex object removal in mocha Pro’s Remove module.

As the father of a three-year old, I prefer to think of it this as Parent/Child because the child can move the parent around as easily as the other way around. Obviously this is to keep the edge seamless no matter what you do. Of note, it is only the Parent control point that can break the join. How to tell a Parent point? Both from the air of world-weariness that hangs around it and its fancy new icon (a triangle).

There are more small improvements to make life easier, better keyboard shortcuts, including nudging and Viewer. Also is the move away from the Curve Editor to do keyframe manipulation. Taking over that mantle is the Dope Sheet. Instead of fighting the Curve Editor to delete, move or copy/paste keyframes, the Dope Sheet displays them in a straightforward manner. The next time the tracker goes awry, you pop into the Dope Sheet select the dodgy keyframes and delete them. No fuss. No hassles. No selecting the same keyframes multiple times. It’s always the small things that make me happiest.

With that being said, I come to what may be the biggest “new feature” of the release, the 3D Camera Solver module. That’s right, mocha is breaking away from 2.5D tracking data. You can now get two different types of 3D solve out of mocha, using your planar tracking data. Either take out a nodal pan (Pan, Zoom, Tilt) from a moving camera at a fixed point (e.g. mounted on a tripod), or a Perspective Camera, calculating data from a more complex camera move (e.g. dolly, crane or even handheld). The number of shapes you have to track (read, amount of work you have to put in) obviously depends heavily on the type of movement and your material. Once your camera is solved, you can then export this either to After Effects or FBX. An additional plug-in is required to read your camera data in After Effects and FBX export is limited to mocha Pro.

The camera solver doesn’t offer the same sort of interactivity as a dedicated 3D Matchmover, such as PFMatchit or SynthEyes, but as a Value Add to the already impressive toolset, it is a worthy addition. It is also a fast way to get even more use out of your tracking data.

I have been working with Imagineer Systems’ products for almost 10 years now, and when it was released, the mocha product really bridged the gap in the market between incredible tracking data and flexibility of use. mocha Pro greatly increased the software’s internal range of functionality by skimming off the best features of the now defunct mokey and monet. But by focusing on core functionality both in terms of workflow and rotoscoping tools, this threatens to be the most important release in its history. It opens it up to large workgroups, working on big projects and offers even more tools for the lone wolf, who just needs to get their project finished before deadline.

mocha is becoming a Swiss Army knife of post – it sits between so many apps that the export drop-down is on the verge of unwieldy. Its feature-set is specialized enough that it does stuff better than the tools built-in to your compositing app to justify the roundtrip into another interface. The way this software has matured into the new Version 3, makes me genuinely excited by this release.

Hands-on video

Brownlee works through some of the new features in the software, in this video overview.

Ben Brownlee is teaching a new intermediate level mocha pro v3 course at fxphd during the April term. The new version of mocha pro will be available on the VPN, which allows members to run the software while learning.