Note: this article was written in collaboration with Christoph Zapletal, a freelance Flame and Nuke Artist and has been working in the industry for a little over twenty years. One of his specialities is beauty work and retouching, where he utilizes Mocha on the job. He is currently working on a new Mocha Pro course for our sister site, fxphd.com.
Mocha is easily the leading planar tracking software available and the last several releases have seen improvements in regards to the utilization of the generated tracks, both in regards to applying the tracking data inside the app as well as exporting to various host applications. However, while its core planar tracker feature has been optimized in the last releases, the general principle on what kind of data is generated inside of Mocha has largely remain unchanged. That is until now.
With Mocha Pro 2021, a new feature is introduced: PowerMesh sub-planar warp tracking. While that’s a mouthful, what it in essence means is that you can track a mesh of points inside a defined spline. This mesh is either generated by Mocha automatically using areas where there is a lot of warping and/or contrast as points for the mesh or the mesh can be uniformly generated by the user. Points can, of course, also be added manually to both kinds of meshes.
This mesh allows something which up until now has always been a weak point of planar tracking. While planar trackers like Mocha have always excelled at tracking rigid surfaces such as a billboard, license plate, and the ever-popular screen insert, tracking organic surfaces like skin, cloth or paper has always been a hit-and-miss scenario combined with a lot of manual, tedious labour. That is why in recent years optical flow solutions based on motion vector estimation have become the favored solution for these kind of tracking scenarios. But those have their own individual draw backs, one big that they are fairly processor intensive, the other one is that it is a lot less controllable than a traditional point or planar tracker.
As with a lot of the other features in Mocha, PowerMesh lets the user decide on most parameters whether Mocha should guess the appropriate values or if the user wants to set them up for themselves. This is especially nice in regard to setting up the “Smoothness” value for a mesh, which in essence allows you predefine the rigidity of the object you’re tracking.
From a user interface point of view, not too much has changed. You set up your spline and define the kind of movements you want the tracker to consider for its solve. There are the well known options for Transform, Scale, Rotation, Shear and Perspective. But below that is a new button simply called “Mesh”. Apart from the other features, which work iteratively (meaning that you can only track the Shear if you also got Transform, Scale and Rotation active), Mesh can be selected even if not all previous options have been selected. That leaves a lot of freedom to define individual setups for very different kind of moves.
But what is equally of interest as the mesh generation itself is its application. There are roughly three different use cases that Mocha expects this data to be used: an Invert Stabilize Workflow, Alembic Export and using mesh data for Roto Spline.
The Invert Stabilize Workflow is pretty straightforward. Once a good mesh track is generated, this data can be used to stabilize the clip in the Stabilize tab. Without calling it that way, what you basically get is an UV Unwrap of that Mesh. You can either export that clip as a sequence or, if you use one of the Mocha Pro plugin versions, it can be output right back into your host application. After you’re done with any modification to that stabilized plate, you basically invert the process by either reimporting that footage into Mocha or duplicating that plug-in and inverting the stabilization, e.g. using the retouched plate as a texture for your mesh.
The second intended use is exporting the generated mesh to an Alembic file. From there you can use it for any kind of workflow, for example a projection setup or as a base for 3D modeling. This is something that should really be stressed: with the ability to generate meshes and exporting those as alembic files, Mocha becomes interesting to a whole lot of new artists, as these files can be imported into Cinema 4D, Maya, Unreal and many other applications.
The third use case – helping with roto – is really interesting as well. It actually came up during the beta program for Mocha 2021 and was not originally conceived this way. How can a warped mesh help with Roto? Well, the idea is pretty simple: you draw a shape around the object you want to roto and create a planar and a mesh track on that shape. Just like before, you can use the planar tracking to drive your roto shape. But now you can additionally opt to warp your roto spline by the deformations tracked by the mesh. This way, especially organic movements can be applied to the roto shape and cut down on the manual key-framing. It certainly is an interesting approach to such a mundane task as roto.
Adjust Track 2.0
Planar trackers by nature are much more stable and more resilient to occlusions and lighting changes, as they track whole textures rather than individual patterns like a traditional point tracker. However, this also makes them much more prone to drifting. From its earliest beginnings Mocha supplied an adjust track module to let users compensate that drift. However, as it was always intrinsically tied to the planar surface of your track, it had a lot of limitations. No matter what kind of movement you tracked, It always needed all four points of that surface plane to be adjusted. Also, they couldn’t be offsetted from those four corners, which could make dealing with occlusions cumbersome.
These kinds of things finally have been addressed with the new version of the Adjust Track Module. Mocha will now supply you with the number of reference points actually needed for the kind of movement you tracked. For example: If you just tracked Translation, you’ll only get one Reference Point to start with. With Rotation and Scaling you’ll get two, with Shear three and with Perspective we’re up to four. However, unlike before, these are just starting points, as they can be appended by any number of additional reference points. All of these points — the initial ones as well as any ones added — can be offset to suit your needs.
Overall, this makes adjusting your track much more intuitive. However, to maintain backwards compatibility and to suit users who grew accustomed to the previous workflow, the old Adjust Track Module is still inside of Mocha 2021, being rebranded as “Classic” Mode.
The OFX Version of Mocha has become more and more popular, especially Flame and Nuke Users seem to enjoy the tighter integration into their workflow and their pipeline.
To help with the latter, the OFX version of Mocha Pro 2021 now features the Python Script Editor, previously only available in the standalone version. Now users proficient in Python (or a friendly TD) can use python for various automations, ranging from scripting tasks to adjusting the U.I.
Under the hood improvements
In addition to the forward facing feature improvements, there have been a number of changes under the hood which help improve performance and will lead to benefits in the future.
With this release, the Mocha Pro plugin now runs “out of process” which means it runs in a separate process from the host application, freeing up resources and improving stability. “One of the main reasons we did this is also to enable the ability to use the Python scripting editor in the plugin,” explains BorisFX’s Ross Shain. “Previously there we conflicts running Python inside a plugin of a host that has its own Python dependencies. Users might notice the plugin takes 1 second longer to launch, but should appreciate the overall experience.”
Another development focus on underlying tech for this release is support for the GStreamer project. This is an open source multimedia framework helps removes Mocha’s older dependency on QuickTime. Now the Mocha standalone application supports more video formats for ingest and render. For editorial and broadcast workflows, the ability render to Apple ProRes has been added. “The QuickTime issue was just something we had to do to move the application forward,” according to Shain. “Boris FX develops plug-ins and applications. For the applications we look at GStreamer as technology that will be used in all our standalone apps including Mocha Pro, standalone version of Particle Illusion, and will be coming to Silhouette.
Moving forward, the team aims to bring more GPU acceleration across Mocha Pro modules and they are also looking at supporting technology such as Apple’s Metal. In addition, they have big plans for adding even more features to PowerMesh. “For PowerMesh, we are just scratching the surface,” says Shain. “Just like the core planar tracker, we see the sub-planar tracking and denser data being incorporated into more modules and workflows. It is a great accomplishment from our dev team and we will continue to develop it in coming versions.”
While the improvements to Adjust Track and the OFX version are welcome additions to this upgrade, the real game changer is PowerMesh. This is really a feature not immediately associated with a planar tracker and will no doubt greatly improve in further making Mocha the problem solver tool it already is. It will be interesting to see what kind of different workflows will be derived from this new feature.