Cubic Motion’s complete Persona facial solution

Cubic Motion is the world leader in real-time model-based computer vision and digital facial animation. The company does work for a variety of major AAA games for companies around the world. In addition to this, Cubic Motion has built an incredible record in realtime facial capture for live environments. The team has amazed an incredible track record from the Senua HellBlade project to Siren last year. They have been a main stay of cutting edge face research for years at SIGGRAPH, GDC and in Europe.

Cubic Motion handle al the stages after rigging to live data deployment, typically in UE4
Manchester head offices

The company recently launched Persona an end to end hardware and software solution for emotional performance capture. Designed from the ground up for live performances, Persona enables immediate character animation in real-time engines normally in Epic Games’ UE4.

The system supports multiple actors each wearing a bespoke HMC and on-body animation system facilitating a completely untethered performance and allowing the actor to move freely around the stage or volume. We visited the company in Manchester to see the system first hand.

“Persona is pretty much the productization of all we have been doing in real time over the last couple of years” – Steve Caulkin, Chief technical officer, Cubic Motion.

Cubic Motion brings digital characters to life across video games, films, broadcast media, and immersive technologies. The company aims to fuse science and artistry with their own Computer Vision and Machine Learning alchemy to provide both offline and real-time model-based digital animation.

Persona is powered by the next generation of Cubic Motion’s high fidelity, facial animation technologies. This is built on the experience learned at a host of live events including GDC and FMX and that won the 2016 SIGGRAPH award for Real-Time Live. The company provided the facial tracking for fxguide’s own Mike Seymour as MEET MIKE in 2017. These earlier award winning projects advanced the companies proprietary computer vision techniques which are now available to anyone in Persona.

The tech in Persona uses the next generation of facial tracking and solving at Cubic Motion, as product manager Tony Lacey explained “we’ve designed the new HMC, the on body system and all the software . It is all new and its been developed over the last year here at Cubic Motion”.

The system can deliver animated characters who respond and engage with real-world events immediately, and whose performance may be interwoven with real human actors. The system aims to provide a faithful reproduction of the actor’s performance whether for immediate broadcast at a live event or for character previsualisation during capture.

“Every Persona system comprises of the head mounted camera system, the on body computer and battery packs. This does a full track solve, streaming the data back on set wirelessly. Each actor has an HMC and their own small computer. Persona also comprises the management system which sits on a separate computer at the side of the capture area. From there you can manage any configuration and monitor all the multiple actors from a single point” Lacey explains. The data stream that comes from each actor is directed to the Unreal Engine plugin. This provides the realtime output animation. The configuration and monitoring software can run on a separate notebook or tablet computer. This allows the technical director to move themselves wirelessly on set, while still controlling each of the actor’s Persona systems. Key is the reduced data that flows from each actor’s Persona computers to the Unreal plugin. As the full video footage is not being streamed, the system is robust and able to work without special cabling or transmitters.

Each system comes in a road case and is designed for studio or location work. The team aim to service both live and more traditional high end MoCap projects. The system is not designed for consumer use, but rather premium or professional high end productions.

HMC

Persona supports multiple actors each wearing a bespoke Head Mounted Camera system (HMC) and on-body animation system facilitating a completely untethered performance. The core HMC head unit is 3D printed in a set of difference sizes, and custom HMC head rigs can even be ordered. The new HMC units are light weight and very well padded and counter weighted to reduce neck and head stain.

Note the side and front cameras

The HMC is designed with the new side and front camera array that Cubic Motion has been perfecting. Unlike some system with just dual mounted stereo cameras, Persona‘s HMC carries Cubic Motion’s signature front-facing and profile camera configuration. The HMC also has built in lighting that allows the system to operate in a wide variety of conditions, but as the lights are infrared, they are not off-putting or annoying to the MoCap actor. The inbuilt infrared lighting ensures unambiguous recovery of even the most subtle mouth shapes and poses under a variety of environmental conditions.

Having fitted and tested one of the HMCs units in Manchester, we can testify that the redesigned light weight form factor and generous padding, coupled with balanced weight and infrared lights makes the system very actor friendly.

On Board Computer

The HMC is connected to a portable computer worn by the performer. This is one of the major difference between Persona and other systems. The on board computer does actual processing of the video coming from the HMC. Only a much smaller complex data stream and ‘preview’ video stream are sent from the on board system to the control system. The original data is also recorded on the actors computer, for later reference if needed.

This allows the actor to move freely around the stage or performance volume. The Persona HMC system is Cubic Motion’s proprietary design. We found the Persona system was optimised for comfort and ease of use allowing Persona to deliver superb results over prolonged periods. Having personally worn HMCs for days on end in a ‘production’ type environment, comfort is a major aspect of any HMCs success or failure.

Cubic has a large team of artists, experienced in building custom solutions

Performance Pipeline

The facial performance is captured and solved onto the target character by the ‘Performance Pipeline’. This is the on-body computing appliance that is the heart of each actor’s setup. The pipeline computer is a lightweight and battery powered unit. It may run either tethered or wirelessly and comes with a flexible harness system.

Individual Identities

Each Performance Pipeline is  configured using the Identity System software. The Identity system is unique to each person. It ensures a faithful representation of the actor’s performance is acquired and that an engaging and intimate portrayal of the character is delivered. Cubic Motion’s Identity System is perhaps the second key to the Persona system. It is a purpose-built tracking model for each actor.  The turnkey Persona system includes matching each actor with a bespoke solver model for their character rig. Each Identity is constructed using state of the art machine learning algorithms. These analyse the appearance of the actor and the capabilities of the character rig in order to specifically tailor Persona for every unique human and digital character pairing.

This pairing can be turned around in a few days. A set of recordings are sent to Cubic Motion, based on a specific Range of Motion (ROM) instructions and some general performance footage. Additionally, the target CG head rig file is also sent. The team at Cubic Motion then build the specific Persona pairing files.  This individual solution is required for the extremely high end results the company is known for.

At the lower end of the market, iphone rigs are now not uncommon, but these can not reach the level of subtle performance capture that the Persona rig is able to provide. Persona ‘out sources’ to Cubic Motion team the calculation of the highly complex facial tracking solutions that convert the video streams into high fidelity rig commands. By taking this approach, Cubic Motion hopes to take their years of experience in solving faces and make it widely available to any serious high end production world wide.

Performance Manager

Performance Manager is a GUI application that allows MoCap operators to oversee all performances.  It provides the ability to configure and monitor every aspect of an actor’s performance from a remote interface.  Actors are automatically discovered by Performance Manager and single instance can be used to manage multiple actors simultaneously. The software can run on a PC or tablet as the video stream is just the compressed review video and not the raw high quality feed that goes direct into the on-board computer.

While the system is focused on facial animation, it is also design to integrate with common MoCap suits and provides all the timecode and sync signals needed for a full body complete solution.

Summary

HMCHelmet mounted camera system designed specifically to support live.

  • 3 Helmet sizes available with selection of padding to provide a bespoke fit
  • Dual booms with twin Ximea 1K cameras in front-side configuration
  • Built-in IR lighting & timecode sync electronics.
  • Mobile app to manage lights and timecode.

Performance PipelineCapture, track, solve & stream animation in real-time direct from actor.

  • On-body appliance pre-installed with all software, worn by the actor for an untethered performance.
  • High fidelity tracker & posed based solver pipeline able to deliver animations at 60 FPS.
  • Wired or wireless streaming of animations directly into Epic Unreal Engine V 4.19 & later.

Other Components

  • 2x batteries with charger giving a typical uninterrupted performance time of 1hour/battery.
  • Actor harness with configurable mounting system.
  • Individual Peli Case for each performer.

Performance Manager

  • Manage 4 simultaneous pipelines in real-time from a single seat.
  • MS Windows GUI application.
  • Remotely monitor pipeline HMC’s including all camera, tracking & animation feeds.
  • Control the broadcast, recording and export of performance data.
  • Import and manage Cubic Motion Identities and assign to pipelines.