Headshot is a new AI-powered Character Creator plugin that generates 3D real-time digital humans from a single photo. This new technology can be seen at Real-time Live at SIGGRAPH ASIA (Wed 4pm). We spoke to the team about this new technology and user-friendly tool of both AI and traditional 3D.
ABOVE: Preview of one of the RTL demos at SIGGRAPH ASIA
Tomorrow at 4pm at Real-time Live (RTL) at SIGGRAPH Asia in Brisbane, John Martin, Vice-President of Reallusion, is showing the technology behind Headshot. The Reallusion RTL demo is one of seven presentations at RTL (but it is the only one not in competition).
Martin’s demo maps out the story of a director asking for a 3D real-time animation of a scene with an actor, – based on only one Jpeg headshot photo of the actor, and all done from start to final live animation in just 9 mins.
The RTL demo uses the technology of Headshot. This new tool uses intelligent texture blending and head mesh creation to produce a believable rigged face with hair and clothes but done with an artist-friendly user interface that allows for computer-added artist control. As part of the process, the system provides both preview and advanced modeling tools to allow an artist to adjust the synthetic interpretation of the face.
The output is a generated digital double that is fully rigged for voice lipsync, facial expression, and full-body animation. Headshot supports two AI modes: Pro mode and Auto mode.
The Pro mode includes Headshot’s 1000+ sculpting morphs, image mapping, and texture reprojection tools. The Pro Mode is specifically designed for production-level hi-res texture processing (4096×4096) and artist face shape refinement. The Auto Mode makes a lower-res virtual head, with additional 3D hair, in a fully automatic process.
The program initially creates a face based on Machine Learning of the input Jpeg.
It automatically locks the 3D model to the image layer based on the analyzed camera lens from the jpeg. This lets the artist easily align the scene camera to the orientation of the photo. The system then allows adjustment of the 3D head meshes using on-screen Sculpt Morph or Headshot Morph Sliders
Even from the initial creation based on a single Jpeg, Headshot can achieve a close resemblance from the first fitting. From there, an artist is able to utilize the sculpt morphing tools to shape and refine the face.
This fine-tuning process relies on some “500 to 600 face morphs very systematically designed so you can fine-tune every detail”, explains Charles Chen CEO, Reallusion. The combination of editable morph shapes and re-projection produces almost multi-camera scan level outputs. The process works with an Image Matching Layer which can be overlayed as a reference plane for advanced head shape refinement. Users can vary the layer opacity to verify the photo-mesh alignment. The principle is very simple, the overlay allows the artist to check the AI results and use an intelligent UI to adjust the new model. “You can just mouse over and then immediately see what would be affected, it is a type of sculpting mode that quickly lets you see what part of the face, will be controlled, .. it is very intuitive,” Chen points out.
Headshot has a new set of morph targets specifically created for photoalignment. While the manual adjustments should normally be small, headshot morph adjustments are designed to be correctly isolated, for a clean adjustment. For example, an adjustment of an eyelid will work with minimum effect on the surrounding face, and thus also animate well.
Headshot heads are fully rigged and ready for animation with iClone or exported to real-time engines such as Epic games’ UE4. There are also a set of presets to blend the image projections with the base textures, which use different skin types and masking areas. “You can choose different skin types, depending on if you are matching an old man, who would have more rough skin, or a young woman with very soft skin,” explains Chen. “Even if someone has a beard, we can work with this, which gives you incredible freedom.”
Additional textures provided for realistic digital human rendering include normal, AO, roughness, metallic, SSS and Micro Normal.
The reprojection helps greatly with avoiding the softening of the textures from morph shape manipulation. Reprojection can be done at any time in the adjustment process.
There is also the capacity to adjust any texture-to-mesh alignment discrepancies resulting from the texture generation, as well as lens correction. The system applies a UV reprojection to bake textures onto the newly photo-aligned head mesh.
Reallusion is also now shipping a new Digital Human shader seen implemented here in some of the example shots above. This is based on the Unreal UE4 Mike Seymour shader (UE 4.22). This is now a standard shader shipping in Character Creator 3.2, which was released yesterday. It provides a new improved subsurface scattering and micro-normal texture channels.
The Auto mode allows for automatic generation of real-time 3D head models but it also separates hair meshes. The Reallusion intelligent process automatically removes hair from the doubles images, and provides natural side textures. It is tolerant to face shadows as such the Auto Mode is suitable for creating mid-res crowds for film, or NPC characters in 3D games. “Automatic mode has one drawback,” explains Chen, “The resolution’s not as high as Pro mode, as our original training data was not as high res.”
The Headshot-generated-3D-hair is fully compatible with iClone’s Character Creator’s hair format (.ccHair). These can be added into the hair library, and then applied to other CC characters.
“Auto mode is the quick solution, you can use it for something like an NPC or any background character that is in a mid to long shot. For close up shots, you should use the Pro mode,” explains Chen. “But you know this whole system is like a Police station Identikit”, he jokes. “You have all the capability to create a face look like any possible human look”.