The Champion directed by Maciej Barczewski, is the first full theatrical feature film to be converted to English using neural rendering. All of the performances were changed so the actors not only are revoiced into English, but their faces were replaced so the actors are also visibly speaking English as well.
The film tells the true story of pre-war boxing champion Tadeusz “Teddy” Pietrzykowski, who in 1940 arrives with the first transport of prisoners to the newly created Auschwitz concentration camp. Using advances in AI and machine learning the entire film has the actor’s faces replaced by an inferred version, visually built from actors re-recording dialogue in a sound studio.
The film was shot in January 2020. The movie was initially scheduled for release in October 2020 but has been delayed due to Covid until March 2021 in Polish. The film was successful in Poland and a creative technology company and distributor with headquarters in Tel Aviv; Adapt Entertainment, secured the rights to release the film in various English markets. CEO of Adapt, Darryl Marks had long wanted to use visual effects or neural rendering to convert a film. Darryl Marks’s quest for the last few years has been to “broaden the global reach of content so audiences can experience an enhanced storytelling alternative that maintains the integrity of the original project without dubbing or subtitles.” Adapt put together a team including Pinscreen Technology in LA, headed by Dr. Hao Li, a world specialist in neural rendering. It also included fxguide’s Dr. Mike Seymour, an expert in digital humans, who served as the English version VFX supervisor. Together the team developed Adapt’s proprietary Plato process incorporating new advanced machine learning and neural rendering.
The film stars the award-winning Piotr Głowacki, a Polish actor who has appeared in more than sixty films since 2004. To be as authentic as possible the team was keen to get the original actors where possible to re-record their performances. Thankfully Piotr was able to lead the European voice recordings in Poland. Even with the large variations in his appearance in the film, Piotr was recorded in one session by five cameras. His recordings were supervised by the film’s director Maciej Barczewski and supported by Darryl Marks who flew to Poland for the audio sessions.
The Plato process is a professional neural rendering pipeline for dialogue and facial reanimation.
What makes it unique is three things
First and foremost, the Plato pipeline is designed around maintaining the performance of the actors by replacing their original language performance with a faithful high fidelity new version.
Secondly and very importantly, it is a production pipeline, so unlike some great tests and demo pieces it works with
- It only needs limited training data, this is huge. It only used the finished film; no special work is needed to be done when the original film is being shot.
- Plato is achieved with unobtrusive filming of the new dialogue – so no special lights or facial markers or complex intrusive tech
- It works with all different types of dramatic lighting, head and camera angles, notably side head angles, even when the actor is wearing special effects makeup.
Thirdly, it works at volume. Some processes required tens, if not hundreds of hours per shot to accomplish. This is okay for an ad or TikTok piece, but not for a feature film with hundreds and hundreds of shots.
The Plato process is unique in delivering at volume while also being flexible enough to have the technical artists control the process, should the director want to adjust the dialogue. This is because it is not just a single black-box machine learning algorithm. Plato is a complex series of carefully designed new technologies, bespoke implementations, and industry best practice – that delivers professional high quality, high-fidelity, and authentic performances.
Pinscreen has long been researching and developing neural rendering applications. The team has done extensive research into avatars, digital agents, and machine learning. The company has been a regular contributor to SIGGRAPH and numerous Real-Time Live events. While part of the Plato process involves pre-existing ML technology the team had developed, they undertook a specialist research project with Adapt prior to the Champion to develop the new technology needed to meet the demands of a production pipeline. Zimo Li was the head researcher and team leader for the ML team based in the USA but with the team working remotely due to COVID restrictions.
It has been shown to work on day and night footage, front on and side on shots, – women, men, or children, and even action sequences. This moves dialogue replacement neural rendering from custom hand finishing to industrial level applications and thus provides producers a whole new range of options for financing and distribution of quality films and TV shows.
The Plato pipeline allows for some shots to be individually adjusted if desired and this part of the team worked remotely in a distributed post-COVID arrangement. As the re-recording was supervised via zoom in Poland, the VFX editing was in Australia, and the neural rendering was done in LA and Sydney, – not to mention the producers being in Israel, the team relied heavily on frame.io for the vast amount of film clips and training data involved.
This style of reimaging the faces of actors in an authentic way could vastly expand the market for foreign films. In intensely emotionally dramatic films such as The Champion, it takes the viewer out of the moment to have to switch their gaze from the actor’s performances to read the subtitles. Furthermore, traditional overdubbing can break the authenticity and fidelity of the performances, often clashing the original tone and cadence with the new narrative language.
The film dubbing market was valued at $2.4B in 2019 and is projected to reach $3.6B by 2027. There is a growing appetite from Hollywood for international content. In the last few years there has been great interest in foreign films with the success of Squid Game and films such as Parasite, The Worst Person in the World, and Drive my car. Despite these projects’ success, subtitles are still an obstacle for studios trying to tap into the international market. Adapt has already done selected shots in other Netflix films, for small scene inferred reenactments, but The Champion is the first fully ‘adapted’ feature film using neural rendering technology.
The Champion is a powerful and emotional film that now appears to an audience to have been shot remarkable in both languages for each version of the film. The Plato dialogue replacement process provides neural rendering with a new positive application, beyond deep fakes, and it’s common satirical, or ‘wow factor’ demonstrations. The actors and creative team believe their delivery, acting choices, and direction have been served in both languages.