Using Neural Rendering to Convert a Foreign Movie to English
Polish Holocaust drama The Champion became the world's first feature film that was converted into English with dialogue replacement using neural rendering.
The Champion, a Polish Holocaust drama that was bought by Adapt Entertainment to release the movie in the English markets, became the first feature film that is converted to English using neural rendering. The actors were not only revoiced into English but with the help of AI and machine learning, their faces were replaced to create the visual image of speaking English.
To convert the film into English, Adapt gathered a team of global specialists in neural rendering and visual effects including Pinscreen Technology, the firm that creates photorealistic AI-driven virtual avatars, and fxguide’s Dr. Mike Seymour, an expert in digital humans. Together the team has created Adapt’s proprietary Plato process – the digital technique that embraces machine learning and neural rendering that is used for dialogue and facial reanimation.
To re-record the actor's facial performances, the original cast was invited to the European voice recordings where they were shot by five cameras from different angles to capture their facial expressions while they were dubbing the film.
With the help of the Plato process that is integrated into the production pipeline, Adapt managed to put the new dialogues into scenes. Despite various facial and camera angles in the film, the technology still allowed to achieve a seamless dialogue replacement.
According to Seymour, the major advantage of the Plato process is the ability to produce a huge number of shots "in a full production environment in very high quality". The technology only needs limited training data and no special work is needed to be done when the original film is being shot. It can work with various types of dramatic lighting and even no special makeup for actors is required.
You can learn more about the film and the technology here. Also, don't forget to join our new Reddit page, our new Telegram channel, follow us on Instagram and Twitter, where we are sharing breakdowns, the latest news, awesome artworks, and more.
