LogInSPARK analyzes videos and then tracks faces in unseen footage.
Kelian Baert et al.
Researchers presented SPARK, a new method that creates a 3D face reconstruction from videos and then enables real-time tracking of unseen footage.
In their work, SPARK: Self-Supervised Personalized Real-Time Monocular Face Capture, they offer more details of the process. SPARK reconstructs a detailed relightable face avatar from multiple portrait videos, "capturing both precise geometry and appearance" to build a personalized decoder using inverse rendering.
In this stage, called MultiFLARE, they "disentangle the color into illumination and intrinsic material" and "adapt a generalizable feedforward 3D face capture model" by replacing its decoder with the custom geometry model and "tuning its encoder on the video collection using the pre-computed reflectance functions."
"Using our pre-estimated image formation model, we obtain a more precise self-supervision objective, enabling improved expression and pose alignment. This results in a trained encoder capable of efficiently regressing pose and expression parameters in real-time from previously unseen images, which combined with our personalized geometry model yields more accurate and high fidelity mesh inference."
Kelian Baert et al.
SPARK shows admirable results, at least in the researchers' materials. The creators say this method can be used for face editing or other visual effects applications.
You can't try it out yet, but if this project interests you, read the full paper for more. Also, join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, LinkedIn, TikTok, and Reddit, where we share breakdowns, the latest news, awesome artworks, and more.