Researchers Present New Method For 4D Face Reconstruction From Any Image Sequence
Face Anything aims to provide stable geometry and tracking, even under large expressions and viewpoint changes.
To address the difficulties of reconstructing and tracking dynamic human faces from various sequences, caused by simultaneous changes in shape, expression, and viewpoint, Umut Kocasari, Simon Giebenhain, Richard Shaw, and Matthias Niessner proposed a new model for high-quality 4D facial reconstruction based on canonical facial point prediction.
Called Face Anything, their approach maps each pixel to a normalized facial coordinate in a shared canonical space. This turns the problem into a canonical reconstruction task, allowing temporally consistent geometry and reliable correspondences within a single feed-forward model.
"By jointly predicting depth and canonical coordinates, our method enables accurate depth estimation, temporally stable reconstruction, dense 3D geometry, and robust facial point tracking within a single architecture. We implement this formulation using a transformer-based model that jointly predicts depth and canonical facial coordinates, trained using multi-view geometry data that non-rigidly warps into the canonical space.
Extensive experiments on image and video benchmarks demonstrate state-of-the-art performance across reconstruction and tracking tasks, achieving approximately 3× lower correspondence error and faster inference than prior dynamic reconstruction methods, while improving depth accuracy by 16%
These results highlight canonical facial point prediction as an effective foundation for unified feed-forward 4D facial reconstruction. The model, dataset, and code will be publicly available."
Results of Face Anything on NeRSemble and VFHQ:
You can find more details about Face Anything, including its architecture, dataset creation process, and comparisons with state-of-the-art tracking methods, here. The GitHub repository will be published soon.
Subscribe to our Newsletter, join our 80 Level Talent platform, and follow us on Twitter, LinkedIn, Telegram, and Instagram, where we share breakdowns, the latest news, awesome artworks, and more.
Are you a fan of what we do here at 80 Level? Then make sure to set us as a Preferred Source on Google to see more of our content in your feed.
