LogIn
HUGS uses 3D Gaussian Splatting to make it work.
Image credit: Apple
Another great tech invention has arrived from Apple's researchers, called HUGS. HUGS (Human Gaussian Splats) can create a 3D animatable avatar based on a video using 3D Gaussian Splatting.
3D Gaussian Splatting for Real-Time Radiance Field Rendering is a rendering technique that leverages 3D Gaussians to represent the scene, thus allowing one to synthesize 3D scenes out of 2D footage. Bad Decisions Studio has demonstrated its abilities nicely by turning sequences from famous movies into 3D scenes, which can be edited in DCC software.
HUGS takes a monocular video with a small (50-100) number of frames so that the algorithm learns to disentangle the static scene and a fully animatable human avatar in 30 minutes.
"We utilize the SMPL body model to initialize the human Gaussians. To capture details that are not modeled by SMPL (e.g., cloth, hairs), we allow the 3D Gaussians to deviate from the human body model. Utilizing 3D Gaussians for animated humans brings new challenges, including the artifacts created when articulating the Gaussians."
Image credit: Apple
According to the researchers, this method enables the synthesis of new poses and views. The rendering speed reaches 60 FPS while being about 100 times faster to train than in other works.
HUGS seems like a continuation of NeuMan, Apple's framework that reconstructs people and scenes from a single video. There, the authors trained two NeRF models to estimate the rough geometry and created a warping field from the observation space to the canonical pose-independent space.
Video-to-3D methods are not a novelty, although they don't usually use 3D Gaussian Splatting. If you're interested in this field, check out these articles:
- Researchers Unveil a New Solution For Reconstructing 3D Human Motion From Videos
- Vid2Avatar: 3D Avatar Reconstruction from Videos
- 3Dpresso: New AI Software That Turns Video into 3D models
- NVIDIA Presents New AI Model That Turns 2D Videos Into 3D Structures
- New Generation Method Turns 2D Images into 3D Models
Find Apple's paper here and join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.