New Diffusive 3D Human Recovery Approach

During CVPR 2024, a group of researchers from Rutgers University presented Score-Guided Human Mesh Recovery (ScoreHMR), a novel approach for solving inverse problems for 3D human pose and shape reconstruction. Like traditional model fitting approaches, ScoreHMR fits a human body model to image observations, yet the alignment with the image observation is achieved through score guidance in the latent space of a diffusion model.

This diffusion model is trained to capture the conditional distribution of the human model parameters given an input image, and by guiding its denoising process with a task-specific score, ScoreHMR solves inverse problems for various applications with no need for retraining the task-agnostic model. According to the developers, ScoreHMR is superior over several benchmarks and settings, including single-frame model fitting, and reconstruction from multiple uncalibrated views and video sequences.

This is a comparison of ScoreHMR to an optimization approach (ProHMR-fitting) for temporal model fitting to 2D keypoint detections. ProHMR-fitting has more jitter and can sometimes fail on hard poses or unusual viewpoints.

In the demo below, ScoreHMR and ProHMR-fitting are run on top of ProHMR-regression. ScoreHMR can effectively refine the less accurate ProHMR-regression estimate, and results in more faithful 3D reconstructions than the baselines.

The developers also compared their approach (green) with ProHMR-fitting (blue) and SMPLify (grey). ScoreHMR achieves more faithful reconstructions than the optimization baselines.

All papers, code, and models are available here. Check them out and don't forget to join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.