LogInMeet WHAM, a novel framework that can capture 3D human motion in diverse conditions.
Researchers Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J. Black from Carnegie Mellon University and Max Planck Institute for Intelligent System have introduced WHAM (World-grounded Humans with Accurate Motion), a brand-new framework for accurate reconstruction of 3D human motion from videos.
According to the team, WHAM learns to translate 2D keypoint sequences into 3D by incorporating motion capture data and merging it with video features, seamlessly integrating both motion context and visual information.
Subsequently, WHAM utilizes the estimated camera angular velocity from a SLAM method in conjunction with human motion to determine the body's global trajectory. To enhance accuracy, the framework employs a contact-aware trajectory refinement method, enabling WHAM to effectively capture human motion in diverse conditions.
"WHAM takes the sequence of 2D keypoints estimated by a pre-trained detector and encodes it to the motion feature," commented the team. "WHAM then updates the motion feature using another sequence of image features extracted from the image encoder through the feature integrator. From the updated motion feature, the Local Motion Decoder estimates 3D motion in the camera coordinate system and foot-ground contact probability.
The Trajectory Decoder takes the motion feature and camera angular velocity to initially estimate the global root orientation and egocentric velocity, which are then updated through the Trajectory Refiner using the foot-ground contact. The final output of WHAM is pixel-aligned 3D human motion with the 3D trajectory in the global coordinates."
Learn more about WHAM here and don't forget to join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.