LogIn
SpatialTracker "sees" movement even in hard-to-track situations.
Image credit: Yuxi Xiao, Qianqian Wang et al.
Researchers from Zhejiang University, UC Berkeley, and Ant Group presented SpatialTracker, a new method of tracking 2D pixels in 3D scenes.
The project addresses the issue of recovering dense and long-range pixel motion in videos. Current trackers don't always follow movement right, leading to occlusions and discontinuities, and SpatialTracker is supposed to fix this.
"Our method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth estimators, represents the 3D content of each frame efficiently using a triplane representation, and performs iterative updates using a transformer to estimate 3D trajectories. Tracking in 3D allows us to leverage as-rigid-as-possible (ARAP) constraints while simultaneously learning a rigidity embedding that clusters pixels into different rigid parts."
TAPIR tracker. Image credit: Yuxi Xiao, Qianqian Wang et al.
SpatialTracker. Image credit: Yuxi Xiao, Qianqian Wang et al.
The authors claim their method is more stable and provides better results than other state-of-the-art solutions, "particularly in challenging scenarios such as out-of-plane rotation."
This can be used in a variety of situations, for example, for camera pose estimation. The researchers first manually identify the static parts and estimate the camera poses from wild videos, then estimate the 3D trajectories of pixels in the video using our SpatialTracker, decompose the scene into different rigid parts, and finally estimate the camera poses using PnP for the static part.
With accurate tracking, you could replace a moving object in a video with something else without much problem.
Image credit: Yuxi Xiao, Qianqian Wang et al.
Check out the project here and join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.
Keep reading
You may find these articles interesting
New Free Add-on Brings Object & Camera Tracking to Blender
Automatically Count Your Soccer Ball Juggles Using Object Tracking
KeenTools Released GeoTracker 2023.3 for Blender
This Great Tool For Tracking Changes in Houdini Projects Gets Updated
Wrap4D Developers Showcase Neural Profile That Automates Facial Performance Tracking