Professional Services
Order outsourcing

New Method of Tracking 2D Pixels in 3D Space

SpatialTracker "sees" movement even in hard-to-track situations.

Image credit: Yuxi Xiao, Qianqian Wang et al.

Researchers from Zhejiang University, UC Berkeley, and Ant Group presented SpatialTracker, a new method of tracking 2D pixels in 3D scenes. 

The project addresses the issue of recovering dense and long-range pixel motion in videos. Current trackers don't always follow movement right, leading to occlusions and discontinuities, and SpatialTracker is supposed to fix this.

"Our method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth estimators, represents the 3D content of each frame efficiently using a triplane representation, and performs iterative updates using a transformer to estimate 3D trajectories. Tracking in 3D allows us to leverage as-rigid-as-possible (ARAP) constraints while simultaneously learning a rigidity embedding that clusters pixels into different rigid parts." 

TAPIR tracker. Image credit: Yuxi Xiao, Qianqian Wang et al.

SpatialTracker. Image credit: Yuxi Xiao, Qianqian Wang et al.

The authors claim their method is more stable and provides better results than other state-of-the-art solutions, "particularly in challenging scenarios such as out-of-plane rotation."

This can be used in a variety of situations, for example, for camera pose estimation. The researchers first manually identify the static parts and estimate the camera poses from wild videos, then estimate the 3D trajectories of pixels in the video using our SpatialTracker, decompose the scene into different rigid parts, and finally estimate the camera poses using PnP for the static part.

With accurate tracking, you could replace a moving object in a video with something else without much problem.

Image credit: Yuxi Xiao, Qianqian Wang et al.

Check out the project here and join our 80 Level Talent platform and our Telegram channel, follow us on InstagramTwitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.

Join discussion

Comments 0

    You might also like

    We need your consent

    We use cookies on this website to make your browsing experience better. By using the site you agree to our use of cookies.Learn more