NeuralMarker: Switching Images in a Video Using AI
The images look neat even in tough conditions.
Researchers from Zhejiang University, The Chinese University of Hong Kong, and NVIDIA presented NeuralMarker – an AI-powered framework that can help replace one image with another in a video. The framework seems to be working well with different viewpoints, lighting conditions, deformations, and motion blur.
NeuralMarker trains a neural network "estimating dense marker correspondences under various challenging conditions". The researchers noticed that the challenges of marker correspondence estimation come from geometry variation and appearance variation and designed two components: first is a synthetic dataset FlyingMarkers containing marker-image pairs with ground truth dense correspondences. By training with FlyingMarkers, the neural network is encouraged to capture various marker motions. The second is Symmetric Epipolar Distance (SED) loss, which enables learning dense correspondence from posed images.
According to the authors, learning with SED loss and the cross-lighting posed images collected by Structure-from-Motion (SfM) makes NeuralMarker robust in harsh lighting environments and helps it avoid synthetic image bias.
The code and models should be released soon, so check NeuralMarker out on GitHub, read about the research here, and don't forget to join our Reddit page and our Telegram channel, follow us on Instagram and Twitter, where we share breakdowns, the latest news, awesome artworks, and more.
