LogInResearchers from Adobe Research and The University of British Columbia presented Interactive Neural Video Editing (INVE) – a real-time video editing tool that can change and track objects in video footage.
It is supposed to "assist the video editing process by consistently propagating sparse frame edits to the entire video clip." Simply put, you can add an image to the video or draw something, and the algorithm will integrate the change in the footage.
The researchers used "highly efficient network architectures, powered by hash-grids encoding, to substantially improve processing speed. In addition, [they] learn bi-directional functions between image-atlas and introduce vectorized editing, which collectively enables a much greater variety of edits in both the atlas and the frames directly."
This layered editing pipeline supports various types of edits:
- Sketch Edits, where users can sketch scribbles using the brush tool;
- Local Adjustments, users can apply local adjustments (brightness, saturation, hue) to a specific region in the scene;
- Texture Edits, users can import external graphics that track and deforms with the moving object.
INVE is inspired by Layered Neural Atlas (LNA) – a research presenting a method that decomposes an input video into a set of layered 2D atlases. INVE creators believe their method "reduces the learning and inferencing time by a factor of 5 and supports various video editing operations that LNA cannot."
I think the authors could have shown more serious use cases to make more of an impression, but the tech behind it is still inspiring and makes you think about all the possibilities there will be in a couple of years.
If you'd like to learn more about INVE, check the paper out here and don't forget to join our 80 Level Talent platform and our Telegram channel, follow us on Threads, Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.