NVIDIA Study: World-Consistent Video-to-Video Synthesis

NVIDIA has shared a paper presenting a new vid2vid method that helps generate photorealistic videos from semantic inputs using AI memory. For such an approach, the team has created a new tool that uses prior generated 3D world data and applies it to the new objects. To provide the world structure data, the team has introduced guidance images, physically grounded estimates that show how an object should look like.

In total, the study introduces a new architecture based on the multi-SPADE module that uses semantic models, optical-flow warping, and guidance images to create hyperrealistic and smooth videos.

Check the full paper on Github.