New AI Imaging Technique from NVIDIA

A team of researchers at NVIDIA has presented a new state-of-the-art deep learning method.

A team of researchers at NVIDIA has presented a new state-of-the-art deep learning method that is capable of editing images and reconstructing a corrupted image with holes or is missing pixels. The team also states that it can be used to edit images by removing content and filling in the resulting holes. The method revolves around a process called “image inpainting” which could be implemented into photo editing tools to remove unwanted content by adding a realistic computer-generated alternative.

Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.

Image Inpainting for Irregular Holes Using Partial Convolutions 

“Our model can robustly handle holes of any shape, size location, or distance from the image borders. Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing,” the NVIDIA researchers said in their research paper. “Further, our model gracefully handles holes of increasing size.”

The researchers started by generating 55,116 masks of random streaks and holes of arbitrary shapes and sizes for training. 25,000 more have been generated for testing.

An example of the masks generated for training.

During the training phase, holes or missing parts are said to be introduced into complete training images from the above datasets in order to enable the network to learn to reconstruct the missing pixels. Then, at the testing phase, different holes or missing parts, not applied during training, are introduced into the test images in the dataset by the team, to perform unbiased validation of reconstruction accuracy.

The outputs for missing pixels used to depend on the value of the input that was supplied to the neural network for those missing pixels, which led to artifacts (color discrepancy and blurriness, for example). The NVIDIA team developed a method which uses a “partial convolution” layer to solve the problem.

Are you excited? You can learn more about the news here

Join discussion

Comments 0

    You might also like

    We need your consent

    We use cookies on this website to make your browsing experience better. By using the site you agree to our use of cookies.Learn more