LogInThe proposed framework utilizes a PBR diffusion model in combination with a frozen RGB model.
Unity Technologies' research team, in collaboration with Stabile Diffusion developer Stability AI, has recently published a research paper outlining an innovative approach for the geometry-conditioned generation of PBR images.
The proposed method models the joint distribution directly, avoiding issues surrounding photometric consistency and inverse rendering. To address existing paradigms' limitations for cross-modal fine-tuning, the framework employs a frozen RGB model and links a newly trained PBR model using a novel cross-network communication approach, resulting in the production of high-quality PBR images conditioned on geometry and prompts. Compatible with any control method in a plug-and-play fashion, the framework enables one to generate texture maps for 3D meshes using only text prompts.
"We have shown that this bi-directional control paradigm is extremely data efficient while retaining the high quality and expressiveness of the base RGB model, even when faced with text queries completely out of distribution for the PBR training data," commented the team. "The plug-and-play nature of our proposed approach is compatible with existing adaptations of the base RGB model, which we have illustrated with IP-Adapter for style guidance of the PBR content. The availability of high-quality PBR content generation as offered by our proposed approach opens up new avenues for graphics applications, specifically in Text-to-Texture."
You can read the full research paper and try out Collaborative Control yourself by clicking this link.
Also, don't forget to join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.