LogInLead Product Manager at Adobe Baptiste Manteau has shared an extensive breakdown of Substance 3D Sampler's most recent release and discussed its novel photogrammetry feature 3D Capture.
Introduction
80.lv: Please introduce yourself. What companies have you worked for in the past, and how did you join the Adobe Substance 3D team?
Baptiste Manteau: I’m Baptiste Manteau, Lead Product Manager in the Substance 3D team working on Sampler, Integrations, and Automation. I joined the team in April 2017 when we were Allegorithmic in Lyon, France. At this time, there were five of us sharing one office to kick off the development of Sampler (formerly known as Alchemist).
Before joining the team, I discovered 3D during my Architecture and Civil Engineering studies, designing buildings in Sketchup and Photoshop. After that, I spent a few years in an e-commerce company based in Africa and then joined Allegorithmic. I had the opportunity to join the Sampler team at a very early stage to bootstrap the future of a new product with an amazing team. Personally, it was also the occasion to (re)discover 3D and how much this world evolved in a few years.
The History of Sampler
80.lv: For people that might not be familiar with Sampler, can you give us a bit of the history of the product?
Baptiste Manteau: Sampler, formerly known as Alchemist, really started in the material creation world. We wanted to offer an accessible tool to easily create and edit materials by leveraging and combining the best technologies – Substance Designer power, AI, etc.
We leveraged the existing Substance community by sharing early versions of Alchemist to help us iterate and develop the product. A strong need became clear around material digitization, and that was our first focus until the 1.0 release in November 2019. But we didn’t want to stop at just material creation. In June 2021, we introduced the digitization and authoring of environment lights. And today, we have come full circle with 3D objects!
Developing a New Version of Sampler
80.lv: When did you start working on the new Sampler build? What were your main goals?
Baptiste Manteau: Our main goal is to provide an accessible professional end-to-end workflow to capture human-size objects. Today, the photogrammetry pipeline can be really fragmented and complex. Just to get started, you need to learn multiple software that can be very complex and provide so many controls and possibilities that you can easily be lost at first. So, we set out to create a photogrammetry tool in Sampler that was straightforward and used automation to simplify and streamline the process.
Development really started this summer, but I can tell you that we had many passionate and intense design and tech discussions before that. To accelerate the development, we partnered with Bentley for Windows and Apple on MacOS to use their powerful technologies for photogrammetry. We also looked to Adobe’s own Sensei AI tools for opportunities to automate or simplify processes such as image prep, filling holes in materials, and masking. This allowed us to really build an accessible user experience connecting multiple technologies without compromising the quality of the results.
Introducing such a big feature in Sampler was a big challenge in multiple aspects, but the team was so focused and determined that we managed in a few months to ship the first version that matched our goal and vision, and it’s just been getting better and better since then.
The 3D Capture Tool
80.lv: Let’s talk about 3D Capture, could you please tell us about this new tool and its strengths?
Baptiste Manteau: What we really wanted to create is an end-to-end workflow. Besides the great photogrammetry tech that we use under the hood, our first goal was to simplify the user experience covering all steps of a 3D capture workflow from the dataset preparation using the auto-masking to the mesh optimization with connected processes such as decimation, AutoUV, texture reprojection, and baking normal, height & AO maps.
In other photogrammetry workflows, the mesh optimization step can be tedious and requires multiple steps in multiple applications. With 3D Capture in Sampler, you just need to define the target face number, and Sampler does the rest to give you an optimized version of your mesh with maximum details.
Our second goal in this version was how we want to keep the process non-destructive. After reconstruction, we keep all stages of the 3D object in Sampler’s layer stack, allowing users to tweak the mesh results and the textures at any time. And this is just the beginning. Figuring out how to deliver this workflow opens the door to many new applications and filters in Sampler’s layer stack.
80.lv: What are its use cases?
Baptiste Manteau: Many companies, industries, and users have the need to create immersive experiences or use 3D content on their platforms (e-commerce, virtual showroom, etc.), however, the skill and time needed to create a model from scratch in traditional 3D tools can be problematic. We see our 3D Capture workflow as the perfect way to start creating 3D content as an alternative to methods that require time and advanced 3D skills. What we’ve seen is that many of our beta testers – many of whom are very proficient modelers – are really excited by 3D Capture as a way to get to a base mesh faster. So in some ways, it’s for anyone that needs to create and use 3D content.
With Sampler in the Substance 3D Collection plus Adobe Creative Cloud, users have access to a wide range of tools. 3D Capture could help you get a first shape with good proportions to be used as a base in Substance 3D Modeler. After Effects just introduced support of 3D textured objects in their current beta. I strongly believe that VFX and Motion graphic artists will also benefit from an accessible and professional solution to create 3D content for their videos and animations.
We also realize that advanced photogrammetry users today will prefer to stay in their existing tool and have all the controls and parameters they have known for years but our solution is the perfect alternative for creative users to start creating 3D content for their needs. But I would encourage anyone that has access to Sampler to try out 3D Capture and let us know what you think. We strive to build these tools with users for users, and we love feedback.
Utilizing Machine Learning
80.lv: Could you tell us about the ML-powered Adobe tech that helped you build the tool? What foundation did you use?
Baptiste Manteau: At Adobe, we have the chance to work with amazing researchers, computer scientists, and product teams. And we knew that one of the biggest headaches with photogrammetry was the need to mask out the background on photos to focus and for faster and more accurate reconstruction in 3D. We looked at the ML-powered solution called object selection developed by the Photoshop team to automatically detect people, animals, and objects in photos, and we adapted it to our workflow to limit the need to create masks manually. We took the same approach with Content-Aware Fill, a well-known feature of Photoshop, and used it in Sampler to fill in holes in materials. So this ability to take advantage of Adobe technology from other products and apply it to our Substance 3D tools is really exciting. Within the Substance 3D team, we believe that AI & ML are amazing technologies when they’re used to help artists and let them express their creativity and get great results faster.
Other New Features
80.lv: In addition to the new 3D Capture tool, what are some of the other new features added to Sampler recently?
Baptiste Manteau: Last year was a big one for us. Our main objective was to evolve Sampler into a complete material toolbox. We first introduced an end-to-end physical size workflow to match the real-life physical size of your samples in a digital context.
2022 was also the year of exporting parametric materials by exposing parameters to keep your material editable in other software (as you can do in Substance Designer). It helps speed up the iteration process when using the material in another context and creating multiple materials from one.
We also introduced a complete Python API to customize and automate your workflow in Sampler by creating plugins with your own UI and scripts for batching operations.
Creative Cloud
80.lv: Sampler also has some integrations with Creative Cloud tools, can you tell us a bit about those workflows and how you and the team prioritize what workflows to include?
Baptiste Manteau: Today, Sampler is connected to Photoshop and Illustrator. Any image or texture generated can be opened from Sampler in Photoshop for advanced editing with automatic reload in Sampler. We have the same mechanism with Illustrator to edit vector files and accelerate the iteration of graphics design and materials creation. These workflows have been especially important if fields like fashion and product design, where 3D is often used alongside traditional 2D tools and workflows.
It's also key to how we build all the Substance tools – all assets created in Sampler are made to be used throughout the Substance 3D ecosystem for texturing and rendering. And it's important that they can be used in other non-Adobe 3D tools. For us, it's critical to be connected to our ecosystem and our users' ecosystems in ways that reduce friction as you move from one creative task to another and between the various tools used to create.
Collaborating With Other Adobe Teams
80.lv: As part of Adobe, how does your collaboration with Adobe teams help you prepare better releases in general?
Baptiste Manteau: It’s been four years since we joined Adobe, but we’re still discovering it every day. When preparing a new development project, the first step is to identify the right people and technologies within Adobe! We meet with Adobe Research and discuss with other product teams to understand if they are already working on a similar problem to get a sense of what we could leverage. Often, we find help and team up.
When we’re facing a new unsolved problem, we collaborate with Adobe Research to define use cases, create a dataset of examples, and iterate until everyone is satisfied with the solution.
Adobe Automated Tools
80.lv: What Substance automated tools does the new release use? How did previous advancements help you create new features?
Baptiste Manteau: The introduction of 3D Capture has long been anticipated as soon as we decided to go beyond materials. We had the ambition to be the first tool to digitize materials, environment lights, and 3D objects. All our previous developments considered this big change. Be ready for some under-the-hood tech reveals, as under the hood, Sampler layer stack is a node graph composing different results (material, image, value).
We leveraged all the work done to introduce a new type of asset, mesh, in our graph. This allowed us to have a unique layer stack that combines mesh and material data for advanced authoring and it opens tons of possibilities for the future.
Working With User Feedback
80.lv: What feedback did you get from users testing 3D Capture? How did the feedback help you improve the release?
Baptiste Manteau: We had the chance to collaborate with amazing artists during the beta phase. I want to take the opportunity to thank them all! The first reactions were so great when we announced our intention to introduce a 3D Capture feature. Beta testers had access to the very first versions end of July with just the minimum workflow – drop your photos and get a result in the file explorer – It was very rough, but it worked and gave good results!! This helped us confirm the quality and robustness of the reconstruction process. Tester feedback directly drove the development and the prioritization of what we should focus on to improve results and the experience, and with each new beta release, we felt the excitement growing internally and externally. This gave us tons of energy and ideas!
Conclusion
80.lv: What is next for the Sampler team? Could you share some behind-the-scenes details on your plans? What should users be excited about next?
Baptiste Manteau: First, get some rest! On a more serious note, we want to consolidate and improve the 3D Capture with all feedback we will get with the release. We don’t want to stop there on this topic, as there’s always room to improve results and efficiency. I can’t tell you anything specific about upcoming releases, but I can say that now that 3D Capture is shipping in Sampler, we’re already working on some new exciting AI capabilities for material creation – stay tuned!