Creating a Photorealistic Chocolate Panettone With RealityScan

Introduction

Hello 80 Level readers, I'm so glad to be here. My name is Anderson Rohr, 31 years old. I'm an Unreal Engine artist, 3D Generalist, and Video Editor from Brazil, owner and founder at Third Move Studios. Everything started in 2008 when I was in high school and we had to create a video for English class. I learned Movie Maker and edited the video of our group. After that, I discovered a passion for video editing.

So I went to college and did a Bachelor's Degree in Audiovisual Communications Technologies/Technician at the University of Santa Cruz do Sul (Unisc) at the same time I was an intern at Unisc TV working as cameraman, producer, video editor, soundtrack composer, etc. A colleague introduced me to some tracking features in After Effects and it intrigued my curiosity at the same time I was studying 3D stuff from YouTube tutorials. I'm a Cinema 4D guy.

In 2013 I started working at a video production company. They wanted some 3D freelancing work and I remember I proposed to be a full-time employee. So I was the first employee of that company. They gave me total freedom to continue studying 3D stuff even in working time when we didn't have client work to do.

In April 2020, I was attending RTC 2020 (RealTime Conference) and I watched a panel discussion about Virtual Production. Ben Grossmann, Co-Founder at Magnopus, was one of the speakers. In the end, we exchanged some e-mails, talked about Unity and the Lion King movie, Unreal Engine, Jon Favreau, and the Mandalorian series. Ben encouraged me to be good at Unreal Engine as it was a shortage of people who are fluent in it. And he also told me that were lots of good people learning in Brazil, including at least two that worked with him on the Lion King movie. All those conversations were fuel for me. At that time I already had left my full-time employee job and was working only with my one-person company, and I finally started learning Unreal Engine thanks to Ben!

As usual, I started watching some tutorials and I also bought an Unreal course from Unhide School. Curiously, the teacher Marcos Ramone (a Brazilian guy) also worked on the Lion King movie. After almost three years working as an Unreal Engine Artist, I had learned a lot and worked with many talented people on amazing projects with some clients like NVIDIA, Rock In Rio (Brazil and Lisbon), Rabbit Films (aka Guilherme Coelho), Leartes Studios, etc. And I'm so grateful for all that.

The Chocolate Panettone Project

I started with photogrammetry with the Polycam app on my iPhone 11 in September 2022. I did some 3D scans, and it was piquing my curiosity more and more.

I already had a Sony a6400 24 MP camera, and I thought this amount of megapixels would be great for the textures compared to a smartphone, so I decided to learn RealityCapture, and I immediately fell in love with that software.

The easy workflow from RealityCapture to Unreal Engine 5 motivated me to continue exploring the photogrammetry world, as the Nanite feature in UE5 deals greatly with a huge amount of polygons. I want to say a big thanks to William Faucher for his amazing tutorials about photogrammetry. I've learned RealityCapture with him.

My first 3D scan with closed mesh (including the bottom) was featured on Capturing Reality social media. I was very happy, and it really motivated me to keep learning and creating more 3D scans.

When the RealityScan app was released in December 2022, it blew my mind. The app delivers great results, and the AR guide is a powerful feature that helps you a lot.

The RealityScan Workflow

I used my iPhone 11 with RealityScan app and I shot 200 pics. I wanted to test with a minimum number of equipment to see the power of RealityScan, so I only used the warm led light of my living room and daylight coming through the window.

RealityScan is really easy to use. You can shoot a maximum of 200 pics, so you crop the region of your object and then generate a preview mesh (in my case it shows 'update preview' because I already generated it). After that, you can send it directly to your Sketchfab account.

For this chocolate panettone, I used Blender to delete some unwanted triangles. So I downloaded my model from my Sketchfab account, adjusted it in Blender, reuploaded it to Sketchfab, and then I started playing around with post-processing filters inside the Sketchfab platform (like that depth of field filter showed in the video).

As I mentioned before, the AR guide is very useful. If you're connected to the internet when you're shooting, the pics will be placed in the position you're shooting. When you move, it's all there tracked in real-time.

Also with the cropping box, you can remove the unwanted data captured around your main object. It's really handy and a time saver. With the preview mode, you can quickly check if your model is ok before uploading it to Sketchfab. And then, when you export to Sketchfab, RealityScan usually creates a model around 300k-1M tris (those were my experiences so far, but it depends on the size of your object) and automatically uploads the raw data with 15 or 20M tris for example as another FBX file. It means, when you download your model, you get the raw data as well.

Working on the Details

For this chocolate panettone, I didn't need to work on the texture as I took care of it on shooting. But sometimes if I don't cover some side of an object, it may have some kind of weird or white texture. So for that, I use the clone stamp in Photoshop or the clone feature for painting the texture in Blender. It's a really fast workflow as you can clone some part of your texture and immediately save the original file with a few clicks, and then reupload to Sketchfab again:

The details were challenging. In photogrammetry, you can't have too much space between two pictures, so I was moving around the panettone and shooting some close-ups in order to get more details of the cracked chocolate and always keeping my eyes on the number of pictures taken, because as I mentioned before, you can shoot a maximum of 200 pics with RealityScan. In my opinion, it's good but sometimes you need more pics for details.

Rendering and Lighting

I started switching on the lights in my apartment and testing the lighting conditions with the daylight coming through the window. The daylight matched the warm led light and it almost didn't have shadows around the panettone when I was shooting.

I remember some guy asked me on LinkedIn if I baked the reflections into the texture, but no. RealityScan did a great work blending the texture. Photogrammetry doesn't recognize well a reflective object, but on this panettone, it was some kind of smooth specular that worked well with the chocolate.

For 3D-scanned foods, I think it's nice if you have some parts in focus and others not (for example the big slice of chocolate) to enhance the details. And Sketchfab post-processing filters have a nice look for that. You definitely should give it a try. But I usually create some scenes in Unreal Engine 5 with UE Marketplace assets (free or paid) to showcase my 3D scans.

Conclusion

Good lighting is the key. If you're outside, prefer overcast days because it makes very soft shadows on your objects. You can't have big exposure changes when you're shooting around the object, because it will blend all together with the final texture (too bright and too dark). Prefer manual settings if you're using a professional camera.

Also, I recommend the RealityCapture + Unreal Engine 5 workflow because as I mentioned before, the Nanite feature in UE5 deals greatly with a huge amount (million and million) of polygons. So you're capturing real objects and using all those detailed meshes and textures (up to 16k with RealityCapture software, and up to 8k with RealityScan app) without losing anything. And best of it: in real-time!
We don't have to wait hours, days, or even weeks to render out a scene anymore.

I'm currently on vacation, but my latest work was creating a 3D scanned photorealistic pack for selling on Unreal Engine Marketplace with some of my 3D scans created with both RealityCapture and RealityScan. I did a cinematic short film for that and I promise – it will make you think about virtual and reality.

Again, thanks for this opportunity and I hope to help the community with these answers. Please feel free to follow my work here.

Anderson Rohr, 3D Generalist and Unreal Engine Artist

Interview conducted by Theodore McKenzie