LogInPhotogrammetry Artist Olivier Mourey a.k.a 3Dystopia has shared the working process behind Reality VS Unreal Engine 5 project, explaining how the scene was recreated with RealityCapture and rendered in UE5.
In case you missed it
You may find these articles interesting
Introduction
Hi, my name is Olivier Mourey. I'm self-taught in photogrammetry and got the pleasure of being contacted by Gloria Levine to share my experience on 80 Level. I hope this breakdown will help you learn how to use photogrammetry for content creation, level design, or any experience you are willing to create.
My first contact with 3D was three years ago when I discovered 3D with Quixel Megascans. At the time, I was 25 and had really no idea how people made video games or virtual content. I was amazed by the revolution this 3D scans library could offer. It meant endless possibilities for creation, and it looked very easy.
I made my first environment using starter content in UE4 and Megascans materials. Even if I didn't fully understand how textures worked, I managed to create a replica of the famous azure swimming pool of Pripyat near Chornobyl.
A few months later, in 2020, I discovered photogrammetry. To my surprise, I found out that even a newcomer like me could use it, despite it being considered something professional. Back then, I didn’t have any knowledge of 3D modeling or photography, but my enthusiasm led me to quickly learn enough to create my first models. At the time, I wasn't doing it full-time and didn't delve too deeply into the technical aspects.
I didn't start with RealityCapture, trying different free solutions and trial software to compare which one would be the best because it was just a new hobby. My personal computer was more a gaming one than professional hardware so model creation was very slow.
Later, I heard about Reality Capture and its fast speed compared to competitors. I tried it with a Steam license and was able to scan bigger objects more quickly. I also liked the fact that its UI is beginner-friendly but also very complete when you decide to create more advanced photogrammetry.
A year later, I made the decision to go full-time with photogrammetry and improve my technical skills. Since then, I have been working on model creation every day and have made around 500 scans, more than 300 of which are uploaded to my Sketchfab account.
I have also worked on cultural heritage collections, both in my home country of France and in Nepal and Japan, either in person or through my teammates.
In 2022, I dedicated my time to improving my ability to scan everything I wanted to create photorealistic fake environments.
I made a collection of 80 medieval scans (unpublished) and an urban-industrial collection of around 200 assets, most of which I published on Sketchfab. It represented so many different types of objects and so much practice that it finally became easy for me to visualize how to proceed with any type of object (architecture, sculptures, modern stuff like pipes, etc.).
Doing photogrammetry on the go in challenging situations, such as during intermittent rain in London or with limited time at dawn, also forced me to learn how to create full models from partial or low-quality photosets. It may have been frustrating at first, but I was always happy to learn from these new experiences.
The Reality VS Unreal Engine 5 Project
For my Reality VS Unreal Engine 5 project, I was looking for a little alley with tall facades that would block the horizon. I didn’t want to have an empty background, I would have needed to put fake trees or houses. This was also easier to scan than large environments, even though the alley was very narrow and I could not really move back there.
When I walked into the area and found this particular street, I immediately knew it was the perfect location for my idea. The old walls made it also very unique and authentic, the houses being from the 19th century.
Capturing the Data
As I was planning to make this "real to unreal" effect, I captured footage using a gimbal-stabilized camera. This was convenient for two reasons: it stabilized the camera, reducing blur at lower speeds, and it provided a reference for micro-movements when creating a virtual camera in UE5. The gimbal also prepared viewers for a stable UE5 film, as opposed to a shaky, handheld-phone look.
Filming was also faster than taking pictures. I made 20 minutes of footage and I was done, ready to leave. I chose to extract 1,100 frames using the scene filter in VLC, and the 6K footage provided a good resolution of 25MP. I had the option to extract more pictures if necessary, but it was not needed.
From my experience, the key to outdoor photogrammetry is to have good control over the speed of your camera. It's best to shoot in overcast lighting, as this provides good textures. It means you are operating in low light conditions but you also need to have sharp pictures to get good details with no blur. ISO helps and you don’t need to be stuck at 100 value to get a good model, but you’ll eventually sacrifice quality if you go too high.
The only option left is speed, so it’s in your hands! You take your pics, and it looks good on the camera screen but when you check on the computer, there’s a blur and it’s too late. Using a tripod is a solution but what I really like is the monopod, now I never go for outdoor photogrammetry without it. It allows you to use very low speed, the value that would even do a too-bright picture in most cases, so instead you use higher speed and in case you shake, it’s still okay. That’s probably the best tool you can have for photogrammetry and no need to go for an expensive one.
The weather on this day was ideal, with cloudy but bright conditions. I used a middle-sized micro 4/3 sensor camera, which doesn't receive as much light as a full-frame camera. This meant I set the ISO to auto and it varied from 400 to 1000 during recording, which was acceptable.
I preferred this type of weather for another reason as well. I use Ultra Dynamic Sky to create realistic weather in UE5, and a pure white sky is easy to simulate and looks realistic.
Processing the Scans in RealityCapture
The cleaning can be a hard part, it can be a long process and heavy for hardware. I would first recommend giving as much attention as you can to get the best pictures you can get. Spending ten minutes instead of twenty can represent hours of additional work. Cleaning can be a nightmare but the worst is that you might simply lose important parts of your objects.
For example, if you want to scan an old abandoned building, you could get most of the surface in a few minutes, while missing important details such as locks, rusty chains, etc. By simplifying your model, these details will be erased or will be so incomplete that you’ll prefer to delete these precious details. Always aim for the best and allow the experience to help you go through mistakes, that’s exactly how I learned. A full cleaning tutorial would probably take hours of content but 90% could be avoided with better pictures.
Still, here are a few simple rules to be sure to get a correctly reconstructed model:
- Pay attention to your overlap level between pictures but don’t forget to take close-ups of the most important parts of your object. At the same time, two or three pictures with the full object on frames can save you from alignment issues and other bad surprises.
- Take pictures that go beyond your objects. I made this mistake a few times before taking this habit. If you scan a building part and take a little more surface in your shots, you’ll delete this part after having clean detailed building borders.
- Finally, don’t shoot the theoretical way. In most situations there’s simply no use in spending 3x more time, taking hundreds of pictures for an object that is mostly simple. First, you won’t need that to get your model successfully reconstructed, then, taking 300 pics instead of 100 will mean 3x more time post-processing your pics, and 5/6x longer time for computing your model.
I’ve done this facade with no more than 60 pictures, which was enough.
The theory is important to begin your learning but still – it’s up to you to find the ideal amount of pictures you need depending on your aim for detail and the amount of time you have.
The Texturing Workflow
The texture set of this street was huge, with more than 40 x 8K UDIM textures. I only used Diffuse and Normal, however, because creating a PBR material would be hard and unnecessary. I understood that by comparing the UE5 scene with my original record. The cloudy weather did not create a lot of reflections around.
The level of detail is explained by that, amount of textures that makes this scene too heavy for most cases. Almost all the assets I sell are not at 100 % of their original quality but around 60/80 %, so they can be used on low-end computers for games or other needs.
The models in Reality VS Unreal Engine 5 were made the way I’ve always heard to be the worst for photogrammetry. I extracted frames from footage and pictures are always considered the most accurate for good results. I extracted the frames as JPEGs, compressed format not as good as heavy PNG/TIFF that are themselves not as good as RAW pictures that contain lighting information and are better for post-processing in software like Lightroom or Darktable.
So, I delighted the JPEGs that contained little blur and I didn’t check for blurry pictures, I sent the bulk directly in RC. That’s far from the way I usually proceed, but once again, it’s proof you don’t always need to follow theory to get where you want. Outdoor photogrammetry is really different from studio, you have to deal with weather, time, passers-by, etc.
The close-up pictures are what matters most, always taking pictures at a closer distance than the one you’ll render the model from.
Rendering the Scene
It took me 7 days to finish the project. I’ve lost some time with the model being too heavy to be imported from a single file into UE5, I had to cut it in half and reproject texture on each. I also had to recreate the road in a different file as it was not well reconstructed the first time.
As I’m still learning UE5, I spent a few days to get as close as the references I had by using Post-Process Volume. I wanted to use path tracing for my render but it was worse than ray tracing, the Normal Maps had too much noise and it gave me inaccurate shadows and asperities. So, I switched back to ray tracing.
I mostly learned by practicing and creating hundreds of scans in the past two years. I would lie if I pretended to be able to do everything with photogrammetry, however, what helps me know how to proceed is remembering the sculpts I’ve done, the buildings, the industrial objects, and round objects like pipes. Each different object is a new challenge and a new way to learn how to clean and get a good-looking model.
It’s awesome to discover uncommon objects in some place you’ll probably only go to once in your lifetime, to go back home and create a virtual photorealistic copy of it. I recommend people try photogrammetry if they are looking to create more personal work. If you want to talk to me about your project, feel free to PM me. I’m open to partnerships, especially if you have outside-the-box thinking.
You don’t need a degree or professional gear to create a good 3D scan. Simply experiment and focus on finding techniques to speed up your process. If you follow tutorials available on 80 Level or YouTube and pay attention to details, you’ll improve!