Making a Photorealistic Spanish House with RealityCapture & UE5

Introduction

Hello everyone! I’m Sergi, a Lighting Artist originally from Barcelona, but I’ve been living most of my life in Granada, a city 'in the southern area of Spain. I’ve always been interested in art. I have loved video games since I was a little kid. My father was quite tech-savvy, so he let me play around with some image editing software. When I was 10, I remember googling "how to make 3D films". At 14, I knew it wasn't just a hobby and that I really wanted to work professionally in the game industry. Although my home city did not offer many art or game development courses, it didn’t stop me from trying.

After I finished secondary school at 16, I started learning 3D with the help of the internet. I began to build a portfolio in ArtStation focused on environment art. Thankfully, my family was very supportive. I always loved to light scenes, and although these environments weren’t very good on a technical level at the beginning, I think they were presented quite decently. After some time, when I was 17, I thought I had a good enough level to apply to companies. Soon I finally got a job as an Environment and Prop Artist at a Valencian outsource studio called Elite3D.

There I worked on games like Call of Duty, Blood and Truth, Cyberpunk 2077 and Saints Row, among a few others. I stayed there for about two years, and at that moment I realized that what I loved was lighting art. So I started to work on my lighting portfolio. I also got into photography, which I’ve loved doing since then. Luckily, a friend of mine told me about a lighting art position open at Crytek. I applied and got the job. Since then, I’ve worked on the lighting for Climb 2 and Hunt: Showdown.

Abandoned House in Granada

I feel very passionate about creating my personal art works. I think it’s a ton of fun, but admittedly it's exhausting to find the strength to do it after work. Yet, from time to time, I still do some stuff and love it.

For this specific project, I was eager to do something and to explore the use of photogrammetry since it would make my life easier in some aspects. I also wanted to explore new workflows that Unreal Engine 5 offers for environment art. So, I started doing some tests with the new Epic’s RealityScan iPhone app. I was so surprised with the results. I started asking friends and family if they knew any abandoned places around my city. We finally pinned down a location, and I went with a friend to an abandoned house east of the city. It wasn't a cloudy day, but it was late and a bit hazy, so the lighting was very soft, making it nice for photogrammetry.

Since we didn’t have much time before the sunset, we started capturing parts of the crumbling old house. The idea was to scan different areas and then make a building kit out of them and to reconstruct it inside of Unreal Engine later on. For the scans, we used a DSLR capturing surfaces to convert them into tileable materials. I also got the window and the corner of the house to use it as a corner mesh. We didn’t take a lot of pictures for each part; it was something between 30-60 photos.

Then with the phone and the RealityScan app, I scanned the door and a bunch of other meshes. This was pretty much my first time doing this, so I guess this shows how easy it can be to get into photogrammetry and get great results. Of course, I previously watched basic YouTube tutorials and asked more experienced friends for some tips, to make it less of a trial-and-error process.

Processing the Captured Data

While this probably sounds like the most scary part of the process, it was surprisingly straightforward. RealityCapture was used for the DSLR data and did very basic color correction to flatten the image contrast a bit more. It was very easy to learn RealityCapture with the official documentation and tutorials they have on their YouTube page. And the Pay-Per-Input service was great for just playing around with the software and paying whenever you are actually happy with the result.

As for the iPhone data, the RealityScan app already processes the photogrammetry for you, giving you a 1 million polygon mesh with a diffuse and a Normal Map ready to use. This is great because it saves so much time, but the biggest downside is that you do not have access to the raw data. Luckily, in our case, the lighting conditions were quite favorable.

To clean the meshes, I decided to use Blender. This is not the best of ideas since ZBrush will work with huge polygon counts much better than Blender, but in RealityScan I had previously reduced the meshes to something Blender was comfortable with. I just removed some areas that were not scanned correctly or remapped the UVs of some texture-less areas to others. Since they were quite organic, the cleaning process was quite easy. I also "dissected" parts of scans to use as details in the walls of the building, like bricks popping out, and other details. To create the tileable textures, I baked down a 2x2 meters plane of the scanned walls and then processed the baked Albedo, Normal Map, Displacement and AO in Substance 3D Sampler so I could make the textures tileable.

Texturing 

I knew I wanted to use Nanite, and when I was working on this project, there was no mesh displacement option like there is now in Unreal Engine 5.2. My approach was to use Nanite for details and modules and create a very basic modular set, and then displace it. I created a very basic blockout shape of the house wall, made its UVs, applied one of the tileables, and in Unreal Engine I started adding modular patches on top of it to smooth the corners.

It was very important for me to have these patches and scanned mesh corners to blend smoothly between them in Unreal Engine. I created a very simple dithered blending shading which pushes the pixel information further back on a dithered pattern, creating the illusion that the meshes are smoothly blended. 

The texturing of the roof was very straightforward, too. Modeling and making the UVs were very simple, but they took a lot of time. I had to place every tile to give it more charm. The texturing is unique for all the elements of the roof. This might not be the best method to use in production, but it allowed me to give the roof a more unique and charming style. I used Megascans assets to place a tree near the house. I created a quick leaf generator with geometry nodes in Blender, got the resulting mesh, and made a normal transfer so it would look more fluffy and correct.

Lighting

I'm a big fan of smooth gradients in my work. I like things to flow, from the lighting to the composition. Since the start of the project, I knew I wanted a late afternoon mood for the shot. The sunset shot uses as a base Sky Creator plugin from Dmitry Karpukhin. The main reason for me to use it is because of the awesome volumetric cloud features it has. I changed many settings like atmosphere height, the scattering, and so on to get the colors I liked. In lighting, you move sliders until you are happy with the result. 

The sun hits the front of the house and there are support area and spotlights to make the gradient smoother and more impressive to look at. 

The Alternate Lighting shot uses a more simple setup, it’s UE5 default atmosphere, directional light and skylight. I wanted to have the photogrammetry details on the wall pop a lot in this shot. The angle of the sun comes from a very specific angle that acts almost like a camera flash by giving many subtle details on everything. I really love global illumination so I boosted the Indirect Lighting value of the sun to get more gradients and a warmer look, but it’s important to still keep the blue shadows so there’s some contrast.

For the foggy version, it’s a very simple setup: a blue skylight with a lot of volumetric fog, to get that fog scattering blurry effect. In the post-processing volume settings, you can boost the convolution bloom scattering dispersion and inside Local Exposure. You can lower Detail Straight and Blurred Luminance Blend. It will give you that beautiful blurry scattering look.

Final Thoughts

Making a personal portfolio piece is always a lot of fun. Trying new workflows and approaches makes it a lot of fun and a cool challenge. From my point of view, the key to making an environment stand out is to have beautiful composition and lighting. By focusing on these aspects, it is easy to make an environment leave a lasting impression.

Something worth pointing out is that photogrammetry is more accessible than ever before, and it's actually quite easy to learn the basics. There are many resources online for free. It was a pleasant surprise to find that the official channels were extremely useful. Thank you for joining me on this journey as I explored lighting art and photogrammetry, and I hope your artistic pursuits are filled with inspiration and success. 

Sergi Marín Cantó, Lighting Artist

Interview conducted by Theodore McKenzie