Building an Old Hotel Using Unreal Engine & Photogrammetry

Introduction

My name is André Castman and I’m an Environment Artist from Stockholm, Sweden. I started my journey in this industry in 2016 when I was accepted to a Bachelor's program in Computer Game Development at Stockholm University. Halfway through studying there, I got to try out 3D modeling for the first time, and I immediately knew that this is what I wanted to work with. I struggled a lot in the beginning, and about a year later I actually started delving deeper into it, following tutorials and creating some more ”advanced” models.

During my last year at Stockholm University, I decided to apply to Futuregames in Stockholm, which is a Higher Vocational Education for game developers. They had a program for 3D artists that appealed to me, so I started working on a sample that eventually got me accepted to Futuregames.

At Futuregames I dove even deeper into 3D graphics and started focusing on Environment Art. Earlier this year I got an internship position at Paradox Interactive where I got to try out working on different projects and tasks. It was a great internship period, and I got to learn from very talented people. Now I’m searching for my first job in the industry, and I’m really looking forward to starting working!

Inspiration

Since I first heard of photogrammetry about two years ago I have been completely hooked. I started watching YouTube tutorials and creating my own scans with an old iPhone. Eventually, I got myself a ColorChecker and was able to borrow my brother’s DSLR camera. Being able to color correct my scans made a big difference in the quality of my scans, but I still felt like they weren't good enough. It was a lot of trial and error.

I’d previously been trying out different cameras, objects, times, and locations for my scans, but I had never actually done a whole environment using only my own scans. I saw an ad for a Photogrammetry Course through Vertex School with Vlad Vanzariuc as a teacher, and I figured this would be the perfect opportunity to learn from someone experienced and finally start on my own scene.

References

I took some time trying to figure out what I wanted to do for my environment. I wanted to create a building that stands out, but at the same time fits the criteria for building modularly. I’d never done a bigger building modularly so I thought this would be perfect for combining those goals. After having walked around the city I live in, I found a building that seemed to fit my criteria and started gathering references.

The building I decided to go for is called Norrtälje Stadshotell, and it’s a hotel that was built at the end of the 19th century. Back in the days, the hotel was the watering hole for generations of Norrtälje residents and summer guests, with dancing and joy in the tavern, but it made room for apartments and offices in later years.

The hotel itself is a colorful brick building, with lots of details from that time. While Norrtälje is in the middle of a big expansion, the hotel is one of few remaining buildings that hold on to that small-town feeling Norrtälje is famous for. The looks of it, the dimensions, the materials, and the different pieces attracted me and made me start this project. The surrounding areas also contain elements for an interesting environment with trees, flowers, and other vegetation. This posed a perfect challenge for me, both difficulty-wise and learning-wise.

Photogrammetry Workflow

When doing photogrammetry I use the following software and tools:

• Photoshop (Camera Raw) for color correcting and removing shadows from my .raw files
• Reality Capture for processing my photos to high-quality scans
• Maya for creating low polys
• xNormal for baking 
• Substance 3D Designer for polishing the bakes and creating materials
• Substance 3D Painter if I need to create tiling materials or edit details

The camera setup I used for this project was a Canon EOS 800D, with a Sigma 18-35mm f/1.8 lens. I also used Calibrite ColorChecker Passport Photo 2.

I used the following camera settings:

• RAW image format
• Manual Shooting mode
• Aperture at around 8 almost every time. It can go higher, but pictures take longer to capture
• Shutter speed at 1/200 or more
• Manual White Balance and exposure
• Manual ISO at around 200-400 max. 

It's okay to sacrifice a little exposure as long as the sharpness is still there. If you go with too much brightness, you might lose details. Therefore, I tend to have my shutter speed at around 1/200 or 1/250. It makes the photos a little darker, but less blurry, and it's easy to correct them afterward.

Getting Started

So, to begin with, I started gathering lots of references. I took photos of the building itself, the surrounding areas, trees, the town square, etc. Then I made my first blockout of the building in Maya. To do this, I managed to get hold of the blueprints of the hotel and tried to follow them as closely as possible. I wanted to have as many of my walls as possible in 4x4m, without deviating too much from the actual blueprints. I knew I would have to change the dimensions a little bit to have the building modular, and I managed to find a way that wouldn’t change the shape of the building too much.

Photogrammetry

When I started this project, I knew right away that I would have to do a few tiling materials, like the bricks of the building and the ground beneath it. I also knew that I would need a Trim Sheet with tiling details for the building and roads, etc. For all these tiling materials, I used pretty much the exact same workflow. 

For tiling materials, I always try to cover an area of maybe 3x3 meters. Since I needed to have a ~2x2m square in my final texture, I tried to cover a little extra, so I had some room to edit details and get more variation. I took my photos with around 50% overlap over each other to cover every little part of the surface and objects. I used ColorChecker to calibrate my images before taking them into RealityCapture. While it can, of course, be done without one, I find it smooth to have it for changing the white balance and colors in my photos.

You can see the example pictures of before and after taking my images into Camera Raw below. This took me 1 minute and it shows how I change the colors and remove some of the shadows. It’s really nice to be able to use the Pipette tool to set the white balance in the picture.

Below, there are some examples of how I took my photos for the brick wall and the cobblestone surface.

When I had taken my calibrated photos into RealityCapture, I created a high-resolution 3D model that I used for baking. I also created a medium-resolution 3D model of around 200k Polygons that I put into Maya to create a low poly model. You can do this either through Retopology (if it’s an advanced model) or just a plane if it’s a surface. Then, I took my high poly model and baked it onto my low poly in xNormal. I like using xNormal because it handles models that are +100 million Polygons without problems.

From xNormal I baked out Base Color, Normal, Height, and AO, and then brought these Maps into Substance 3D Designer. In SD, I used AO cancellation to remove some of the shadows from the Albedo. I also created a Roughness Map and then packed the AO, Roughness, and Metallic into one map.

Below, you can see an example of how I polished my textures in Substance 3D Designer. I sometimes create my Normal Map in SD from my Height, but it depends on the quality of my Height Map.

You can see some tiling materials I created below. The two last pictures are Trims to complement the road and the building.

You can find the Trim Sheet I created for my scene below. In that Trim Sheet, I have curbstones for the road, the lower wall of the hotel, the decorative shelf borders that tie the windows, manhole covers, and the town square cobblestones together.

In order to create the tiling details for the Trim Sheet I used the following workflow:

1. First I created a scan that is large enough to be able to be tiled without too much duplication.
2. Then I created a low poly version of it in Maya (the same one I used for baking).
3. I used Duplicate Special, so I had 3 pieces of the low poly that I could put together into one long piece and match the vertices, so there were no holes.
4. I took this 3-piece into SP and used the Clone Tool to remove all Visible Seams.

In the pictures below you can see one piece of the curbstones that I put into 3 pieces with Duplicate Special to create a tiling material for my Trim Sheet. This is the lower wall of the building, before and after Seams have been removed in Substance 3D Painter. I used this workflow for all of my pieces in the Trim Sheet.

Besides the tiling materials, I also had to scan separate objects as the windows, the door, the benches, etc. The door and the door frame were tricky, so I decided to divide them into 3 parts: the frame, the door, and the awning. I designed the awning above the door myself in ZBrush and created a simple material for it in SD. I found it difficult to scan it because of the placement and differences in shadows (above and under it) and I felt I could probably do it just manually. This way I could get even the colors and the shadows all over the door area of the building, not having to do much cleanup. I’m pretty satisfied with how it turned out to look. In the pictures below, you can see the door frame in RealityCapture before calculation as well as in Substance 3D Designer after cleanup.

You can see an early test of the tiling materials, the windows, and the door frame in UE4, and one of the benches that I have all over my town square in front of the hotel.

Assets

My biggest challenge with the photogrammetry part of the project was probably the shadow elimination of my assets. While it wasn’t too difficult, it was definitely time-consuming. I think the door frame was the most tricky since I didn’t have a ring light for my camera. It was very dark under the awning, so I had to do a lot of experimenting and editing in Substance 3D Designer in order to get this right. Besides, the awning was covering a lot of that area either way, but I wanted to have good consistency in my workflow and assets.

Since the course I took at Vertex School I've focused a lot on building modularly, I figured I would try to build the hotel in modular pieces to the best possible extent using the modular pieces – the pieces that can be added together seamlessly in Unreal Engine with the help of locators in Maya.

As mentioned earlier, I managed to get hold of the blueprints of the hotel and tried to model as close to them as possible. Since many of the walls already were around 4x4 meters it was quite easy to build everything in a modular workflow. It was only the roofs and the upper part of the building that I decided to create as bigger pieces since I felt it was easier to do it this way.

I created the whole hotel and surrounding surfaces, areas, and roads in Maya. Almost all the materials were created through photogrammetry. For my modular pieces, I used 2 UV sets: one for the tiling material and one for the object mapped material (Trim Sheet). Then I added the materials to my modular pieces in Unreal Engine 4. You can see a picture from my first blockout below.

And this is a picture from when I started adding some more assets to my UE4 scene:

I did not create all the assets in my final scene. Examples of assets that I didn’t create are some of the vegetation and the big fountain in front of the building. I will talk about these a bit further down in my breakdown and explain why I decided to do it this way, but in a few words, it had mostly to do with time and availability. I also used some Megascans to create decals and some Vertex Paint on various places in the scene. To bring some story to my scene, I also borrowed a few free assets, like the bike, from Sketchfab.

Vegetation

In the beginning, I wanted to do all the vegetation myself, but later I decided that I want to showcase this later in a project dedicated to vegetation only and focus more on the other parts of my environment instead. So, I decided to go with SpeedTree, and use existing templates to create my trees.

For the trees, I took 3 different SpeedTree templates and tweaked the parameters. The first tree was an oak tree which, if I remember correctly, was one of the free assets of the month for SpeedTree users. This tree had a swing attached to it, which I didn’t need. It also had visible big roots that stuck into the ground. They wouldn’t be seen anyway so I removed them and brought down the number of smaller branches. I didn’t want the foliage to be as dense as in the template, so I tweaked the knockout amount of the leaves to thin out the crown. This is pretty much the same as what I did for all of my trees. I played around with the density of the leaves and branches and tried to find shapes that would fit my composition better. In the pictures below, you can see the oak tree before and after the tweaking.

I figured out that the balcony was needed to be fleshed out a bit more to not feel as empty, and I created an ivy in SpeedTree. To create the ivy, I followed a tutorial by Emiel Sleegers which I found suitable for what I wanted.

The grass was created in SpeedTree as well, and I used a straightforward setup. I used the Absolute Mode in SpeedTree and increased the number to scatter the blades of the grass around. Then I added some deformation and size changes to the individual blades. For the grass, I used an Atlas from Quixel Megascans. There was nothing special in the tree and the grass setup in UE4. I exported everything from SpeedTree with a Non-Wrapping Atlas and when I imported it to UE4, I chose to create materials automatically with SubSurface and Wind Data from SpeedTree. This is probably not how I would’ve done it otherwise, but as I said earlier the vegetation was not the focus of this project and I will do it differently for my next scene.

The Final Environment and Render

Once I had assembled most of my assets in UE4, I started adding camera angles and playing around with the composition. I asked for feedback from a lot of people in the industry, especially about the composition and lighting, since this is what I struggled with the most. 

For the lighting, I started with a basic setup with a sky shader. I didn’t use the Dynamic Sky feature in UE4 because I’m not too familiar with it. I played around with it a bit, but I never felt satisfied with the results. So, I decided to go fully dynamic with an HDRI Map instead because it brought a lot of realism and lighting information. I wanted to keep consistency through my scanned assets, and this way I felt like it was the right choice. I downloaded a Midday HDRI Map that would fit my environment from Poly Heaven. I tweaked the Lower Hemisphere Color and Distance Field Ambient Occlusion a bit (the distance and the tint).

While setting up my scene in UE4, I got the tip to set up my shaders so that they would support Contact Shadows. Contact Shadows is a screen-space effect that casts shadows onto a flat surface using the Height Map. The Material Function was shared by Andrew Svanberg Hamilton when he talked about his amazing Australia scene. I copied the Material Function and plugged it into my shader. This was great for adding extra details without being expensive.

For my post-production settings, I disabled auto exposure and played around a bit mostly with the saturation and contrast. I had a hard time getting the lighting right. I had to try a few different HDRIs before I got satisfied. For me, this was probably the biggest challenge of the whole project, and I have to be honest, I still feel like there is much to improve.

For my final renders, I used the Movie Render Queue for my video, and high-resolution screenshots at 200% screen percentage. I adjusted the temporal AA samples and Frame Weight to reduce ghosting, and I put some Sharpen in the Tone Mapper to get more crisp screenshots.

Challenges

There are a few challenges when working with photogrammetry, one of them being the hardware demands. You need a  powerful computer when working with bigger scans. During the time I created this project I was lucky enough to work on a really good computer which I got to borrow during my internship at Paradox Interactive since we were all working from home during the pandemic. This made a huge difference compared to when I had worked on my personal computer (a 5-year-old gaming laptop). The computing time was a lot faster, and I didn’t have as many crashes as I’d had on other computers when processing my scans in RealityCapture. 

As I mentioned earlier, I got to borrow my brother’s DSLR camera. Being able to take high-quality photos in raw format makes a big difference when processing them for usage in photogrammetry software. A DSLR camera together with a ColorChecker can be quite high-priced, and I’m glad I could avoid these costs. 

While working on my scene I was able to walk 15 minutes from my apartment if I needed to do more scans or take reference photos. This was convenient and one of the main reasons I chose this building for my project. Any other way, this can be one of the challenges when working with photogrammetry – the accessibility of things you want to scan. If I wanted to scan a baobab tree or lava rocks it would be slightly more complicated since neither of those is in close vicinity to me.

When you’re building assets for an environment, you’ll still need to find a way to create them regardless. This also applied to the upper part of the building, which wasn’t possible for me to scan, and, therefore, I had to make those materials myself in Substance 3D Designer.

When building this scene I struggled with a few more things. One of them was the fountain in the town square. I really wanted to have the real-life fountain in the scene but it turned out to be almost impossible. I went there one day in the spring and started scanning it. It was a very sunny day and I knew I would have to do a lot of color correction and shadow elimination. This was not a huge problem since I figured this would be great practice for me.

However, soon I realized I didn’t take enough pictures the first time and, therefore, I needed to go back and take some more another day. I decided to scrap this scan and redo it on a day when the weather would be more suitable for photogrammetry. When that day eventually came, I discovered, to my horror, that they had turned on the fountain for the summer. Water poured from it and the surface was soaked. This made it impossible to scan it since reflective, glossy surfaces were a huge obstacle when it comes to photogrammetry. Instead, I decided to go with an already existing fountain from an environment I found on Unreal Engine Marketplace.

André Castman, Environment Artist

Interview conducted by Theodore Nikitin