Learn How to Make Ultra-Realistic 3D Scanned Environments with Unity

Introduction

Hi, I'm Paul Sandoval, an Architect and 3D Environment Artist born in Mexico, passionate about traditional drawing, nature, and gaming since I was a child. Some of my favorite games at that time were action-adventures, like Tomb Raider or Resident Evil. In high school, Unity showed up to me as a fun tool for a hobby, and I learned the basics of the engine on YouTube and the Unity Docs page.

In 2014, I decided to study architecture. I have done a few things with Unity before, nothing serious, but in the middle of my career, I decided to use Unity on my school projects instead of using traditional architectural rendering software.

I saw an opportunity in using Unity because I was capable of importing my architectural 3D models to bake realistic light, using first and third-person controllers without the need for programming or knowing a lot of technical details, and generating something interactive and incredibly useful to integrate into the architectural design.

Here is an example of what I started to create in Unity professionally:

Becoming an Environment Artist

In my education as an architect, I learned the basics of 3D modeling and texturing and got training on topics like composition, forms and shapes, light, color, conceptualization, architectural design, and others.

I decided to go even further with 3D modeling because I liked the idea of showing my designs in real-time rendering, and I needed 3D objects to use in Unity for my school projects.

As I said, I'm passionate about nature, I always saw the process of conceptualizing architecture like "How to merge the local vegetation with the architectural idea?". That was the thing, I went on creating my 3D vegetation assets because I needed local vegetation. At last, I ended up creating game assets to use in Unity.

Image of cactus garden design, the plants were made in Maya, and the young trees with SpeedTree:

When I was studying architecture, Uncharted 4 was one of my favorite games, I was in love with the environment art of the game. I watched many conferences of the artists talking about the creation process of 3D environments and learned a lot from there, even about technical details, that inspired and motivated me at the early stage as a 3D Environment Artist.

After that, I wanted to exploit my knowledge of making 3D natural environments, but in my city in Mexico, there are not many opportunities for that as an architect. Normally, an architect prefers to buy the 3D assets and save as much time as possible in the rendering process, but there was no time and budget on the architectural projects to create original and unique 3D assets, so I decided to submit my models to marketplaces. I received positive feedback from the community, and that's why I was able to continue making game assets.

Here are some of the first game-ready vegetation assets I made for Unity:

One of the strongest sides of Unity, from my perspective, is the High Definition Render Pipeline. I have been working with it since 2019, and my experience was a good one, HDRP can be very optimized, and there are a lot of settings to play with at the time of generating Scene Lighting. Shader Graph also expanded my artistic possibilities.

There are a few projects that I can mention being part of, one of them is Delivery Dancer's Ward, winner of the Golden Nica award from Prix Ars Electronica, and most recently, Ferocious. I hope I can talk more about it in the future, as of now, here is a video shared by IGN on YouTube:

Environment Creation Pipeline

As an example of my creation pipeline, I want to use my latest project, Mossy Cedrus Forest. It's my most ambitious artwork so far, and it feels very exciting to share my experience:

Here's a list of objects needed for creating natural environments based on my practice. That list separates the objects into different layers:

• Ground, debris, and objects (ground textures, rotten logs, branches, dry leaves, rocks, sticks)
• Ground cover (little plants with fine details like clover, moss, etc.)
• Short plants (ferns, grasses, and other weeds)
• Middle-size vegetation (young trees, bushes)
• Trees (the highest elements)

I decided to use photogrammetry on as many assets as possible, after that, I started with the scanning of the objects from the list. It was not a linear process, I was adding more objects in time and removing some others. Those layers from the list also help to organize the textures of objects, like merging different plants in one material.

The scanning stage takes a long time. For this project, I started scheduling walks to the forest, scanning the objects, processing them to turn into game assets, and repeating that several times over several months.

In the middle of that, many compositions were made with the assets in Unity, merging them to have a good idea of how those can be combined. These blockouts are also a good help in adding or removing objects on the list. I should mention that it is impossible to scan all the vegetation, so the scanned plants were the most prominent in the forest and some others, which I thought were the most relevant. Here is an example of some blockouts I made:

Modeling and Texturing Objects

I already did a quick explanation of how the 3D assets were tackled because, in this project, I started with the creation of the assets. Those need different approaches and even different scanning techniques, for example, the following object:

It was processed using Agisoft Metashape, and then the object with a very high density of triangles and a second decimated model with less geometry density were imported into Maya, some adjustments then were made on the low geometry density object (closing holes, remeshing, setting up UVs) to be ready for texture baking. The aim of textures is 4K resolution, so the textures were baked at 8K and then compressed to 4K in Substance 3D Designer. This step makes the textures have more quality instead of simply baking it in 4K.

After finishing processing all of the textures, the game asset was put into Maya, creating the correctly named and organized LODs. That way, Unity recognizes LODs and generates the levels' distances automatically.

The ground textures are also processed with Metashape and imported to Maya with all information baked into texture maps (Base Color, Normal, Height, and Ambient Occlusion).

To make it seamless, I use Substance 3D Designer, but if the texture is too complex with significant details like fallen leaves or scattered rocks, the process takes place in Substance 3D Painter.

Modeling and Texturing Vegetation

The foliage is tackled with a photometric stereo technique, and the textures are processed completely in Substance 3D Designer.

Smaller objects like sticks, fine details on the ground, and even some mushrooms were also scanned with this technique.

The graph for it is very basic, but the pieces of the plants are separated into 4 scan groups. My camera takes 4K images, so those 4 different scans are merged to produce a large 8K image to achieve the same quality as Maya exported textures, compressing the final images to 4K.

The modeling varies depending on the plant kind, the trees are way more complex than a fern for example, but basically, those follow the same rules, creating pieces and then forming the 3D models:

Normally, I send the asset to Unity after this to build blockouts. When I'm satisfied with the compositions made in Unity, I proceed to finish the 3D plant by going back to Maya. The final steps are adding vertex colors (those are used for wind animation and Vertex Ambient Occlusion) and creating different plant varieties, LODs, and billboards here in Maya.

Here are some compositions I made in Unity to display the 3D assets created in 7 months for the project:

Scene Blockout in Unity

First, I quickly made the scene map and compositions and added a trail so the player could see a path to follow.

The red points from the image above are the views I needed to focus on. I had to take care of all the details around the path but some elements like trees, large rocks, and other plants were placed and moved from the spots because I wanted to make those my main composition views. Here are some images of the blockout:

Some 3D assets were made right in the blockout process, like plant variations, fine details, etc. After that, some images were taken and I started all over again, almost from scratch, this time being more careful with the compositions, details, and density of objects.

Here is a speed-level design video:

The locations marked with red points on the top view image shown at the beginning of this topic are the player's limited area made with elements like the rock wall.

It adds a more interesting look to the environment but also allows me to block the player's view and alternatives of moving, so the player has to walk on the path I designed, and there I have a nice chance to add a composition.

Lighting and Final Setup

There is nothing extraordinary about the post-processing and render settings, HDRI Sky, Indirect Lighting Controller, Screen Space Ambient Occlusion, Volumetric Fog, Film Grain, and a few color adjustments.

Shadow Filtering is one setting that was important for me at the time of generating the lighting in Unity's HDRP. Setting this on High Quality does not have a big impact on the performance but adds more parameters to the lights, for example, different blur amounts depending on the object distance from the surface where the shadow is projected. For example, the tree shadow gets more blurry than a plant shadow because of the height difference. Look in the following image how the shadows coming from the trees are more blurry than the fern shadow:

I used this amazing realistic effect in the forest lighting to get that cloudy atmosphere. Also, baked lighting was not an option this time, my GPU memory was not enough for this scene, so I decided to take advantage of the High Quality Shadow Filtering to fake indirect lighting, and to optimize it a little more, I used light layers. Here is a video on how the light changed after the post-processing and light adjustments:

I also used this fake indirect light to highlight fine details. A good example is the moss on the objects:

Conclusion

To new artists, I'd say it's important to be patient in creating environments like this. Sometimes, you won't feel your design the same way from one day to another. Achieving your vision can take more time than you expected, and it could feel like endless work because after finishing a project, you will keep finding more opportunities for improvement, at least that happens to me, and not only in Unity but in architecture design too.

If you are interested in learning more about my environments, I recommend visiting my ArtStation profile. I also post updates about my game assets on Instagram. If I have something to show, I post videos on YouTube, and, normally, I'm on the Discord server, where I discuss my game assets. I'll answer even if you just want to say "hi."

Thanks for your time, and I hope you find this information useful!

Paul Sandoval, Architect/Environment Artist

Interview conducted by Theodore McKenzie