Working with Photogrammetry in UE4

Oskar Edlund talked about his most recent Nordic Forest scene. It is an amazing UE4 project developed with the help of photogrammetry. The artist gave some useful tips and tricks on modeling the landscape, building assets, working on foliage and taking pictures for 3d scans. 

Introduction

Hi! My name is Oskar Edlund and I’m a 3D artist from Sweden, currently studying 3D art at FutureGames in Stockholm. I have previously worked as an intern at Arrowhead Game Studios. They are a small game company based in Stockholm and have worked on games such as Helldivers and Gauntlet.

Nordic Forest Scene

I have always been interested in new techniques and I knew early on that I wanted to try out photogrammetry since it’s so accessible, all you really need is a decent camera and a place to shoot. I wanted to make an environment to try it out and luckily for me I grew up in the middle of the forest in northern Sweden so I had plenty of interesting places to shoot at.

Production

I first put together a mood board / reference sheet on what I wanted to create with the kind of environment I had access to.

After I had established a mood, I needed to scan things from the forest that I could turn into modular assets. With that in mind, I went outside and looked for real world objects that I thought would work together. I also tried to think outside the box and be creative with the surroundings. For example, I wanted to make a stone wall but didn’t find a real one, so I took a photo of some rocks that I later turned into a stone wall.

For the modular assets, I scanned things like mossy stones, tree trunks and debris on the ground. For each object I took around 30 to 100 pictures from as many angles as possible. It all depends of the complexity of the object. Then I used Autodesk Remake to make meshes with textures out of the pictures I had taken. It’s a very easy program to start with and I recommend it to those who want to try out photogrammetry.

Here’s an example showing the process I used to convert the scan into a low poly mesh in just a couple of minutes using Zbrush.

To the left is a scan with textures and next to it the scanned high poly mesh inside of Zbrush. I used the different polish features in Zbrush to reduce the noise of the mesh and simplify the low poly process. This method was especially useful with objects that had a very much small to mid-size details that I only wanted to bake into the texture.

After I polished the high poly I used Zremesher to get a low poly base. I turned the target polygon count to 0.5 and turned down adaptive size a bit to get a uniform topology since I wanted to use a tessellation shader on the final result. Then I divided it and projected it back to the polished high poly version to get a more accurate shape and turned down the subdivision steps back to 1.

Keep in mind that this is not necessarily the best way to do it. This was just my way to quickly get a low poly mesh into Maya to finalize and optimize the mesh since I had a lot of meshes to do in a very short period of time.

I also removed most of the directed sunlight, shadows and ambient occlusion from the scanned texture, since it’s nearly impossible to get clean results directly from the photos.

To get rid of the AO I inverted the green channel from the baked normal map and screened that on top of the albedo with a low opacity. Then I tweaked the settings in Image -> Adjustments -> Shadows/Highlights until the texture looked good enough.

This is an example of the material setup in Unreal Engine 4 for all the props I used. This way it was easy to just swap the unique textures between different assets. I would also recommend using tiling detail textures to get a crisper look when the camera is close to the object. I didn’t add those details in my materials but it’s definitely something I would do if I was to go back to the project.

I used a lot of world projected materials in unreal engine that was masked with vertex colors to make the assets blend into the surrounding environment, such as moss on the part of the rocks that connected to the ground. This way I got rid of most seams in the world.

As I mentioned earlier, I wanted to make a tileable stone wall from a real world non-tiling object. This was a tricky process that required some experimentation.

I began with using the same process as described above. Then I separated the baked non tiling low poly into different segments that made sense. In this case my scanned object was made up of different stones so I split up all the stones into separate objects. Then I reassembled the stones with the help of the grid inside Maya. This way I got a tileable stone wall.

To get rid of the texture seams between the reassembled stone pieces I used Mudbox to clone stamp the texture. I also used vertex paint to blend in a world projected moss material.

This method was used to make the stone wall on the side of the road.

For the world materials, I baked a diffuse, normal and height map from different scans. I then tiled the textures in Photoshop. After I tiled the textures, I made a roughness from the albedo. This way I had full control on what details I wanted in the textures.

Working with Photogrammetry

I choose to work with photogrammetry since I wanted to accomplish a highly detailed forest environment in a short amount of time. It was my first attempt using photogrammetry and I wanted to see how far I could go with it. I think photogrammetry is becoming more and more popular today and a lot of studios are already using it in games. A great example is DICE with Star Wars Battlefront.

In the end I learned that working with photogrammetry is a great method to achieve realistic graphics in a fast way, but not in any way the magic bullet that many think it is. It comes down to combining new techniques with the old-school ways to do things. However, I still have a lot to learn about this process.

I used a Nikkon D1200 with a 50mm lens to capture the pictures, but any decent camera will do just fine. The best advice I have here is to take the pictures during a cloudy day when everything is lit by ambient light. That way there is less clean up to do on the textures. To get the photos into a 3D mesh I used Autodesk Remake, mostly because it’s an easy program and Autodesk have student licenses so I could use the program for free. Also as mentioned earlier, don’t scan everything that looks cool. Break down what you have access to and think about how it will be used in the game engine.

Foliage

For the foliage I didn’t use photogrammetry at all. Here I only took pictures of branches that I had placed on a blue background to simply mask out the leafs and pine needles to create alpha and subsurface textures. I created the normal maps from the diffuse using Quixel NDO. I made all the foliage in Maya using planes, except the leaf trees which was made in Speed Tree.

What I did here was to create a sphere around the plant and used that sphere to transfer its vertex normals to the plant using Maya’s transfer attribute function. That way I got smoother shading.

I modeled the planes to fit the alpha texture as close as I could without going totally crazy with the poly count budget, that way they looked good without the alpha applied, I then used the unreal shader to remove the alpha texture based on the camera distance. This helped the lod’s of the plants look thick in distance without having as many planes as the lod 0. I don’t think this is a good way to do foliage at all, it was my first time trying and it worked out for me but I would recommend watching some tutorials on how to properly do it.

The foliage shader itself isn’t anything fancy at all, it’s just a standard shader put to double-sided with an albedo, subsurface, normal, alpha mask and an alpha-fade set up, but in the big picture it looks good enough for the result that I was aiming for.

Post-processing

The lighting in my scene is very simple, I used a directional light for the sun combined with a sky light inside of Unreal Engine 4. I then added spotlights where I wanted more highlights: in this case one on the road in front of the camera and another at the end of the road where I wanted to have a strong focal point.

The composition here is very basic, but it helps to guide the eye.

I used Unreal Engine 4’s post process tools to achieve the result that I was after. It’s a really strong tool that helped me a lot. I used a strong vignette and also played around with bloom tints to get a warmer light and made the shadows slightly darker and cooler.

Here’s what the scene looks like with and without post process effects. It really makes a difference.

I found Unreal Engine 4 to be a very suitable engine for these kinds of scenes and I am looking forward to do similar stuff in the future.

Oskar Edlund, Game Art Student at Futuregames

Interview conducted by Artyom Sergeev

Follow 80.lv on Facebook, Twitter and Instagram