How to Set Up Stylized Troglodyte Village Material in Substance 3D Designer

Introduction 

Hello everyone! My name is Cyril Armand Kessin, born in 1991 in Vitry-sur-Seine, France. I've been creating procedural PBR materials for over 7 years. I’m currently living in Madrid, Spain, and I've been working in the video game industry as a Material Artist for the past 3 years.

I’ve been a Material Instructor in 2020 along 2 years for bachelor's and master's for 3 different schools in Paris, France (EICAR, SAE, and LISSA) and since then, I’ve been working as Material Artist in the video game industry at Saber Interactive Spain (Turoc Origins), 31st Union (Project ETHOS) and Gaya Simulation Madrid (Microsoft Flight Simulator 2024).

I’ve done a “professional conversion” and went to a multimedia school called SAE Institute Paris in 2017. There, I had the option to discover all the formations they were offering, including Game Art Animation.

I didn’t know absolutely anything about 3D, but I knew I wanted to learn about 3D in general when I watched a video by Anastasia Opara about creating procedural buildings with Houdini, and it blew my mind, it was just incredible to see!

Learning Houdini was way too advanced for a beginner, but I kept one word in mind: procedural. I just loved that way to create.

In 2018, after my first 3D model was created, I was quite frustrated about the texturing because it was my favorite task, but I couldn't get any control over the textures to get the overall look I wanted (also, we were taught the old way with Photoshop, no Substance 3D Painter). 

Not long after, I discovered Substance 3D Designer from Allegorithmic (the actual Adobe Substance 3D Designer creator).

The procedural way of creating materials was just amazing and magical. Creating 3D elements from height/displacement maps (2D extrude) was also something that caught my attention, and I think this is what fascinated me the most. And then I knew I wanted to focus mainly on material creation. 

This software could do everything I loved and I wanted: creating materials procedurally, having control over parameters linked to the materials for any modifications or iterations. And again, I’d been told that it was way too advanced for someone who had discovered 3D after 2 months. 

But this time was different; I wanted to learn Substance 3D Designer, no matter what. Then I watched YouTube videos (any kind of videos related to Substance 3D Designer) and practiced as much as I could besides my studies, sometimes 20 hours per day, to absorb as much information as possible when I had days off at school. My results were not convincing, not at all! But it was still incredible to finally create my first textures! (Spoiler: you can see my first materials on my Instagram profile, but it's not recommended for your eyes.) 

Substance 3D Designer became my main software when I wanted to create an environment. I was thinking first about materials and how I could adapt my meshes to those, and not the reverse, how I could avoid creating extra geometry and instead use normal maps, for example. 

After creating many kinds of materials, I felt the need to push further experimentation in this software.

Stylized Troglodyte Village Material 

For many years, I've been thinking about creating this project and “answering” this question to myself: Could I use only Substance 3D Designer to create a static environment for a 2D/3D scroller game? Why not? 

I like experimenting with things inside this software, going beyond the boundaries, like with those weapons I made:

And also those weapons, a few years later, still entirely made in Substance 3D Designer:

Before, it was impossible to think about using tessellation heavily and everywhere inside a game because of optimization.

And I think because of those projects I’ve made (in part) and the huge help of Nanite tessellation inside Unreal Engine 5, I could finally make the test!

I’ve also been inspired by many artists! I’m not going to mention all of them because the list would be very long, but more or less all material artists who could have made, for example, world creations or materials for an entire building or house facade.

A few artists who inspired me are Enrico Tammekänd, Emrecan Cubukcu, Javier Perez, and Daniel Thiger.

For this project, I didn’t use any references because I didn’t want to be influenced as much as possible by any other stylized directions, but only by what was in my head. Same for the overall troglodyte village look — I only remembered that when I was young, I went to see one of those villages in France, and then I tried to recreate “my thoughts”.

My approach was to get simple shapes detailed enough to give the idea of the objects, with simple colors and a filter that gives darker and brighter tones for cavity and convexity values.

Material Properties

The idea was to create a general background (the cliff), add elements/objects, and apply a gradient to elements with the highest height values to avoid stretching effects as much as possible.

Then, to scatter all elements on splines to drive them easily by moving points that create the spline.

After, create a second material with the fences and foliages that still follow the same splines to fit any spline variations on the first material.

Rotate the object (in that case, a simple 3D plane primitive tessellated) with the material applied from a few degrees to get the wood planks that create the path in a horizontal aspect and not diagonal anymore. 

Inside Unreal Engine, import materials and add simple collision boxes along the wood planks' path to avoid the character falling into the empty. 

Masking Elements

This process can be a nightmare, but depending on your blending modes, it can also be quite easy to do. Add Linear and Max Lighten are the easiest blending modes to fit or extract your masks to your shapes. Afterward, it's only a matter of how you are layering your elements to get roughness and color values for those.

The Add Linear mode will keep all shape values you're trying to blend. Then, getting a separate node of those shapes in full white value will give you a mask fitting those shapes. For your other maps (e.g., base color, roughness, metallic), if an element inside your height map should have some parts below another one, then this element should be blended in the background input of the Blend node. The element on top should be plugged into the foreground input, and the mask from the top element should be plugged into the opacity input.

Max Lighten mode – here I’m not talking about the mode itself inside the Blend node, but the process it performs: mixing shapes by keeping their highest values. This blending process is the same inside the Height Blend node, and you also get a mask output after the blend, based on the shapes plugged into the foreground input.

And here is the ID map result from those masked elements:

Bench

The bench was probably the most interesting element I had to create because it is the only object in direct contact with both the wall and the floor (the wood plank path). It wasn’t hard to create, and it was very fun to see how the height extrusions reacted to it!

I created this object in a separate graph to avoid an unnecessary number of visible nodes. Once the bench was created, there was no need to keep all the nodes that built it visible.

As you can see, the graph is quite simple – it was just a matter of getting the correct height angles and height intensities for each element.

After the bench was created, I had to splatter it along the splines. Even if the splatter process can give random results, it still needs to remain under control.

But how could I avoid benches being splattered in front of doors and windows, or along the bridge where there aren't flat surfaces? Masking those areas was necessary. A first mask was generated from the bridge to isolate flat areas as much as possible, and a second one from the doors and windows to subtract those zones from the bridge, where these objects shouldn’t appear on the material.

But I still had one issue: some parts of the cliff had very high values, and since I was blending all the benches using Add Linear mode, some areas ended up looking extremely flat. A third mask was generated to capture the highest values from the cliff and subtract them from the bridge mask.

That way, all benches are now generated only on flat surfaces of the bridge, avoiding doors, windows, and high cliff height values.

Emissive Map

As I wanted to create an entire environment with Substance 3D Designer, I thought, "Why not include a part of the lighting inside the material?" This allowed me to create new elements like the cables and wall lights. The first lighting result wasn’t that great because, even with a high intensity value, the light from the emissive doesn’t bounce around surfaces like a normal light.

But there is a node called RT irradiance. Through a ray-traced process, this node allows you to calculate direct light influence from a few inputs: base color, metallic, height, emissive, and environment maps (if needed). Then we can use this direct light map output as an emissive map to help achieve better lighting results and more indirect light influence inside a graphics engine.

Challenges

One of the most difficult parts of this material was dealing with the stretching effect produced by the 2D height extrude and turning it to my advantage – in this case, extruding a bridge or wood path with a strong displacement value directly from the material.

Also, the graph optimization. As I was working in 4096x4096, the choice of a stylized direction was mainly for this reason — I couldn't add too many details or effects to each element's normal, roughness, or color information because the number of nodes made the graph quite expensive in terms of memory usage (64 GB of RAM, 55 GB dedicated, 53 GB used by the graph.)

Let me know if you'd like this further adapted for technical documentation or a portfolio description.

The main nodes I used are:

• Spline (Poly Quadratic): create a spline from 0 to 10 points to get the desired spline shape.
• Spline Append: merge all created splines.
• Scatter on Spline Greyscale: scatter all created elements along the splines.
• Flood Fill nodes: used for each element to generate color and roughness variations.
• Cube 3D: used to define angle directions for elements (e.g., the bridge, support planks, clay tiles).
• Level: set the extrude intensity of each element.

By simply moving the points that create the splines, you can easily move all elements, including the second material with the fences, to follow the wood path where the character should walk. This also makes it easy to create level design variations from the materials, since the wood path defines the platforms.

Unreal Engine 5

The scene inside Unreal Engine was quite simple. I first had to create the project in third person to get the character, and later on, I just had to move the camera to the side to get the scroller game view and enable nanite tessellation in the project settings.

The next step was to delete all 3D objects in the scene and import one 3D plane with a few divisions on it to help with the tessellation process, and then create shaders for both materials.

For that scene, I also had this question: "How simple can a shader be to enable the tessellation and displacement process on an object?" And it was even simpler than I thought. I had to import all maps into Unreal Engine – base color, normal, emissive, AORMH (ambient occlusion, roughness, metallic, height compiled in one map) – enable nanite tessellation in the main shader node, and set a value in the magnitude parameter. And nothing more for the material, except one parameter to set the emissive intensity.

The last step was to create simple 3D cubes for the collision path and fit each cube to the position of each wood plank's path, to prevent the character from falling into empty space.

And finally, pressing play to see the character moving!

Advice for Artists

It took me one week to finish this project. I could have spent a few more weeks on it, but the goal of the experiment was also to create something quick, functional, and satisfying enough to say “stop.”

There were many things I enjoyed, but if I needed to choose one, it was walking with the character to see how it finally reacts to the materials in real time.

“Faire ce qui n’est pas recommandé, permet parfois de recommander ce qui n’est pas à faire.” Sometimes, people around you won’t share the same enthusiasm you have when you talk about creating something, but stay focused and try. If you truly believe something could work, then give it a shot.

Practicing and experimenting, I think, is the key to everything. I remember seeing a video by Wes McDermott many years ago, where he mentioned discovering a way to create organic cracks just by randomly mixing shapes inside a Tile Sampler node.

Be curious, and don’t be afraid to start and fail your projects – you’re still going to learn something from them. Gather references so your brain doesn’t trick you. References are there to remind you how the model you’re following should actually look.

Feedback is important. Don’t be shy about asking for it, and don’t take it personally. If someone tells you that what you’ve done isn’t good and explains why with constructive advice, then that’s the best thing you could get to level up.

In my opinion, any tutorial can be useful. It’s only the way you approach them that truly matters. Most beginners make the same mistake: they watch the video and press pause every minute to copy and paste what they see into their graphs. And that’s completely understandable – Substance 3D Designer can be very overwhelming at the beginning.

Instead, let’s say you want to make a wood material. It’s better to gather your own references first, then watch a tutorial about wood pattern creation, and adapt the node patterns to fit your references. This also leads you to question what kind of result you want: should the wood pattern be more stretchy, wavy, grainy, glossy? All those kinds of questions will naturally come to mind and guide you toward creating a material that actually resembles your references.

Even tutorials that might be considered “bad” can still show you node combinations that end up being useful for your projects – things you might not have thought of otherwise.

Conclusion

I was finally able to answer a question I had been asking myself for many years: "Can we create an entire static environment for a 2D/3D scroller game?" The answer is yes – partially. But at what cost?

Even with UE5’s Nanite tessellation and smooth performance on 3D planes with millions of polygons, production will always favor optimization. While generating an entire static environment inside Substance 3D Designer is incredibly fun, the classic approach to environment creation will still be the preferred choice, at least for now.

Many thanks to 80 Level for giving me the opportunity to do this interview. And also, a huge thank you to all the material artists around the world who share their work and projects. You’re giving new 3D artists the chance to break through the frustration wall of learning Substance 3D Designer!

Cyril Kessin, Material Artist

Interview conducted by Amber Rutherford