Creating a Mysterious Scene with Lifelike Foliage Using SpeedTree

Introduction

I am Louis Sullivan, an Environment Artist at Electric Square. I've been in the games industry for over four years, having most recently worked on Diablo IV: Vessel of Hatred. Since starting my journey as a 3D Artist, I have found breakdown articles hugely insightful, so I'm excited to be able to write one of my own and shed some light on how I made this piece.

Project Inspiration & Goals

This project was inspired by the A24 film The Green Knight. I went to see the film in the cinema and fell in love with the rich mythological symbolism, cinematography, and world design. I found the mysterious Green Chapel scene especially captivating and decided to create my own interpretation. 

My technical goals for the project were to explore foliage pipelines, advanced shader creation, and Lumen and Nanite in Unreal Engine 5.

The chapel in the film is small and oppressive, which fits the themes of the film well, but I wanted to create something more grand and mystical. I was playing a lot of FromSoftware games at the time and thought it would be cool to redesign the space to feel as if you're walking into a boss arena with the knight's throne pride of place.

Concepting & Blockout

To conceptualize this space, I worked with my good friend Nate Bedford. I think it's great to collaborate and share ideas with other people if you can, as you always get inspiration and new perspectives that you never would have thought of by yourself.

I started with research and reference gathering of old English churches. These buildings have a very recognisable structure, informing the overall composition of the scene. The vaulted ceilings, columns and arches found in churches are evocative of forests and trees, so I wanted to include a tree breaking through this repeating structure, resembling part of the chapel itself and blurring the lines between man-made and natural.

After research and a few initial thumbnails from Nate, I created a blockout of the scene to explore how these ideas worked in 3D. Big changes are quick and cheap at the blockout stage, and you can focus on nailing the composition of your main shots. I found it helpful to do a rough pass on lighting, which was made easily with Lumen. 

Nate then did a paintover, exploring the overall tone of the environment. This was really helpful in giving both direction and motivation. As you can see, the final environment turned out quite differently in mood compared to this concept, but that's not an issue. It's good to be flexible and iterate when creating a scene. 

Chapel Architecture

Kit Pieces

I started by creating modular meshes for the walls, arches and pillars. Look at your reference and identify how to split it into modular pieces, and which elements can fit into your trim sheet. Planning this will inform how you create these modular meshes. It also gives you a good idea of how much work you have to do.

Once happy with the primary read of the meshes in greybox, I created a tiling brick material in Substance 3D Designer. It was important to get the scale of the bricks in this texture correct early, as I then created a sculpted brick set which had to match the tileable texture.

This looks like quite a boring kit, but it was very useful throughout the environment. The larger stones are used for the broken floor and stairs leading up to the throne, whereas the smaller bricks match the tiling material so blend seamlessly into the damaged walls. Variants with different amounts of damage are useful as you have more to play with when using the kit in the engine.

Trim Sheet

I modeled the trim sheet using Sub-D before bringing it into ZBrush, providing a good base from which to start the sculpt. Try to keep your trim widths on the grid as this makes it easier to UV unwrap meshes later. 

Sculpting the bricks and trim sheet was straight forward. Trim Border Smooth and Trim Dynamic for the edge chipping gives nice bevels. Dam standard and a few Orb brushes were used for cracks and further surface breakup. I could have spent more time sculpting surface details, but decided to do the tertiary detailing whilst texturing in Substance 3D Painter.

For the ornate leaf pattern, I followed Rogelio Olguin's tutorial on Dekagon's Artstation Learning course on how to take a pattern from an image and use photoshop to turn it into a grayscale height map. I then used this in ZBrush to displace the mesh and get the repeating pattern.

I baked the sculpt onto a 2D plane and textured the trim sheet in Substance 3D Painter, making sure to match the values and colors to the brick wall material. Once happy with the base stone material for the trim, I used it as a smart material for the sculpted bricks and throne to give everything a unified appearance.

Nanite Deformation

The version of Unreal Engine 5 I was using did not support tessellation on Nanite meshes, so I used Unreal's modeling toolkit to displace the meshes using the brick wall height map. For the damaged walls, I manually placed the brick meshes in Unreal and merged everything together using the modeling toolkit. The combination of the displacement and the brick models gives the kit a nice sense of depth.

I used the trim sheet to texture more detailed architectural parts like the large window and columns. Nanite allowed me more polys to model according to the trim texture and add more detail to the silhouette. These meshes could have used a unique sculpt to be more bespoke, but using the trim sheet significantly sped up the workflow, and foliage would cover much of the surface anyway. It's satisfying to see how far you can get with trim sheets once you've put the work into making them.

Master Materials

I set up some master materials to give myself full control and reusability. The main master material has some parameters for texture tweaking, and the foliage master material supports subsurface scattering and wind animations. I created material functions for the moss, Runtime Virtual Texture (RVT) blending, and wetness shaders. Splitting materials using functions is a great way to keep your graphs tidy and allows for more reuse as your materials essentially become modular.

Moss Shader

The moss shader blends a tiling moss material on top of meshes, using the base material's Normal Map to add detail and breakup to the blending. Various parameters control the normal influence and moss level and opacity. It also has the option to use an additional tiling alpha texture to support vertical faces like walls.

I followed Lukas Koelz's YouTube tutorial for setting up the blending, which I found through Jonjo Hemmen's article. I added some color tinting based on the world position, giving the moss yellowish areas, as I noticed patches of decaying moss in my reference. I added a section to scale the UVs based on the object scale, so if I resized any objects, such as rocks, the moss texel density stayed consistent. Another good option is to use world space UVs, though this is more expensive.

Having this function in the master material meant I could give every asset various levels of moss build-up to suit their uses and give the scene a unified and natural look.

RVT & Landscape

I set up a landscape material that allowed me to blend between a few different landscape layers using textures from Quixel. As this is a relatively small scene, it didn't require much complexity; I kept it to a grass, dirt, and wet mud layer to blend the water into the environment.

I then followed Unreal Sensei's tutorial to create an RVT blending function, which blends static meshes into the landscape to avoid ugly-looking seams.

Water & Wetness

To create the water shader, I followed Stef Velzeboer's Artstation learning course made for Dekagon. This gave a great insight into how to achieve a believable water shader with ripples, caustics, and depth coloring. The water on the course was very clear, so I tweaked it to give the water a murkier look. 

To make the water appear more stagnant, I added surface algae to the pond. I found Tharlevfx's Water Materials pack on the Unreal Marketplace, which had exactly what I wanted. I looked through the pond material in the pack and was able to integrate parts of the graph into my own water shader to add the pond scum. I used the rapids material from the asset pack to create the stream running down the center of the chapel.

I used to be wary of using asset packs in my own portfolio scenes; for some reason, it felt like cheating. However, if you credit where you got the assets from and make sure to explicitly show what work is yours, then there is nothing wrong with it. Using these effects added character and dynamism to the scene; authoring these elements myself would have needlessly extended the project timeline.

I then created a wetness material function based on Stef's Artstation Learning course. This ensured that meshes near the water level looked wet. I did make a change to this material, as I noticed when looking at the reference that surfaces submerged in water don't look shiny.

To recreate this effect in Unreal, I added a band of low roughness above the water level, but anything underneath was darker and more saturated, with a slight increase in the roughness value to mimic the way water scatters light.

Foliage

Stunted Oak Tree

I was inspired to create a stunted oak tree after visiting Wistman's Wood on Dartmoor, one of Britain's last remaining ancient temperate rainforests. It's a Pygmy forest, meaning the typically tall oak trees are much shorter and instead grow outwards with characteristic gnarled and twisted branches. The ancient, creeping look of these trees was a perfect fit for the tone of my scene. There is a wealth of references to gather from the internet, but ultimately, nothing beats taking it firsthand.

It was challenging to replicate the level of detail in the reference. The tree has many material variations, including moss, peeled-off bark, lichen, and ferns. The tree also needed to fit inside the chapel and support the overall composition by arching over the central nave, becoming a surrogate part of the chapel architecture.

I created a blockout mesh to get the correct scale and shape. I then brought this into SpeedTree and used it as a guide to create the trunk structure. Due to the unique shape of the tree, I used a hand-drawn trunk to get the exact shape I wanted and SpeedTree procedural nodes to create the branches.

This took many iterations to get right, and I had some great feedback from my coworkers Dennis Welling and Mike Fong. My main mistake was missing out on the intermediate branches between the large branches and leaf cards. Looking at trees without leaves in winter is a good way to see the underlying structure, so it's good to check your model with leaves removed.

Whilst I am happy with the tree overall, the structure of the canopy is still the weakest element. The geo branches have a good flow, but the transition into branch clusters is not very believable, and it's missing the crucial intermediate branches in many areas. This is something I will consider for future tree creation.

To create the branch texture, I imported a Megascans leaf atlas into SpeedTree to bake into clusters. I included large, small, bare branches to build out the canopy and some individual leaves to add detail.

Using anchor points meant I could grow the canopy logically, starting with large bare branches, then attaching smaller filler branches, and finally using individual leaves to add fluffiness.

I imported the trunk geometry into ZBrush for the sculpt, focusing on the primary and secondary forms. I aimed to capture the natural twisting forms from the reference and add details like knots, peeled bark, and vines growing on the surface of the tree.

I exported the usual high poly mesh for baking, and a game ready mesh, which, thanks to Nanite, could be higher polycount to capture the geometry detail. I also exported a highly decimated mesh for unwrapping. Lower polycounts are much easier to work with, and in Maya, you can use the Transfer Attributes tool to copy the UVs onto the game model.

I created two UV sets. The first fits the UVs into 0-1 space and was used to bake the normals and AO and create the masks used for material break-up.

The second set had all the shells oriented facing up, with the correct unified texel density. This was used to map the tiling bark textures onto the tree in a realistic way. 

I baked the mesh in Substance 3D Painter using the 0-1 UV set. I exported the baked normals to get sculpt information from the high poly and a packed mask map with three channels:

• Baked Ambient Occlusion;
• Texture Mask Map – This mask is used to blend between two different tiling textures, in this case, a normal and peeled bark;
• Gradient Map.

The Gradient Map is a grayscale texture used to map a color gradient onto the mesh. Black values map to the start of the gradient, and white to the end. To author the gradient map I started with the baked curvature and position map, and added some grunge, noise, and hand-painted elements like the lichen spots.

Gradient mapping is used often in VFX, but it has some great utility in environment art. It's flexible, letting you change your color palette by swapping out a single gradient, and as it's a grayscale texture you can pack it with other masks to improve optimization. The gradient is inexpensive as it's essentially a 256x1 texture.

Unreal now supports creating Colour Curves inside the engine, meaning you can create and edit gradients and see the results in real-time. There's some great documentation on Curve Atlases from Epic if you would like to learn more.

Ivy

I blocked out my leaf shapes in Maya using Sub-D modeling, and then sculpted them in ZBrush, focusing on the venation, edge damage and silhouette breakup.

I baked the sculpts onto a flat plane and textured them, exaggerating the veins and edge discoloration to make the leaves pop. Then, I imported the textures into SpeedTree to create the cluster atlas. I find SpeedTree's node-based tools really powerful when creating these branch atlas textures.

For the meshes, I imported a template wall mesh into SpeedTree as a force. I used this force in the Spine section to make the vines grow along the model's surface. Using hand-drawn trunk nodes gave me more control over the shape of the vines.
 
I added large ivy clusters along the length of each vine and used anchor points for smaller clusters, individual leaves, and dead branches to provide depth and believability.

This method requires quite a lot of manual tweaking, but the ability to grow the vines along meshes enabled me to make custom ivy variants for pillars and arches, which would have been time-consuming to create manually.

Ferns

I took a different approach to creating the fern atlas, using Substance 3D Designer to procedurally create the alphas and masks. Dennis Welling has a great workshop on Beyond Extent that goes into depth with this method.

I based the ferns on the Spreading Wood Fern, which I found in Wistman's Wood. Ferns are structured in a fractal way, with shapes repeating from the smallest leaves up to the large fronds. I capitalized on this by creating a generator for the small pinnules, which ultimately allowed me to construct the full leaf.

Once I had the pinnule shape, I packed the opacity alpha with other information, like a vein and height mask, in an RGBA merge node before plugging it into an output. This allowed me to transfer this information throughout the entire graph.

I then used this generator with exposed parameters to create a variety of pinnules in another graph. I used the Designer's new spline tools to create the pinna structure.

The Scatter on Spline Color node allowed me to scatter the pinnules along these splines. I created a stem and blended this into the graph before packing the channels, ready to be laid out in the atlas.

I made some variants with my new pinna generator node and laid them out using transform 2D before splitting the RGBA masks. These masks were used to author the albedo, roughness and height information for the final export. Eric Wiley also did a great breakdown of building procedural foliage atlases for Substance 3D Designer using this method:

I created the frond atlas in SpeedTree, using these Substance 3D Designer textures. You could do this step in Designer if you'd prefer, though I find SpeedTree's tools great for creating branches, fronds, and clusters like this.

The fern meshes were made in SpeedTree using trunks with the visibility set to “Spine only” in the "Skin" section to lay out the structure of the fern. I used multiple trunk nodes to create different layers of the fern, from the larger dead fronds on the bottom up to the young, curling ones at the top.

I used frond nodes for the leaves which follow the length of the trunk node they're put on. I added a curl force for the curled up baby leaves, and finally some individual leaves with anchor points which break up the flat look of the frond cards.

With this template created, I could then very quickly make size and age variants of the ferns in Speedtree by tweaking some of the values and leaf placements.

Lighting

I wanted to explore using Lumen in Unreal 5 for this scene. The initial setup is fairly straightforward; a directional light and skylight with an HDRI map to provide more realistic fill lighting. Volumetric Fog was key for the ambiance of the scene, though I would advise not going over the top as it's easy for the scene to start looking washed out and fake. I wanted the chapel to feel mysterious and magical, so I pushed this whilst still trying to retain value, contrast, and realism.

I was finding it hard to achieve the exact look I wanted with this setup. Even with the indirect intensity turned up using Lumen's real-time GI, I couldn't get enough light into the scene, and it felt murky and flat. I decided to use a few key Rect lights, along with some smaller spot and point lights, to be able to tune the lighting more.

Post-Processing

I wanted to do as much of the post-processing in Unreal as possible, so that the final visual fidelity is visible in the engine, not just in the screenshots. Some colour grading, pushing saturation, contrast and gamma, helped the scene get away from the default Unreal Engine look. I also changed the temperature and tint settings to bring some warmer tones to the highlights, and some cool purples into the shadows.

Finally, I set up a custom LUT (Look-Up Table) by taking a screenshot of my scene and using adjustment layers in Photoshop for fine-tuning. There's a guide on the custom LUT process.

There are some valuable tests when setting up your lighting and post-processing in an environment scene. The first is to desaturate a screenshot in order to get an accurate read of your value range. It's a good rule of thumb to have a separation in values between the foreground, midground, and background. 

For exterior scenes, atmospheric perspective means that objects further away blend with the background more. In this case, it meant that things further away become lighter and more orange due to the low angle of the sun; the foreground is darker and more blue. The desaturated test is important for values, but don't forget color contrast, too!

Zoom out from your desaturated image and view it at a thumbnail size. Does it still read well? Can you tell me what you're looking at? Strong composition is the foundation for this, but your lighting, fog, and post-processing settings are just as important.

The second test is squinting at your image or applying a Gaussian blur in Photoshop. Again, you're looking to see how readable the image is. Generally, you want contrast and depth in the scene and a clear focal point that your eye is drawn to.

Cinematic

From the start of the project, I knew I wanted to do a cinematic. The music I used from the film O Nyghtegal) is beautiful and evocative, and there are many elements, including the running stream, wind foliage, and particle effects, which, when seen in motion, make the scene come to life.

I used the falling leaf and water VFX from the Realistic Starter VFX Pack Vol 2 by FX Cat UA and also created my own dust mote particle effect and water caustic light function.

For the sequencer, I followed some advice from Jonjo Hemmen's Tequila Sunset article and William Faucher's YouTube tutorials. Using individual sequencers for each shot and then assembling them in a master sequence makes editing and reordering your cinematic much faster and cleaner. I added some subtle camera shake with a basic blueprint to make the camera movement feel more natural.

I used a 2.39 cropped aspect ratio for a cinematic look, and did some keyframing of the aperture and focus to get some dynamic depth of field effects. I recommend keeping camera movements simple in flythroughs. Restricting the camera to simple pans and dollies makes it feel more realistic and less disorientating, as if it's being filmed on an actual camera.

Using the Movie Render Queue (MRQ) to render out the cinematic gave me a lot of control over the final quality of the renders. I disabled AA and instead used Temporal Sample Count set to 16 for crisper visuals. I also recommend using 'warm up frames' if you have particle effects to make sure they look correct in the render. 

I did my color grading in the engine, so I exported it as a .jpg image sequence instead of .exr and used DaVinci Resolve for the final edit. This involved syncing the shots to the music, adding a title card and transitions, and some final minor post-processing tweaks.

Conclusion

I'm very happy with the final result, though there were many challenges and lessons learned. I didn't sufficiently plan at the start and wasn't strict with time management throughout development, so the project dragged out much longer than originally intended. 

Due to the long development, I started reworking large parts of the scene, which I felt were not up to my current standards. However, this ended up becoming a vicious cycle, and as I pushed myself to wrap up the project, I had to be more ruthless about what to cut and when to call it done.

At the end of a project, it's natural to look back and be critical and focus on all the things you could do better now. That's a good thing! It means you've learned a lot and can apply it to your next endeavor, but you need to find peace that it's not perfect. This was a real challenge for me, and pressing that publish button was surprisingly difficult. There's a quote attributed to Da Vinci that I think sums this up well: "Art is never finished, only abandoned."

Thank you for reading. I hope you found this helpful! Feel free to check out more of my work on ArtStation or send me a message if you have any more questions.

I would like to thank everyone who helped me with feedback and motivation on this project and 80 Level for the opportunity to share this breakdown with you.

Louis Sullivan, Environment Artist