Building a Procedural Shell Grotto with Substance 3D Designer & Unreal Engine

Introduction

Hello everyone! My name is Coraline Lyu, former Material Artist at id Software and Naughty Dog. I have always been interested in both the technical and artistic sides of environment production. I enjoy building systems when they give me more freedom to iterate, but I still want the final visual result to guide the technical decisions.

I also like using personal projects to study workflows that I may not have the opportunity to explore during production. Shell Grotto was created in my spare time over approximately two months. I wanted to experiment with procedural ornament creation in Substance 3D Designer and then test the materials inside a complete architectural environment.

Discovering the Shell Grotto

I came across photographs of the Shell Grotto in England online by chance. Both the shell patterns and the unusual structure of the interior immediately caught my attention. The surfaces are covered with shells arranged into flowers, borders, arches, circles, and abstract ornamental designs. From a distance, they read as large graphic compositions. Up close, every pattern is made from small natural objects with irregular silhouettes, colors, and levels of damage.

I found that change of scale very interesting. I wanted to see how a relatively limited set of shell types could be reused to produce many different patterns without making the result look obviously duplicated. The project was inspired by the real location, but I was not trying to reproduce it exactly. My goal was to keep the identity of the grotto while adjusting the architecture to better support the mood and progression of my scene.

Reinterpreting the Original Architecture

In the real building, visitors move through a narrow corridor and eventually arrive in a relatively small square room. The room contains beautiful material details, but after completing my first blockout, I felt that its scale did not give the scene the ending I wanted. Walking through a long, confined passage creates an expectation that something will be revealed at the end.

In my blockout, reaching another small room did not feel like a strong enough payoff. I gathered some additional architectural references and redesigned the final area as a much taller domed chamber. The transition from the narrow corridor into a large vertical space created a stronger reveal. It also provided more room for the shell patterns to continue across the walls and ceiling.

I still wanted to keep the enclosed passages, dense ornamentation, and mysterious atmosphere of the original location. The main change was the spatial rhythm. The scene starts tight and enclosed, then opens into the domed chamber. The opening at the top of the dome naturally draws the viewer's attention upward and lets the shell material become part of the architecture rather than simply a decorative surface.

Planning the Production

I created the architectural models in Maya, authored the shell materials in Substance 3D Designer, and assembled, lit, and rendered the scene in Unreal Engine. The wall materials were exported at 4K resolution. I used Nanite Tessellation and Displacement in Unreal Engine to preserve the depth of the shell forms and make the relief more convincing in the final presentation.

This was designed as a real-time presentation environment rather than a production-ready game level. Tessellation and heavy displacement would not always be appropriate for a shipped game environment, especially across surfaces of this size. For this personal project, I was more interested in studying the visual potential of the material and presenting it with a high level of fidelity.

I decided to present the material in three ways:

• An architectural walkthrough in Unreal Engine;
• Fixed views of the environment;
• Isolated renders of the individual wall designs.

The walkthrough shows how the surface behaves throughout the space. The fixed shots focus more on composition and atmosphere, while the isolated renders make the ornamental designs easier to examine.

Building the Procedural Shell Generators

Before constructing the walls, I needed a reusable library of individual shells. I separated them into two main systems: a shell generator and a snail-shell generator. They belong to the same visual family, but their construction is different. Snail shells are mainly defined by a spiral structure, while the other shells rely more on radial growth lines that expand outward from the center.

Keeping them in separate graphs made both systems easier to build and adjust. I used Splines to construct the base silhouettes and height structures. In some of my earlier Substance 3D Designer work, I built complex forms by combining many individual Shape nodes. Splines made this part of the process much faster and gave me more direct control over the silhouette.

I used them for the outer contour as well as the main height transitions and structural lines. If I wanted to adjust the proportions of a shell, I could modify the Spline instead of rebuilding a large collection of separate shapes.

I created the surface growth patterns with Splatter Circular Grayscale. Directional Warp and Slope Blur helped break up the regularity and introduce small natural deviations.

Procedural shapes can become too clean very quickly, so I focused the irregularity in places where it made sense: around the silhouette, along the radial growth lines, and in the transition between large and fine details. I did not want to cover the entire surface with unrelated noise.

For damage, I blended Grunge maps into the height information and used the resulting structure to influence the Ambient Occlusion. This created chips, erosion, and recessed areas that followed the shape of the shell rather than looking like a separate damage layer placed on top.

The Base Color was based on photographs of real shells. I increased the contrast slightly because subtle color differences became difficult to see once the shells were placed inside the Unreal Engine scene.

I exposed controls for: the main height and silhouette, the intensity and distribution of the surface detail, the smoothness or scatter of the surface patterns, the main Base Color, the edge color, and the amount and range of color variation.

Together, these controls could produce more than a dozen variations from the same underlying setup.

The Base Color controls became more important than I initially expected. When the shells were reduced to their final size on the wall, many of the smallest height details were no longer immediately visible. Color variation became one of the clearest ways to separate neighboring shells and keep the wall visually rich from a distance.

Constructing the Shell Wall

After completing the individual generators, I collected their outputs on the left side of the wall graph. This gave me a small library that I could call repeatedly while building different patterns. For larger ornamental designs, I drew the path I wanted with a Spline and used Scatter on Spline Grayscale to arrange shells along it. I varied their scale and color so the design would keep its overall shape without looking like a row of identical copies.

For circular arrangements, I used Splatter Circular. I adjusted the Pattern Amount according to the size of the circle and changed the input shells where necessary. The overall design could remain symmetrical, but the individual elements still needed some variation.

Once the main pattern was established, I extracted the negative space around it. I used Bevel to create a grayscale transition and then Gradient (Dynamic) to generate additional paths inside the remaining area. I passed those paths into Scatter on Spline and filled them with smaller shells. This made the secondary details follow the structure of the main composition instead of being scattered randomly across the wall.

I approached the wall in several scales. The largest ornamental shapes established the composition. Medium-sized shells formed borders. Smaller shells filled the remaining gaps, and fine damage and color variation helped connect everything. After arranging the shells, I added broader variation and damage before blending them into the lime and cement surface of the wall.

One detail I noticed in the references was the way real shells are embedded in mortar. They do not sit cleanly on top of the wall. When a shell is pressed into wet material, a small amount of mortar is pushed outward around its edge. I reproduced that buildup around the contact areas. Without it, the shells appeared to float above the wall or resemble decals. The additional mortar made them feel physically connected to the surrounding surface.

Creating Variation Without Losing Cohesion

I created eight wall variations for the final presentation. They share the same shell library, framing elements, color treatment, and damage logic, but the central compositions are different. Some designs use circular or symmetrical patterns. Others are based on vertical bands, flowers, arches, or more abstract arrangements. I designed the main composition of each wall manually.

I decided where the primary shapes should sit, how the visual weight should be distributed, and which areas needed to remain quieter. The shared shell library, mortar treatment, outer frame, and color logic helped the different designs feel as though they belonged to the same building.

Working Across Different Scales

The biggest challenge appeared after I completed the first individual shells. I spent a lot of time creating small surface details. When I viewed the shells closely in Substance 3D Designer, those details looked rich and convincing. After I reduced the shells to their final size and placed hundreds of them across the wall, however, most of that work disappeared.

At the same time, the wall lacked the larger height variation it needed to remain readable from a distance. I had to change the order in which I worked. Instead of completing all the fine details on an isolated shell, I started with the largest and most recognizable height changes. I placed the shell into the complete wall, reduced it to its intended scale, and checked whether its shape and relief were still visible.

Only after the large and medium-scale information worked in context did I return to the finer surface detail. The experience changed the way I judge material detail. An isolated preview is useful, but it can also be misleading if the final asset will be viewed at a much smaller scale. The environment became the main place where I evaluated whether the material was actually working.

Bringing the Materials into Unreal Engine

I brought the wall materials into Unreal Engine as 4K texture sets and used Nanite Tessellation and Displacement to create stronger physical depth. Displacement was particularly useful because the surface contains several clear layers: the base wall, surrounding mortar, embedded shells, raised borders, and smaller surface details. Color and Normal information alone did not communicate those relationships as clearly in the final presentation.

I still had to control the displacement strength carefully. If the shells projected too far from the wall, they stopped looking embedded and began to feel exaggerated. The mortar needed to overlap parts of the shells while still leaving enough relief to make their shapes readable. Testing the material throughout the full environment also revealed places where it became too visually dense.

A wall that looked balanced in an isolated preview could become overwhelming when repeated along a long corridor. I used doorways, darker transitions, lighting, and simpler supporting surfaces to break up the repetition and give the viewer areas of visual rest.

Lighting the Environment

I received valuable lighting feedback from Rui Huang. I wanted the lighting to feel relatively natural, without introducing a strong artificial warm-and-cool contrast or pushing the image into an overly dramatic look. The shell material already contained a large amount of visual information, so very stylized lighting would have competed with the patterns.

The opening at the top of the dome acts as the main natural light source in the central chamber. Smaller practical lights guide the viewer through the darker passages. Grazing light helps reveal the displacement and mortar buildup, while softer frontal light makes the larger patterns easier to recognize. I tried to find a balance between the two. If the lighting became too flat, the shell depth disappeared.

If the contrast became too strong, the viewer could only see isolated highlights instead of the complete pattern. Rui's feedback helped me refine that balance and move the lighting toward a more grounded and realistic result.

Presenting the Materials as Artworks

In addition to the environment walkthrough and fixed shots, I created isolated renders for each wall design. I did not want to present them on a sphere, cylinder, or flat plane normally used for material studies. Those shapes are useful for evaluating a texture, but they did not feel connected to this particular project. Instead, I designed a decorative frame made from shells and presented every wall variation as if it were a framed artwork.

Conclusion

Shell Grotto began as a study of an unusual real location and gradually became an exploration of procedural design, architecture, and scale. The procedural workflow allowed me to reuse the shell generators across many ornamental patterns, but the most useful lesson came from seeing the materials fail at their intended viewing distance.

I had initially spent too much time polishing details that were only visible when the shell was shown at a very large scale. Once it became part of the wall, the larger silhouette, height variation, color, and overall composition mattered much more. I now try to establish those large and medium forms first and test them in context as early as possible. Fine detail is still important, but it should come after the material is readable at its final scale.

For artists working with procedural materials, I recommend changing the viewing distance often and testing the material under its intended lighting. Looking at the final context makes it much easier to decide which details are helping the image and which ones are simply adding complexity to the graph.

You can find the complete Shell Grotto project and material breakdowns on ArtStation.

Coraline Lyu, Material and Environment Artist