Stylized Environment Production with UE4 & Toolbag

Tim Boulton did a huge breakdown of his stylized scene Forge Environment made with UE4, Toolbag, ZBrush, and 3DCoat.

Introduction

My Name is Tim Boulton, born in Newcastle Upon Tyne and currently studying for a Master’s Degree in Games Design at Sheffield Hallam University. My instructor for this portfolio piece was David Wilson (you can also check his YouTube channel here). I created similar dioramas before in my undergraduate degree, which were rendered in Unreal Engine.

For this project, I wanted to incorporate PBR into my workflow when creating the stylised diorama. I treated this piece as a mini-environment to try out new techniques and improve my workflow when tackling new environment aspects. I have worked on some interesting projects, such as creating an off-realism diorama for my final project in the 3^rd year of my bachelor’s degree.

We’ve mostly being taught realistic PBR workflows, so it was nice to experiment with stylised rendering and rendering an environment in a different engine instead of using Unreal. 

Start of the Project

The environment had to fit in a 1920 x 1080 frame and had to be rendered in Marmoset Toolbag. We were given an option to create a diorama from an existing concept or to create our own. The other option we had was to go down a hand painted or PBR route, I went for the PBR route and created my own concepts for it. 

I chose to design a medieval armory and created mood boards of existing medieval armories to get an idea of what I want my environment’s design to look like. This greatly influenced my environment’s layout, assets design, and the overall atmosphere. It was also helpful to use existing concepts and different types of references (steampunk labs, dungeons) when designing the environment’s layout and choosing a color palette for the lighting of the environment.

I decided to make the highest texture resolution 2k except if it was a tiling texture. Smaller assets and props would range from 512×512 to 1k. This way the main assets stood out more and to make the environment optimized so that it can be used in a game ready engine.

Blockout

I experimented with white boxing for creating the environments composition and layout quickly, this helped me blockout the environment quickly. White boxing also allowed me to re-shape the environment’s silhouette quickly when altering the layout.  White boxing was helpful in terms of getting the environment’s scale to look right.

Boxes representing the scale of assets, and the blue UE4 mannequin was used for scale reference.

Initially, I created quick design blockouts of the environment to see which one I liked, I had to keep in mind the player’s movement and spacing, reworking the blockouts to make them smaller and creating more interesting focal points. Changing the structures and experimenting prop placement was an on-going process.

Thinking about my environment’s narrative made me add a mezzanine flooring to the design. This was to break up the uniformness and create more of an interesting layout.

Experimenting with prop placement was important as I wanted to establish where the main assets will be shown, and what type of assets would draw the viewers eyes towards the focal points. 

Primary Lighting & Colors

I did a series of simple light passes to experiment with the environment’s atmosphere and to quickly establish the main focal points. All the lights were pre-generated in Marmoset. In my opinion, the lighting process was easier to set up than in Unreal Engine, but there were limitations with reflections and light bounce control.

I tested some light value tests on some of my blockout designs to find the lighting I wanted to go for. I experimented with different color palettes to see what would work best.

In the 2^nd attempt of creating the lighting scheme, I felt the lighting still looked too flat, so I checked the environments values, this was easily done by turning the saturation to 0.  This was an efficient solution.

Using greyscale values was important when setting up the environment’s lighting, so I could know the contrast deafferentation between the light sources, and what is most visible. Doing this throughout the project was useful when adding more complex assets in or changing the layout as the project went on.

Once I had a rough value setup for the environment’s lighting, I started to tweak the colors. I went for an orange and mainly blue color palette due to them complementing one another on the color wheel. The orange light was the direct light source, it represents the cauldron’s and fireplace’s warmth as well as the light coming from torches. The outside ambient blue light represents the moonlight.

The lighting, in my opinion, was too flat, the focal point wasn’t clear which made the overall scene look washed out. 

Messing around with the post effects settings was tricky for the type of light setup I was going for. This was trial and error throughout the project as the lighting changed due to the environment’s layout getting constantly reworked (experimenting with prop placement and due to the assets visibility etc.).

Enabling Gi settings and changing the occlusion resolution gave stronger shadows and created more dynamic light bounces.

With using post effects such as adjusting the curves contrast and adjusting the contrast tone mapping gave the environments lighting a more dynamic quality. Adjusting sliders gave me control on how strong I wanted the contrast of my lights to be, adapting it until I found something which worked well.

With using post effects such as adjusting the curves contrast and enabling GI at high quality,  it gave the overall scene a stronger focal point.

Using greyscale values was important when setting up the environment’s lighting, so I know the contrast differentiation between the light sources, and what is most visible. Doing this throughout the project was useful when adding more complex assets in or changing the layout as the project went on.

Asset Modeling

If I was creating organic assets such as the wooden planks and rocks, I would block out the silhouette shape of the asset and put control loops into the meshes when I import them into ZBrush.  This way I can smooth out the model without any problems occurring. I would sub-divide it to a high division to give the asset a lot of polys to work with, then dynamesh the asset to make the polycount even across all areas. 

Stone platform example:

I used the Dam standard ZBrush tool to add cracks on wood planks and rocks, then chipped the crack sides using the trim dynamic tool. This was to indicate damages and certain scratches on the rock. 

Similar Workflow was used to create the magic stones surfaces and shape silhouette.

I kept the sculpting detail minimal but made the actual sculpting detail large, this was to make sure the detail was visible from my chosen distance/perspective. As I didn’t want a clean beveled look for my high poly models as they would look to plain.

To speed up the workflow, I used the same methods for adding sharp crack and chip damages to most of the assets once. I followed the pipeline when creating the organic assets in my scene. For the unique assets (such as the shield) I would use a similar process but use ZBrush alpha stamps (used from Jones Ronnegard) to add intricate detail, then go over it with the trim dynamic brush to chip damages off the emblem, so that it fitted with the semi-realistic style I was going for. However, the walls and archways weren’t sculpted just to break up some of the main and far ground assets.

If I’m only going to add small details to the high poly asset, I would model most of the asset, so the main detail is there. e.g. using the table as an example I would import the asset into ZBrush, then add chips and wood damages to the table top and legs.

Table parts sculpted in ZBrush, similar techniques to sculpting the stone platform:

High Poly Optimization 

A lot of organic models such as the rocks and stones had to be decimated before being taken into 3DCoat, this was to optimize the high poly sculpts to the fullest extent. This made it easier for retopologizing as 3DCoat could handle the high poly assets more efficiently. This was for the majority of high-poly sculpts to keep a consistent pipeline some assets such as the shield. Stone fantasy base was done using Zremesher to speed up the workflow when I was running out of time.

3DCoat, in my opinion, can handle high-poly assets better than most programs. Retopologizing stones, rocks etc.  did not take long as I would use similar geometry brushes such as the Quads brush and the Add/Split brush when drawing low-poly geometry onto the high-poly assets.

The Quad tool allowed me to draw tris as well when cutting corners, it became very useful when drawing topology on flat surfaces, such as retopologizing geometry onto the rock’s sharp angles. Not to mention you can manipulate the quad geometry effectively, by using the add-and-split tool to create tris. It made it easier when optimizing difficult areas. These were the main geometry tools I used throughout my pipeline when optimizing high-poly sculpts.

Magic stone in 3DCoat process:

Most high poly sculpts went through this process as I felt 3DCoat handled high poly assets more efficiently and that there were more geometry tools to experiment with.

If the high poly sculpts had an equal surface, I would use the symmetry tool and then manually move the geometry using the brush tool. I did this for my wooden planks to speed up my workflow.

Granted I did give some low poly objects a bit more geometry information, just so they didn’t look too low poly and to fit the assets into the art style I was going for.

The cobblestone floor texture was created differently since it was a tileable texture, I sculpted about 5-6 different versions of the cobblestones and made sure I positioned them, so they would tile in 3ds Max and then baked the cobblestone pattern onto a plane. To achieve better results for efficient baked textures I expanded the cage output just a bit to capture the cobblestones shapes and detail. 

Different types of cobblestones sculpted in ZBrush:

I baked every asset out as 4K, with 8×8 sampling apart from the AO and thickness at 2×2 sampling. This was made so that none of the low poly objects had jagged errors from the normal bake and maintained the highest normal quality. This proved useful later down the line when using AO and Curvature maps for texturing.

The workflow for the cobblestone texture was texturing most of the detail in painter using Mask generators and using a baked I.D map to texture the ground separately. 

Texturing

When it came to texturing the assets I would use basic colors to create a base texture before using filters such as the level filters to add some quick and tone and depth to the textures.

Following Dave Wilson’s texturing the chest tutorial,  I learned what types of masks I could use to create interesting effects when it came to texturing the assets.

Using simple tricks such as applying a simple gradient effect to the assets overall texture grounded my assets as it helped to add more volume and depth. Using filters such as Color correct and Color balance was useful when adding my final touches to the textures, as I would use them to add more contrast to the textures tone and change colors quickly to suit the aesthetics I was going for.

Every time I baked the texture maps I felt the generated AO map wasn’t doing enough to add depth to the textures. I used a simple method which was to apply the AO map as multiple layers and only switched on the AO color, this increased the intensity of the shadows which added more depth to the texture.

Making sure the assets fit in my scene and making it believable was tricky, hence I had to experiment with texturing workflows throughout the project. Keeping consistency was the main challenge, I would use basic materials for creating the micro detail and use filters to enhance the overall detail.

Filters such as the baked lighting effect were used to add more color variation to the overall texture.

Using filters to enhance the colors and tones of the magic stones textures:

I used the curvature map to enhance the textures’ highlights and attached the levels to the curvature mask depending on how much I wanted the highlights to show. Texturing the rocks and magic stone this way was a useful technique as it would highlight the rock flat surfaces well, giving the rocks and the magic stone a more hand-painted look.

I would make sure the highlights’ blending modes are either overlay, soft light or passthrough (on a low percentage) and make the highlights a light color (depending on the assets I’m making). This way highlights weren’t too obvious and looked like they blended into the actual texture. 

When making the wood materials, I would use basic wood textures and blended the materials’ elements together to create the main wood effect and would then add filters such as the warp and blur filters to make some the detail look stylized. After this I would create my own smart material from the one I made and apply it to other assets which were wooden, this speeded up my workflow as I used the smart materials to do 80% of the texturing. This meant I had more time to allocate to other tasks.

E.g. Using the warp tool to make the directional wood grain look hand painted and using the blur tool to soften some of the micro detail.

To make the assets look like they were hand-painted, I add mask generators in the textures masks. This was effective as I could fill color onto the sculpted detail well since I had a lot of parameters I could experiment with, such as the curvature, AO and world normal elements.

Using the curvature element, was useful for adding colors into the small sculpted detail such as the magic stones sharp symbol.

Without using the curvature map as a mask, it was difficult to fill color in the magic stone symbol.

Using the main curvature control allowed me to mask the symbol area and fill in with the blue color without going over the edges.

Then I would simply apply Unsharp mask to enhance the detail, so it’s more crisp and noticeable, I would experiment with the settings depending on how much I wanted the detail to show.

Using PBR reflections were useful for making sure the assets have a believable presence. For example, when I came to apply similar texturing techniques to the metal assets I would use binary metal values for the pipes, furnaces, anvil etc. Then to represent how worn out the metal asset is supposed to be I would alter the roughness detail while still maintaining the stylized approach.

I messed around with the omni and point light settings, I made the attenuation lighting settings high. This helped capture the assets reflective detail well while still maintaining a stylized look. I deliberately made some of the assets’ roughness detail stand out, such as grunge marks, stains etc. To represent what type of material the asset has.

Experimenting with effects such as levels and altering the mask opacity settings gave the roughness an interesting result. This improved the assets reflectivity when interacting with the light sources as it was easier to see the stylized detail, however, I did have to make some off the light sources brighter for the detail to be more noticeable from a distance.

The PBR texturing workflow worked well in general, but in some cases when I made the assets in painter have a metalness value the assets texture would become a lot darker due to reflectivity elements. I solved this issue by simply brightening the textures value so that you could see some of the albedo detail.

To make sure the assets materials worked well with the lighting, I messed around with GI settings in Marmoset Toolbag and tweaking the GI voxel detail. This was an ongoing process until I found what I was looking for, I tended to make the settings as high as possible to get the lighting bouncing off the materials effectively. This also casts high-quality shadows in the scene which grounded the assets better.

Experimenting with the G.I settings:

Ideally, local reflections were turned off, as I wanted the lights to cast the environments shadows.

Rendering

I rendered the environment in Marmoset Toolbag 3, it works like Unreal Engine the main difference being you don’t need to create lightmaps for the assets as they are pre-generated in Marmoset.

One of the biggest challenges was to get the window blue light reflection to work, the omni lights were not creating a realistic reflection through the windows. To address this problem, I changed the light coming out from the window from an omni to a spotlight and increased the radius and intensity of the spotlight.

I also had to make the window bars have cast shadows turned on, for the spotlights to capture the reflection of the window well. Rotating the directional light source created interesting light bounces in different areas and helped improve the lighting in general.

To break up the transition from orange to blue I added slight darker and lighter variations of both colors near the main lights to add more contrast and interest. Even adding different hue and saturation level lights helped balance the overall lighting scheme.

Experimenting with the light sources

Messing around with the fog settings help make some of the lights have more of an atmospheric element.

I used the cameras post process effects to alter the contrast and intensity of the different light sources just to add more variation to my environment’s lighting. I made the render settings a high quality and tweaked the ambient occlusion to add more depth to the environment’s lighting.

I placed a blue light near onto the texture of the floor, this was to fake a light bounce coming from the moonlight source to give the cobblestone texture a more 3D effect. 

I simply added an omni light near the floor and made the light intensity strong but made the attenuation radius very small, so that it only reflects a certain distance. Faking this lighting effect made the moonlight source have more of a visual impact when guiding the player around the ‘Forges’ environment.

Challenges

It had taken around roughly 2 months alongside other projects. I would say the trickiest part was trying to establish a more concentrated focal point in a 1920×1080 environment. Designing my environment’s layout was tricky as I had to take parts out as I underestimated how long it would take to create certain assets.

The other task I found difficult was trying to get lighting to cast properly through the window as I felt the reflection was too weak.

Overall it was a huge learning experience. I thoroughly enjoyed working on this piece and have learned a lot about the stylized work route. I will continue to push myself further and develop my artistic skills!

I’m grateful to my peers and tutors for feedback and thank you for checking out my work!

Tim Boulton, 3D Artist