How to Make a Custom Node System in Blender's Shader Editor

Introduction

I began studying the Blender Shader Editor independently during my final year at the academy and completed it this summer. My learning journey was long and difficult, partly because I lacked a structured path, something a good course typically provides. Later, I subscribed to the Blender Foundation and watched Simon Thommes’ Blender training course. I wish I had found it earlier, as it would have saved me months of self-study.

The Thought Behind the System Creation

I began considering this system after learning Substance 3D Designer at the academy. Comparing the pros and cons of the layer system in Substance 3D Painter with the node-based approach in Designer, I found the node system more intuitive initially, but less practical for later modifications compared to a layer-based system. Here is an example of it:

On the left side of the black frame, you can see three grey frames, each containing a different procedural grunge. These are connected to various channels: color, roughness, and normal. I haven’t made any specific changes to the roughness or normal channels, but for the color channel, there is a stack of nodes at the beginning, which modify the grunge they take, followed by a series of mixed nodes to combine them.

Here is the annoying part, and for this reason, I haven’t done the same complexity of modification to the other channels. This is something you can struggle with if you don’t organize and compound the nodes into groups, like this: 

The issue here is the time required to organize the graph, along with the need to make changes and additions, which then necessitates further reorganization.

For this reason, you can create some groups to store in the asset library within a Blender file to have organized and pre-made node setups; you can access it in every other Blender file through the “SHIFT+A” command, typing the name of it or searching through the nested list. Another way is to open an asset library section and drag it to the Shader Editor.

I prefer the first option for ease of use, as you can usually stop at the first two letters of the node group name and press enter. For example, in this case, I wanted to take the custom material mix node. I just pressed “SHIFT+A” and then typed “mm,” the capital letters of the node name “Material Mix,” and then I pressed enter:

There are other two ways to do it faster. One is to assign a shortcut key to the asset node, but this is not practical when you have a lot of custom nodes to assign.

Through a custom menu or mode, we are going into the Python world, where we can make it ourselves (it is not that complicated, especially with the Serpens add-on) or find a prefab menu in certain add-ons like Shader Node Wizard, which I use now for some good things it does. We can also find add-ons dedicated to texturing that entirely change the workflow you can have with texturing in Blender.

Here is another aspect of my thoughts before developing my custom node system.

The Texturing Add-Ons in Blender

Initially, I found these tools impressive, given my limited background in Blender’s Shader Editor and my experience with Substance 3D Painter. However, after learning Substance 3D Designer and studying Blender’s Shader Editor, I recognized the limitations and issues with Blender and many of its texturing add-ons.

Add-ons are meant to enhance Blender by adding features or improving efficiency without affecting the base functionality. However, if an add-on restricts or complicates the core features of vanilla Blender, introduces incompatible workflows, or is poorly optimized, it can reduce its value and create additional challenges.

The major Blender’s texturing problems I found on the way:

• It is not newbie-friendly due to the lack of starter content like Smart Mat, Base Mat, and tools for creating a custom mat with artist-friendly operators and grunge.
• Something similar is for texture painting, lack of brushes, and the brush operators are so simple and old (talking about the texture editor, too) that compared to some other software, it makes you smile and cry.
• Speaking of things that make you smile and cry, here is the worst: the baking system, the vanilla one, is just near to being garbage. So, don’t use it. Switch directly to another software or to an add-on that makes your life easier. In my case, I found the Bakemaster add-on to be the best; the only downside is its complexity and lack of an easily understandable UI, while for the UX, it does a great job. It also has functions comparable to and even better than other software. In this case, the only two remaining problems I found are related to Blender itself, and no one add-on fixes that at the moment. These two problems are: it does have problems baking large pitted meshes, like chainmail; the most important related to the material exportation is the baking time. It is really too big compared to other software and even talking about the export for complex shaders in other programs, you do it as nothing. In Blender, if it’s too big, you run out of GPU memory, and if you go to bake it in CPU mode, you are going to wait a decade for it and a century more for baking the normal map.

For the issue of long baking time, I found a possible solution, thanks to Cartesian Caramel, who brought it alive when the portal node was recently added to Blender.

In summary, Blender can be effective for texturing with the right custom nodes, add-ons, and resource packs. Currently, the main issue is the long export time, which is more problematic than high-to-low baking. As I mentioned, I just figured out how to solve this with a bit of visual Python scripting, made easier, thanks to the Serpens add-on, to automate a better-optimized baking system that is nowadays possible to obtain easily.

Now that we've covered the background, I can discuss the temporary custom node system I created for texturing the Foldable Icepick asset.

I refer to it as “temporary” because it still requires additional components and modifications to become part of a Python-automated system, which will make it more user-friendly and functional.

The Custom Nodes System and the Workflow I Use

The system I developed aims to balance efficiency, optimization, and practicality while also being easy to modify and reuse. It is designed so that both the entire material and its individual components can be treated as reusable assets. The system is structured in a hierarchical manner, with larger components containing smaller, nested elements.

Layer 1: The modified BSDF and the Material Stack

Here is where the material creation starts, using these 2 custom nodes. I also highlighted a third part, which is not a custom node but a workflow utility.

Speaking about the workflow, I want to mention the main add-ons I use for that before starting. They are Voronoi linker, Shader node wizard, Bake master, Noise Nodes, Philogix Pbr Painter (just for the HDRI rotation operator) and Sdf Nodebox.

1. Modified BSDF: I use a Principled BSDF with custom nodes added for better flexibility. This setup allows for easy adjustments to individual materials through exposed parameters.

It includes features for integrating baked maps into the main material, which is useful for a high-to-low poly baking workflow. The setup includes a basic configuration for main maps and nodes for general modifications, making it easy to apply consistent adjustments across all instances of the node group.

It can be useful if you want to use a map, like the AO, to mask out cavity zones from edge scratches, make a dust mask, or something similar. The same is true for the curvature and normal baked maps.

2. Layerable smart material: it is just a prefab container of a single material.

It is made only with inputs and outputs with the BSDF’s base settings and a material mask; it allows a fast connection between multiple materials if necessary. This is permission from the Voronoi linker add-on. It has a multi-channel connector operator that easily lets you connect multiple channels of two nodes in case the outputs of the first node and the inputs of the last node are equal.

As you can see in the images below, all the In & Out are contained in closed panels to optimize the space:

Advice for Organizing Node Groups

Always use panels to organize channels, and apply the panel’s name as a prefix for all channels to maintain consistency and organization. Prefixes can be omitted when connecting to nodes that do not use prefixes in their channels. For instance, I used prefixes for SSS and some special channels but not for the base panel, as the Principled BSDF does not use prefixes for its base and some special channels.

3: Fast switch from node preview to BSDF: This is the second Shader Editor I opened for when I want to quickly switch back from a node I’m previewing into a nested group to the BSDF view without having to switch back and forth between the various groups.

Layer 2: Material Layers

The smart material custom node uses its inputs and outputs to mix layers from different smart materials, thanks to the material mask custom node. This node connects to the first layer in a sequence and applies a unified material mask to all layers.

This node is primarily for reusability, allowing you to save it in a library and use it across different projects. It is not typically used to combine multiple materials into a single Blender material due to potential performance issues from having too many nodes. Therefore, I generally use one material per Blender material and only mix them when necessary to use multiple mats on the same faces.

1. Layers: You have to consider the layers as Substance 3D Painter fill layers, which you have to connect from left to right, as they are also indexed in the two in-and-out frames (START) (END). You can easily connect them with the VoronoiLinker add-on because they work the same as the Smart Mat nodes. Like the Smart Mat, they contain a layer mask to separate the various layers between them, always like the Substance 3D Painter layer mask.
2. IOR: The IOR (Index of Refraction) is kept separate because it generally does not require modifications that would complicate the node graph. Adding it to normal layers avoids unnecessary calculation time.
3. Mat Mask: I just explained most of it previously. Here, I can add its versatility and optimization for a UDIM workflow.

You can see its setup here below:

Layer 3: Layer Sections

To better understand the layer function, it is divided into four sections, similar to Substance 3D Painter layers, from left to right:

4. A layer mask: This section applies a mask to the layer, as previously described.

3. The layer channels: These are the base channels of the custom layer. You can add different types of channels by adding separate layers to save calculation time and improve shader editor performance.

To disable a channel, mute it with the “M” key. To adjust the opacity of individual channels, use an Add math node with negative values.

2: Modifiers: This section allows you to modify the generators connected to the channels, such as adjusting roughness levels or using ramps to modify layer color.

1. Generators/Grunges/Patterns: This is the core of the layer, where the layer details are created. You can use 2D textures, custom nodes, or procedural nodes (like noise textures).

For optimizing viewport performance, consider using a Generator custom node.

Layer 4: Generator

1. Procedural Grunge: This section is used to create your generator, such as a grunge texture combining procedural and non-procedural elements. You can also generate other procedural effects, like edge wear or similar.
2. Baked Grunge: Add a new texture using the image texture node and bake the procedural effects onto it. After baking, mute the procedural setup connections to prevent further processing by your computer. Also, mute the emission shader connection to return to the main material view in Cycles.
3. Emission Cycle Bake: When connecting something to the material output within a group, Cycles will use this output instead of the main one associated with the Principled BSDF. This setup allows for faster baking of the procedural elements in the generator custom node.

Conclusion

Creating textures for this asset using Blender's node system was a valuable experience. It not only gave me more practice with node-based workflows but also deepened my understanding of how things work at a vector and mathematical level. Unlike some other software, Blender lacks pre-made nodes for certain tasks, like distortion, so you need to build them from scratch. This process helps you grasp the underlying mechanics more thoroughly. There are plenty of resources on YouTube to guide you through this.

For beginners interested in texturing with Blender, I recommend first getting comfortable with the basics — learn the shortcuts, understand the interface, and explore useful add-ons. Avoid add-ons that simply offer pre-made materials if you want to learn how to create your own; instead, look for those that enhance your workflow.

While YouTube tutorials can be helpful, consider investing in a comprehensive course like the one I mentioned at the start of the article for a deeper understanding. If you’re transitioning from another software, remember that Blender has its own unique way of doing things, so approach it with an open mind and focus on learning its specific workflow.

Giuseppe Carbone, Hard Surface Modeler

Interview conducted by Gloria Levine