Case Study: Vector Displacement Mapping in Real-Time

About Displacement Mapping

Displacement, not to be confused with Parallax Mapping or POM, is traditionally used to offset vertices along a mesh face normal. Take a rocky ground or brick material applied to a simple quad, offset (displace) its vertices and you’ll get a wobbly looking effect due to its limited polycount. Tessellate that mesh and you'll start seeing a closer representation of your displacement texture. But there’s a catch, your regular displacement texture only contains height data to offset your vertices in a single direction. This is where Vector Displacement Mapping with Amplify Shader Editor comes in handy.

Traditional displacement is usually not overly common in games due to the polycount requirements, developers tend to favor techniques such as Parallax Mapping which provides a decent but limited approximation. Surprisingly, displacement, or offsetting, is quite common in real-time VFX whenever you need to deform a mesh or simply apply a waving effect to something like a waving quad for a burning flame effect or swaying vegetation; a simple height is usually further manipulated and combined with other effects in the actual shader.

One notable exception is terrains, almost every traditional terrain in games is created by some sort of regular height displacement.

The use of Vector Displacement Mapping is even more uncommon, usually reserved for film and other offline rendering fields and regularly coupled with a form of subdivision as the displaced mesh requires a higher vertex count to accurately render its details. This is usually seen in highly detailed characters, be it human or scaly dragons. VDM is one of the most performant ways to render an ultra-detailed model while still being able to work with a simplified mesh. 

VDMs could actually be considered a form of Geometry Image, a technique recently made popular by its use in Unreal Engine 5 Nanite.

Where Vector Displacement Mapping Is Applied

Vector Displacement Mapping is not just for rendering, it's heavily used in asset production. It’s a great way to reuse sculpted details as stamps in order to ensure consistency between designs. You can actually create and distribute your own VDM Stamps for ZBrush and Mudbox. The simplicity with which they can be applied and mixed in a sculpting tool such as ZBrush, somewhat like kit-bashing, makes it a must-have for any pipeline which has spawned a whole community dedicated to producing them.

You can mix and match like Nicolas Swijngedau in his awesome Horror VDM Brush Pack.

Or use it for something more practical like his Beard VDM Brush Pack.

Vector Displacement can be used to offset your vertices in directions other than the mesh face normal, meaning that you can displace your mesh with enough precision to approximate it to the original high-poly vertex positions. 

Take this mushroom as an example: without vector displacement you could only offset it up but with VDM you can actually achieve something closer to the original sculpt.

Experiments with VDMs and Amplify Shader Editor

We’ve been looking at a possible simplified workflow that would allow users to take advantage of VDM in real-time without getting caught up in technical requirements. It turns out that the most troublesome part is actually getting data you can use in engines such as Unity 3D. Once we got a good handle on how VDMs could be created with Mudbox, we began to experiment with Amplify Shader Editor for Unity.

VDMs are actually quite easy to set up in the shader editor and, being fully node-based, it’s a joy to experiment with.

We began by trying it out with the common “Mushroom” we usually see in these types of examples. It’s also important to note that we’re purposely going for extreme examples (quad to high-poly). Users can still use low-poly meshes that closely resemble their high-poly counterparts. 

This type of VDM is great for individual objects but it could also be used in patches of detailed objects; we could even use a cutout shader to hide the unused base of the quad. Since we’re using Tessellation, we can pretty much have free LOD control based on camera distance.

We were pretty happy with the results but we decided to try something a bit more practical so we went for the next logical thing, a Poltergeist/Videodrome style TV that will haunt your dreams.

We started by building a basic low poly TV with UVs that we then took to Mudbox for detailing. Each element was done in a separate layer so that we could export them individually for later manipulation in the shader. 

In this particular effect, we baked the front, side hands, and a skull, into individual EXR textures which are then blended together on command with the base TV textures. We also added an emissive noise effect for extra flair. 

What you see here is actually a simple Lerp between normals and displacement values.

How to Create a VDM in Mudbox and Import It to Unity

The first step should be to test your authoring tool in order to make sure the exported maps can be used in Unity. Our editor package includes an HDRP sample to get you started but this can easily be converted to other renderers using Amplify Shader Editor.

We recommend using Mudbox as it seems to be the easiest way to get consistent results with almost any type of bake using the Absolute Tangent mode (object also works, just be sure to add a One Minus node on the R channel output).

1. Start by creating a Plane in Mudbox, name it low.
2. Clone the Plane, name it high.
3. Sculpt something on the high, like an ear or a cool wall, for example.
4. Select the low and go to “Extract Maps”.
5. Toggle Vector Displacement, set the high mesh on the list, choose Absolute Tangent, and be sure to save it to a 32bit EXR.
6. Be sure to also toggle the Normal map extraction and bake it using the Subdivision method (the normal texture can be 8bit).

That’s about it on the Mudbox side. Follow these steps in the editor:

1. Import your 32bit EXR VDM texture and the Normal map.
2. Make sure that the normal map texture parameters are set to “Normal Map” in Unity.
3. Disable sRGB (Color Texture) on your EXR VDM.
4. Make sure that your project is set to Linear, otherwise, Unity will mess up your negative values by clamping the EXR 0-1.
5. Drag and drop the Normal into your Amplify Shader Editor canvas and plug it into the Normal input.
6. Add the EXR VDM texture; you’ll need to add an append node so that you can swap the Green Channel for the Blue, and you’ll also need to add a Transform Direction node so that you can go from Tangent to Object.
7. If your displacement appears “off”, try adding a Scale multiplier node. 0,10 seems to do the trick for most models intended for Unity.

Those are the basics. If everything works as expected, you should end up with something like this:

We took a few extra steps with our wall using Substance Painter, it’s a quick and painless way to texture your objects. 

1. We exported the low, mid, and high-poly versions of our mesh.
2. The used Substance Painter to generate the base wall without destruction (optional).
3. And imported the damaged mid-poly to create our damaged version of the material (we used the corresponding damaged high-poly version for map generation).

Here’s a complete guide for our wall effect:

Where It Can Be Used

Although you get free LOD control with Distance-based Tessellation, the Tessellation itself is currently performance-intensive. We believe that it’s safe to assume that the next generation of GPUs will greatly improve tessellation performance as well as geometry imaging through the use of Mesh Shaders.

It might not be currently feasible to add it to a typical game; however, there are many other uses for the technique in more contained scenarios not bound by regular game constraints.

There’s a couple of examples that come to mind when it comes to experimenting with VDMs such as detailed characters with complex surfaces (aliens, creatures, dinosaurs, etc.), complex character attire, highly detailed scenery ornaments, sci-fi panels, wall/ground damage decals, actual scenery pieces with details usually not possible with regular displacement or Parallax Mapping, or even tiled VDM textures for a cave wall, interior, or ocean waves.

There’s really no optimal usage rule on how it should be applied, the technique is flexible enough that you can combine it with other effects such as our TV hands where we add a waving effect by multiplying the VDM with a Noise Node.

There are a few notable projects out there that use regular displacement with Tessellation, Ghost of a Tale is one of the most heavy-use games with excellent results. But VDM is fairly underused in real-time applications; we actually don’t know if there are any that use it at the moment.  

You’re most likely to see assets in games that have been created using some form of Vector Displacement Mapping; there’s a lot of content out there in VDM form, for organic and hard-surface art.

At the end of the day, it’s really for users who like to experiment with novel techniques. We look forward to seeing what crazy stuff they come up with!

Note: We tested a few alternative tools to generate the VDM textures but, unfortunately, our tests were inconclusive as to which method is the best. If you have any insight we’d love to hear from you on possible solutions for ZBrush, Blender, or alternative software. You can find us on the official Amplify Creations Discord.

ZBrush provides a scene for VDM testing. In Blender, it is a bit more complex to set up but it seems that some work has already been done by nutelZ (not yet fully compatible with Unity/ASE); we’d love to see this method become a go-to technique for Blender artists (you can download nutelZ's Custom Blender Node here).

Amplify Shader Editor is currently available at the asset store for $60. We also recently announced the Amplify Bundle, users can save up to 25$!

Amplify Creations Team

Interview conducted by Kirill Tokarev