Aquatic Alien: Setting Up A Caustics Effect For Characters

Introduction

A number of people reached out to me asking how I managed to make this scene work in Marmoset Viewer – so I’ve decided to make a little breakdown of it. First of all, I want to point out two things. Firstly, I didn’t use any plugins, add-ons, or custom shaders. Secondly, I made two versions of this scene – one for hi-res rendering in MT3 and another for Marmoset Viewer

Workflow

At first, I was planning for neither the character nor the environment to be animated at all. What I wanted was for the scene to have an underwater feel to it. So the first thing that I wanted to do was the Underwater Caustics effect.
 
Of course, this effect is difficult to achieve in a real-time engine, at least in a physically-correct way – so most of the time it’s an animated texture or projected decal or something like that.

Marmoset Toolbag has this cool feature called Gel – it’s a property of a light emitter that allows you to use an image as a light mask so you can project images via a light source like in a cinema. Here is the solution, I thought – that’s what I’ll use to make the effect. So I found some caustics texture online, plugged it into the Gel slot of one of the light sources and it worked.

Then I thought - wouldn’t it be cool to make it move?! So I searched for an animated texture online and found this free app called CausticsGenerator. It can generate looped sequences of tileable textures of underwater caustics. Also, they have some examples of already made sequences on their website. So I downloaded one of them, made a GIF off of it, and tried to put it into Gel and failed because Marmoset Toolbag doesn’t support GIFs. 

But I wanted to make it animated anyway so I created a bunch of light sources each having its own Gel texture – each texture represents a phase of the caustic animation.  Each of these materials should be rendered in a sequence to simulate the change in the caustics phase. 

Having all that ready I made a keyframe animation turning each light on and off in a sequence – a total of 16 frames in a loop. And it worked!

Ok, that one’s dealt with. Then I decided to test it in Marmoset Viewer and failed again because MV doesn’t support the Gel feature. But it looked so cool – I just had to make it work in MV. The answer was – to fake it!

In theory, I can use decals or UV2 to project additive texture atop the model. UV2 is not supported in MV so decals are the way.

The first step to make those decals was to create a new UV layout for the caustics texture to simulate light projection on the surface of the model. To do so I made a copy of my model and used Planar Mapping based on the camera view in Maya.

The resulting UV layout looked something like that:

I wanted this effect to be visible only on the “lit” parts of the model so I've deleted all the back-facing polys and started to clean up the rest. After deleting all the unnecessary parts of the decal and adjusting it in UV Editor it became this:

Here’s how it looked together, the character model and the decal geo:

After putting it all into Marmoset Toolbag I created a new material for it. Plugged the caustics texture into Albedo and Emissive slots and used B/W Circular Gradient texture for falloff effect.

There were some areas where you can see a hard edge where the decal geo ends, also some areas shouldn’t have caustics at all since they are “shadow” sides. So I drew a mask in Substance Painter to fix those areas and create a smooth transition between the decal and the body.

I’ve blended it with the circular falloff mask I was already using to further smooth out the transition.

This is how the mask affected the caustics:

But of course, at this point, it’s just a static decal – how to make it animated? Animated textures don’t work in MV so the only way is a keyframe animation. To achieve that I needed 16 materials with 16 textures. Of course, I couldn't swap materials for each frame directly so I should duplicate the decal 16 times and turn it on and off, frame by frame. At this point, I had two ways to approach it. The first one was to make 16 copies of the decal in Maya, apply a unique material on each one of them, and export all of them in 1 FBX along with the character. The second was to make 2 FBX files – one for the character and another for the decal and duplicate the decal inside of Marmoset.

Bypassing Technical Restrictions

Since the character was made as an entry for the challenge on Artstation I had to take into consideration that there’s a size-restriction for MV files – 25 MB – so I’ve chosen the second approach. Both cases are heavy on performance because of the number of materials in there – despite what type of rendering MV uses, more materials means lower performance. On the other hand, there might be a huge difference in how geometry affects this scene’s performance. I don’t know how an MV scene is assembled but I bet smart people who worked in that took some time to think about ways to optimize all the things artists put into Marmoset.

With this in mind, I made two FBX-files, duplicated the decal geometry and decal material 16 times, assigned the right textures to all the materials, and applied each material to its corresponding decal. After that, I’ve created a keyframe animation of decals visibility attribute turning on and off one-by-one.

After exporting it all as a Marmoset Scene and seeing it work I thought I should animate the character as well. And so I did!

I made some idle movements of the limbs and body. Of course, I couldn’t fit it in 16 frames so I had to make it longer. 128 frames timeline seemed optimal – it’s long enough to allow slow movements of the character, it can fit several 16-frames sequences of caustics animation and it’s not too painful to make a frame-by-frame animation of 128 frames in Marmoset.

After skinning and animation were done I copied the skin weights from the character model to the decal geo and exported all of that again as two different .fbx files with animations.

Of course, having skin weights and 16 clones of animated meshes bloated the scene’s size but it looked cool.

Unfortunately, I had to lower the texture quality of the scene to keep its size under 25 MB for the challenge thread but again – it was all worth it. 

One thing I want to stress here is that this approach is one dirty workaround and it’s not a good way of making animated characters and scenes. But since Marmoset Viewer has such harsh technical restrictions I don’t see how it could be done differently. 

Ok, here you have it – one of the ways to exploit MV’s limitations and make a good presentation.

Sergei Gereev, 3D Character Artist