Replica: Creating Talking Animations for 3D Characters with AI Voice

Introduction

Hi, my name is Tobias. I’m from Germany and I’m a Content Developer for Reallusion’s Character Creator/iClone and an Animator. While studying Media-Management I developed a love for the animation industry and after finishing my studies I worked as a Production Manager in an Animation Studio. After some years I took the chance to start my own business and worked on projects with clients for the edutainment market. Around 5 years ago, with the release of Character Creator, I started to mainly focus on Content Development for Reallusion tools and since then it has been a great collaboration with Reallusion with hopefully many years to come. 

Using Reallusion's iClone

I came in contact with Reallusion’s iClone for the first time during my studies. It was Version 2.5 back then and compared to the current version 7.9 it was very limited in its features. But somehow I really had the strong feeling that there was a lot of potential in this little software and it already was a lot of fun to create and animate your own characters and projects with just a few clicks. The idea of having a complete production studio on your computer was really mesmerizing and accompanied me the following years.

AI Replica Toolkit

Replica is a company that specializes in providing AI voice libraries. Thanks to the Replica solution, creators now can work with a huge library of natural-sounding voices. You just need to input your dialogue lines in Replica Studio, select a voice from the library and let Replica generate the speech with one click. If you compare this to the time-consuming and cost-intensive process of character voice casting in the animation industry you will see that this solution is a real game-changer: No need to hire a voice actor and no voice recording sessions needed. And with the new AI Voice Actors Plugin for iClone, it has never been easier for animators to create lip-sync animation. In general, characters can now be brought to life in just a few clicks.

With the Replica Plugin for iClone creators can use all the advantages of the iClone/Character Creator ecosystem. All Character Creator characters are fully rigged and ready for face/body animation and can receive facial animation via the Plugin.

Just make sure you have the Replica plug-In installed onto your iClone software. For more information about the plug-In installation, and other helpful hints and tricks, please visit the official Reallusion product page.

You will also need to have the Replica Studio software installed on your computer in order to connect iClone via Replica Link. Click here to create a free account and download the software.

Setting Up the System

To open the plug-in in iClone, go to MainMenu > Plugins > Replica Studios > Replica Link. Now you can select the character in your scene via the Replica Link and it is ready to receive the voice from Replica Studio.

So just switch over to Replica Studio Sandbox and type in some lines of dialogue, then click Select a voice… to choose a voice from the Replica voice library.

You can scroll through the library and preview the voices or use the provided filters to search for specific features of the voice. A total of more than 45 voices can be found here.

After you’ve selected the proper voice you just need to press Play to let Replica Studio generate the voice for your character's dialogue. This provides a quick way for you to review different voices.

Try to play with the “Style“, “Speech Rate“ or “Pitch“ to further adjust the voice to your liking and save the take to export it to iClone.

After you’ve clicked Export to iClone in Replica Studio, iClone will ask you to select an Animation Style and Strength Level that will be applied with the generated voice to your character. There are a bunch of animation styles for different moods from ”Anger“ to ”Surprise“ available for your character's facial animation. Click Apply and let iClone do the rest. 

When finished, your character will have a cool-sounding voice, great lip sync, and a base facial animation. Here you can already see your character talk with a basic facial animation matching your selected talking style. 

If you are not satisfied with the animation style or strength level you can always change it using the Expression Loop presets that can be found under Content > Animation Template > Face > Expression_Loop. Needless to say, you can create your own facial expression clips, but this library gives you another level of automated preset motions.

Tweaking the Result

To tweak the final generated sequences just check out iClone’s built-in animation tools, Face Puppet, and Face Key to finetune your character’s facial animation afterward. It’s easy to add some more facial expressions for the whole face or just certain areas like eyes, brows, or mouth/lips with these tools. 

Face Key is the perfect tool to create individual mouth/lip shapes for certain letters/words of your characters' dialogue to make it look unique. 

Use the Muscle, Expression, and Modify sliders to create special morphs and to emphasize certain letters or words. Additionally, you can manually adjust the auto-generated visemes of the voice track to acquire the perfect lip sync.

Conclusion

With the new Replica Plugin for iClone, the dream of a whole production studio on my computer has definitely come true. Whatever project – a simple character presentation, an animated trailer, a full movie, or a complete game – everything is possible. All good stories start with convincing characters and with iClone and the new Replica Plugin it has never been quicker and easier to bring them to life!

Tobias Kolodziey, Company Owner at ToKoMotion

Interview conducted by Arti Sergeev