Voodoo Eye: The VR-to-Unreal Engine Pipeline in Animation

Introduction

80.lv: Please share a little intro for those who missed our previous story. How did you get into animation and VR?

Eric Giessmann: Hello, I am an independent Animator and a 3D Artist specializing in virtual reality. I am also the founder of the VR team known as LAVAMACHINE.

In my previous article, I introduced myself and my work, but I'd like to provide a quick update. Over the past two years, I've been working on a series of three animation films. The first two films, "What Time Is It?" and "Reincarnation Mall", are already available on Meta Quest and have been screened at various festivals around the world. The third film is currently in its final stages of production.

What sets these animated short films apart is that they are primarily designed, modeled, and animated within virtual reality using an application called Quill. Afterward, I often refine them by applying shaders and adding lighting to achieve a unique stop-motion style that I greatly appreciate. They are rectilinear stereoscopic, meaning that when you watch them in VR, it's like watching a 3D movie on the screen in a "real" movie theatre. Sometimes, I also create full 360-degree animated short films. You can find more about my work on my website.

Joining Epic Bootcamp

80.lv: Could you tell us about how you got into the Epic Bootcamp? What motivated you?

Eric Giessmann: I came across a post on LinkedIn from an artist I had previously met at a local event. He was set to be a mentor for the Epic Bootcamp 2023: Animation, and it immediately piqued my interest for several reasons:

• It aligned perfectly with my area of expertise.
• Unreal Engine was entirely new to me, and I had heard exciting things about its real-time filmmaking capabilities. I was eager to explore whether it could streamline my workflow, especially in VR content creation.
• After completing my last animation film for Meta, I had the time and enthusiasm to learn new skills.

I submitted my portfolio and, a few days later, received confirmation that I had secured one of the remaining seats.

The Epic Bootcamp challenged participants to create an animated short film based on one of four classical storytelling "blocks". I was assigned the theme of "obstacles", with a duration limit of 15-30 seconds. Simultaneously, we had to familiarize ourselves with Unreal Engine.

The entire event lasted four intensive weeks, with about 8 hours of work per day. Our story revolved around a hero who discovers an artifact with the power to control people and encounters various obstacles. I took this theme quite literally to challenge myself, resulting in a short film that I'm genuinely proud of. Among my peers, it was selected as one of the top three.

The Voodoo Eye animation:

Reaction to Unreal Engine

80.lv: What were your first thoughts on Unreal Engine? What did it feel like?

Eric Giessmann: My initial reaction to Unreal Engine was, "It's unreally complex, with numerous plug-ins, workarounds, and an overwhelming number of options." Coming from a virtual reality environment where simplicity and limitations were the norm, with no additional features, game elements, and a straightforward UI, I felt somewhat overwhelmed.

I also sensed that Unreal Engine was not primarily designed for filmmaking but was more of a supplementary feature. Fortunately, my background in Maya gave me the confidence that I could grasp the basics of this engine.

Comprehending the Engine

80.lv: How was your learning process organized? What things did you learn first? What were the main challenges?

Eric Giessmann: Initially, I struggled with the storytelling aspect because the 15-30-second limit was shorter than expected, and I had too many ideas that exceeded this duration. I was determined to stick to the time constraint, as I knew that learning a new software would be demanding enough. I created storyboards for three different concepts, ultimately selecting one with a fantasy genre to maintain clarity from the outset. I also decided to incorporate a small humorous element to connect with the audience.

I was aware that most participants would opt for a hyper-realistic approach, so I chose to create a stylized concept design to align with my portfolio. Additionally, I aimed to independently create and rig all assets in Quill to avoid dependency on assets from the Unreal Engine Marketplace and maintain the charm of VR design.

The main challenge was managing the frame rate. For my stop-motion-style film, I prefer working with a maximum of 12 FPS and even 8 FPS, like in this case, while maintaining 24 FPS for smooth camera motion. However, working with game engines posed difficulties in achieving this, as they typically aim for higher FPS. I had to implement workarounds in Quill to achieve the desired look.

Working in Quill & Unreal Engine

80.lv: How did you mix Quill and Unreal Engine? How difficult was it to switch back and forth between different tools?

Eric Giessmann: Initially, I conducted numerous tests to ensure that characters and scenes were correctly imported into Unreal Engine while preserving the colors, size, and proper sequencing from Quill. I opted for alembics as the export format, leveraging previous experience with this format from my earlier films.

My goal was not just flat shading; I wanted to add subtle shadows and textures. Therefore, I had to ensure that the post-processing volume didn't affect colors incorrectly and that project settings were properly defined. The sequencer posed an initial challenge, requiring me to explore workarounds and techniques to make alembics exported from Quill function seamlessly.

Typically, I would attempt to export the camera from Quill and use it as the source for camera motion in Unreal. However, I struggled to establish effective parent constraints quickly, leading me to recreate the cameras within Unreal Engine.

In Quill, I initially created key animation poses for my characters, imported them into Unreal Engine, and paired them with 2D storyboard images. This approach proved helpful. I timed the cameras before returning to Quill to refine and lock the shot lengths. During this pre-production phase, there was some back and forth.

80.lv: Did you like directing in Unreal? What tools did you use? How smooth is the whole VR-to-Unreal Engine pipeline?

Eric Giessmann: Unfortunately, Unreal Engine 5.2 did not fully utilize both of my GPUs. The delay between cursor movement and menu item selection was somewhat frustrating. Frequent crashes were also a concern, and the save function required some effort to master. It's crucial to back up your project regularly.

However, once I established an effective Quill-to-Unreal Engine pipeline, the interaction became smoother and more efficient. I gradually became comfortable with this workflow. I handled the design, look development, animation, and rigging in Quill, while camera work, lighting, shading, texturing, and adjustments were completed within Unreal Engine.

I could also use vertex colors and the transparency channel from Quill in the materials, which was a valuable feature. For storyboarding, I utilized Storyboarder, and for audio mixing, I turned to DaVinci Resolve.

The Engine's Advantages

80.lv: Would you recommend the engine to aspiring artists and directors? What are the toolkit's main strengths, in your opinion?

Eric Giessmann: Unreal Engine excels in real-time rendering, particularly when using simple color shading and lighting effects. The real-time smooth shadows and consistently high frame rates within your workspace are undoubtedly the future of film production. I also appreciated the material and texturing capabilities, as they allowed for a lot of flexibility. However, Unreal Engine does demand a certain level of programming knowledge in various situations, which could be challenging for artists who are less inclined toward coding.

I recommend this pipeline to artists seeking a faster method for short film creation, as it eliminates the need for lengthy render farms and high-poly counts, allowing more time for creativity. VR artists interested in incorporating VR elements into their games will also find this pipeline enjoyable.

If you want to learn more about the Quill (VR) to Unreal Engine pipeline, you can also check out my brand-new comprehensive video tutorial, which includes all Quill and Unreal Engine scenes from Voodoo Eye.

Eric Giessmann, Director and 3D/VR Artist

Interview conducted by Arti Burton