Creating an Immersive Vlog-Like Experience in Unreal Engine 5

Emir Cerman talked about the background of ROTU Entertainment, shared the working process behind the Luminescent project, and showed how the Hexa creature was made.

Introduction

My name is Emir Cerman and I am the Chief Creative Officer at ROTU Entertainment. 
My journey is a little unusual. I was born in Istanbul, Turkey, and as far as I can remember, I have always been interested in visual art and storytelling. In my early days, LEGO was the first tool I used to express my storytelling. I would also draw and split those drawings into different pieces to build worlds. I was an only child, so I think creating uncharted worlds and imaginary heroes was my way of escape.

My bottomless curiosity led me to discover the early versions of 3D Studio Max and Maya at a tradeshow circa 1999. I remember this feeling of uncontainable excitement because I had discovered a new avenue to explore the depths of my imagination. Back in the day, the only way you could learn these tools was through instruction manuals translated into Turkish. During those years, I discovered my other passion – film music. I started getting into music deeper and deeper and one day I picked up a guitar and tried to learn to play so I could play around the summer campfire.

After many years of trying to explore many musical instruments, I ended up falling in love with writing music. At this point, I knew I wanted to pursue this as a career, so I got introduced by a friend to this prestigious school of music called “Berklee College of Music”. I then knew this was the place I had to be. In 2007, I got to attend one of their auditions in Kuala Lumpur, Malaysia. 3 months after the audition, I received the news that I got accepted to Berklee with a full scholarship. And then in January of 2008, my USA journey began. During my student years, I found myself surrounded by many talented musicians from all over the world, and that led me to create a new project called “Rhythm of the Universe”.

With the help of Jason Parks, one of my classmates at Berklee, I gathered 90 musicians from 90 countries to write a song called “Anthem for the World. This student project led to giant sold-out live shows for many years, and we finally formed the company called ROTU Entertainment in 2012.

Since then, we have created many purpose-driven projects, like going to the Peruvian Amazon to film and write music to bring awareness to illegal gold mining and climate change. At that stage in my life, I was hungry for creating more ideas, and this was the point I started to write a new fantasy world idea based on music mythology.

We then knew that as a team we had something special, but we didn't know how to introduce this universe to people. In 2016, we got to try HTC Vive for the first time. I remember so vividly leaving the room and saying that this is what we need to do. The biggest problem was none of us were game developers, but we all looked at each other and said, "Well, let's go and figure it out!" This was full circle for me and now, after all those years in the music industry, I'm back to playing with the 3D tools again. 

The Background

Our team has a background stretching all the way from Live Entertainment to Virtual Production and Video Games. Back in 2011, we started producing live shows that gathered musicians from 90 different countries under one roof with the goal of creating one-of-a-kind live experiences. Since then, we have become an independent studio that creates VR games, immersive story living experiences that inspire empathy and reinforce the power of taking action, as well as expanded our network and visual art team to provide services for Stage and Virtual Productions using LED and projection walls and exploring the new stories for TV and Film. 

Since 2015, with the expansion of Virtual Reality to larger audiences, we’ve begun producing immersive VR experiences for external parties and services. Starting to work with our partners at PlayStation, Epic Games, and Oculus, we launched our first award-winning game Ionia on September 2021, with the follow-up Eolia set for launch in 2022. Our Art Department has worked with notable companies in the gaming field like Meta, Unity, Skymap Games, and Tigertron to provide high-fidelity art resources for their projects to push the envelope further.

The Team

Our team comes from a variety of backgrounds ranging from Game Developers who worked on the early Devil May Cry titles at Capcom to Virtual Reality Artists. Our ever-evolving network of artists, entrepreneurs, and industry leaders, based worldwide, provides us with unparalleled access to talent, creativity, and innovation.

  • Director – Emir Cerman
  • Producer – Jason Parks
  • Creature Design – Neville Page
  • Modeling/Texturing/Animation/Photogrammetry – Emir Cerman
  • Rigging & UE5 Control Rig – Micheal Hoag
  • Statue Art – Charles Logan
  • Sound Design – Emir Cerman, Simone Scazzocchio, Cesar Suarez
  • Voice – The Famous Uncle 
  • Sound Mixing – Simone Scazzocchio

The Luminescent Project

I wanted to create a very realistic vlog experience. Someone got lost in the middle of a mysterious island cave and decided to capture his reactions to the environment with his phone.

In this instance, I relied heavily on photogrammetry, I used Sony A7RII to capture some of the natural rocks and waterfalls, as well as the Megascans library for the foliage/cliffs. I think the secret is always hidden in nature, the best method to mimic reality is to study reality very closely. How the lighting hits the objects, where the lights are bouncing, and how waterfalls look in the distance are one of a few major ways to activate photorealism. And being able to achieve this in real-time is so exciting. I think with Nanite, Lumen, and being able to animate in real-time, we are just scratching the surface of how we tell stories and create games going forward.

The other element is understanding cinematography, what type of camera or lens to use and why. In order to mimic a phone camera, I decided to use the Unreal Engine's Live Link VCAM app on my iPhone. This allowed me to get the freedom and the natural camera movement data I needed. 

95 percent of everything captured and rendered in Unreal Engine 5. After that, I did minimal editing in DaVinci Resolve with a detailed sound design.

The Hexa Creature

One of the key elements of the short was the beast of the forest, and we knew we had to get that right within our means. The creature in Luminescent is Hexa, an original creation of Neville Page (Star Trek, Avatar), famous in the science fiction field, it's a unique and complex creature. It was such an honor to collaborate with Neville on this project. 

One major technical challenge was rigging and animating the creature. That’s where ROTU Technical Director Michael Hoag stepped in to take on this process. While developing offline animation rigs in content authoring software such as 3ds Max or Maya is a more traditional pipeline for this kind of work, the team dedicated themselves to creating a real-time control rig directly in UE5 for our Hexa. Some of the unique features of this rig include nested IK solutions for the multi-boned legs and how the aim constraints were set up to drive the exterior piston-like muscle groups to assist the skin weighting.

Two major benefits of migrating to this new pipeline are the time saved from not going back and forth between applications and a dramatic improvement to ease of use such that multiple artists can make adjustments without relying on a single animator. In a traditional workflow, some amount of the environment, or proxies from it, need to be exported out into the application where the animation is authored so that the creature can realistically climb over its various surfaces. Animations are baked out and brought back into the engine.

However, if major adjustments are needed in either the environment or in the animation, you start to fall into a lengthy cycle of passing files back and forth. With a smartly designed control rig, 100% of the animation can be keyframed directly in Sequencer. And this can also be done by any artist, even the one without strong skills in animation, in a base authoring app. This allows for changes in pose and timing to be handled in real-time, speeding up rapid prototyping.

Challenges

One of the biggest challenges is always time and speed. Luminescent was created in a week, and the reason it was so quick is that in the past two years, we made digital tools and photorealistic digital movie sets/game-ready digital world catalogs in UE5, turning iteration time from weeks to days. But I would say optimization is always the challenge, especially while working with photorealistic large data now.

When we started working on VR games and trailers, we only had Unreal Engine 4 and already pushed the limits of what we could achieve with digital realism. When Unreal Engine 5 with the Lumen and Nanite technology was announced, we knew our potential was limitless, We want to push virtual boundaries, not just in games but the film as well, inspiring and helping creators bring imaginative words to life.

Conclusion

At the beginning of the pandemic, the use of virtual production in entertainment proliferated. This fascinating new technology requires photorealistic digital environments to be optimized for large LED panels. We saw an opportunity to utilize VR optimization experience for this new era of filmmaking. Partnering with VFX titans Neville Page, Robert Rodriguez, Ian Hunter, and AJ Wedding gave ROTU the technical opportunity to work with the cutting-edge screens at Orbital Virtual Studios, helping to define the future of storytelling.

Luminescent acts as a technical demo showcasing the work of ROTU's art department, giving a sneak peek of its capabilities to utilize real-time production techniques. We are always open for work opportunities and talent to join us.

You can find out more about ROTU on our website.

Emir Cerman, Chief Creative Officer at ROTU Entertainment

Join discussion

Comments 0

    You might also like

    We need your consent

    We use cookies on this website to make your browsing experience better. By using the site you agree to our use of cookies.Learn more