Making Cyberpunk Transmedia Series NEOSHIN

NEOSHIN – A Cyberpunk Transmedia Series

It is the year 2073X. After the invention of Cryonic Reality (CR) in 2073, humanity decided to stop counting time. As long as one stays in the CR, the aging process of the human body is stopped, while the frozen person has access to a virtual paradise in which social life takes place.

NEOSHIN is a broad transmedia project: the core of NEOSHIN is a narrative series that revolves around the eponymous band and the protagonist Ayuko. Here, the band NEOSHIN is not only a character within the series but also a real band, which will perform live as well. Instead of a traditional intro, each episode will begin with a music video of the band summarizing the events of the previous episode. This focus allows elements of a series to be combined with social media, live music, and virtual content.

First and foremost, NEOSHIN deals with the issue of a glass consumer. It is about the drastic exaggeration of everyday social conventions. The aim is to give an insight into a world in which hedonism, self-expression, and mass consumption have become more important than compassion, humanity, and sustainability. 

The music of the band NEOSHIN is an overarching, important part of the whole project.

Supporting this, there will be the protagonist Ayuko – a Virtual Influencer on today's social media platforms, where she posts her daily life in the year 2073X providing viewers with additional lore from the world in 2073X.

Another aspect of our concept is merchandising, which will feature AR functionality. Each design will have a unique camera filter associated with it, which can then be used for selfies and shared on social media. This is intended to motivate the culture of collecting.

Band & Music

The goal of the music is to harmonize the vision of a dystopian neo-Tokyo future cosmos with our musical preferences in terms of style and modern sound design. 

The initial situation was not the best for this purpose as there was no vocalist for the project. So in this case it was essential to – literally – give the music a voice that does not exist in the real world. That's why our musician and scriptwriter Roman Schneider decided to work with sampling – for example, snippets from text-to-speech audio programs or Vocaloid software singers, which were quite popular in Japan for many years. All these snippets of speech are then run through a vocoder. 

Especially challenging, in the beginning, was the intelligibility of the text. But the resulting sound should prove to be extremely profitable for the overall project because vocoder voices sound very digital and thus perfectly fit the band and their appearance as a digital virus in the story. It also contrasts quite harshly with the lyrics, which are very personal and also vulnerable and thus human. 

In terms of instruments, we combine synths and EDM-like beats with low-tuned distorted guitars. We had the latter made especially for the project by Aviator Guitars. In addition to top-notch musical components, both guitars were given a chameleon finish that changes color depending on how the light hits them.

The band wears specifically designed masks that can play motion graphics and videos in real-time. In cooperation with interactive media mastermind Jan Fiess, we created our own circuit boards, which were equipped with LEDs. Everything is controlled via Touchdesigner which also allows the "live" playback of content. The connection to light systems, Unreal Engine, and external LED screens allows real-time VJ-ing during the concert – not only on the masks but on the whole stage set. 

The World In The Year 2073X

At the beginning of each location for the first music video "Episode 1: Cold Blood", there was a detailed concept process.

We used mood boards to get the basic mood and style of each location before we created simple 3D blockouts and concepts. These often consisted of either the original blockouts, which were lit and rendered in Unreal Engine, or overpaints of these initial 3D elements.

When finding the visual style, we put a lot of emphasis on the purpose of each location within the world and story. Since we didn't have much time for exposition in the first music video with just 3:30 minutes, it was also important to have a clear visual difference between locations in the real and virtual CR world. 

The environments range from large gloomy skylines to clean futuristic-looking interiors to abstract but inviting locations in the virtual world. 

Virtual Production

In order to integrate the performance of the real band into the digital backgrounds of a CR nightclub, we decided to use the Virtual Production technique, which was very new at the time. Here, large LED screens are used in combination with a game engine – in our case, the Unreal Engine – and a camera tracked with a mocap system to display the backgrounds with perspective-correct parallax.

The enormous advantage of this technique is the time it saves compared to conventional workflows using a green screen. In particular, it eliminated the need for light interaction on reflective, metallic surfaces – such as the band's masks or drum cymbals – as well as the integration of backgrounds in compositing. Furthermore, it gave the camera crew the opportunity to get a better feel for the final image directly on-set and thus also the chance to adjust various things such as brightness, colors, and set dressing in real-time.

For this shoot, we cooperated with the company LEDCave in Mannheim, which could offer us an almost 360-degree virtual production studio.

We shot with a Sony Venice, which was tracked using both an Ncam and an Optitrack motion capture system. However, since the scenes shot in the studio were very dark and the elements in front of the camera were extremely reflective, the Ncam, unfortunately, did not provide stable tracking, which is why we ultimately decided to use the Optitrack system. 

At the time of the shooting, there was virtually no out-of-the-box solution for external camera tracking in engines, so our technical director Ramon Schauer and the two RnD staff members from the animation institute Simon Spielmann and Jonas Trottnow had some technical problems to solve in the beginning. Overall, however, this approach was definitely worth it due to the great time savings and the good quality of the compositing.

To ensure the smoothest possible workflow, we used a blueprint in Unreal Engine adapted to the shoot to synchronize both the music and the motion graphics within the environment and on the LED masks via TouchDesigner thus also giving the camera crew the option of repeating takes at any time without major changeover times.

Layout

Since we had already set up a large part of the locations for the previz, the concepts, and especially the band scenes shot using Virtual Production in Unreal Engine, we decided to use UE as the central part of our layout workflow. For this purpose, we created a custom plug-in that allowed us to exchange scene layouts between Unreal and Houdini. 

In Unreal Engine, for each mesh used, various data such as transforms, material assignments, and a mesh ID are stored as a .json file. These are then used in Houdini to scatter a set of points with these transforms and instantiate the corresponding meshes from the library on them. The instantiated assets can be either the same as in the Unreal layout or higher resolution meshes as long as the pivots remain the same.

This simple exchange allowed us to use a lot of the layout directly as the final render layout and use the same work for both the virtual production shoot and the previz and final CG scenes. This workflow formed the basis for the layout process and was then augmented by Houdini's procedural capabilities.

For our director and "digital cinematographer" Sebastian Selg, real-time blocking and camera animation within Unreal were essential. This also resulted in significantly more "happy accidents" than in a jerky gray offline viewport. The possibility to see materials, light, fog, glow but also lens flares and many animations in real-time lets you try out a lot more without having to wait for a render.

Lighting

The lighting scenes in Houdini were then set up for a multi-shot workflow, which meant that for each location there was only one lighting scene that contained the light for all shots in that sequence allowing us to switch between shot-specific caches, cameras, and render setups. This saved us a lot of time due to the high number of very short shots. Houdini's Render ROP workflow was very helpful in keeping track of 50+ shots in the same scene. We rendered using Arnold Renderer, which is known for its stability and industry-proven performance. All lights were then split into render layers and later leveled out in Nuke. 

The asset creation itself was left open to each artist. Cinema4D, Maya, ZBrush. and Substance 3D Painter were primarily used. Due to a large number of assets, we tried to split each scene into modular, reusable parts as much as possible.

Ayuko – Photogrammetry, Texturing, and Rigging

The protagonist Ayuko was realized as a digital double of Delia Böttcher. The advantage of this approach was that, besides the possibility to use 3D scanning, enough reference material could be collected to make the digital asset as close as possible to the real template.

An unexpected hurdle here was the creation of a contract that regulates the rights to a scanned digital person as the current legal situation in Germany is very complex and absolutely undeveloped.

For the photogrammetry scanning, we cooperated with the NC3D company from Frankfurt, which made its studio available to us. This studio consists of an array of (at that time) up to 80 DSLR cameras, which can all be triggered simultaneously.

Furthermore, the studio offered neutral lighting conditions, which allowed us to use the scanned textures without any problems. This setup is especially important when scanning humans and animals as the human body is always in slight motion due to breathing, which would lead to unwanted artifacts in the final scan.

In this scanning session, we scanned not only the body and face but also a total of nearly 80 facial expressions, which were based on the Facial Action Coding System (FACS for short). The FACS system divides human facial expressions into different, isolated muscle groups and allows us to map almost the entire spectrum of possible facial movements by combining different FACS expressions.

This allowed us to reproduce the characteristic facial expressions in our face rig in as much detail as possible. In addition, the scanned albedo textures were used to represent the shader aspects, such as blood flow under the skin as well as minute wrinkles that would be too small for the mesh itself. To do this, we used masks to divide the face into different regions, which then fade in the textures of the active region controlled by the values of the individual blendshapes.

We also used this session to record detailed references. We took great care to document everything accurately. In order to be able to adjust the CG version as close as possible to the reference, it was also necessary to reconstruct the lighting conditions on the set. For this, we shot HDRIs of the studio and a big octabox. We also documented all lens data as well as distance measurements of camera and light. Using this data, we then projected the HDRIs onto scale geometry to create our character lookdev scene. 

The scans themselves were then created using Reality Capture and prepared in ZBrush. Wrap3 proved to be particularly helpful in the scan preparation: the tool allows existing topology to be morphed from one mesh to another. This was essential to get all scanned facial expressions onto the same topology to be able to use them as a blendshape in the face rig. Furthermore, we use Wrap3 to be able to bake the scanned textures onto our own UV set, which is standardized between all characters, and also to be able to share and reuse textures and blendshapes between different characters. The optical flow wrapping feature was particularly helpful here.

Additional details, such as skin pores, were then added using both manual sculpting and via displacement textures from TexturingXYZ, which were projected into Mari.

To save some time in the rigging process, we decided to use the Advanced Skeleton auto-rigger for the base rig. On the one hand, this already offers quite a few features, on the other hand, it also allows the use of the Advanced Skeleton toolset, which offers many useful features, such as a MoCap Matcher or the ability to beacon between IK and FK. To further enhance this rig, we used numerous Corrective Blendshapes, which were both obtained from the scans and manually sculpted.

The face rig, on the other hand, was set up completely manually and also kept in a separate file, which was referenced into the main rig using a blendshape. This gave us the opportunity to update the base rig and the face rig separately and to work on both aspects in parallel.

Technically, the face rig is based on the book Stop Staring by Jason Osipa but extends it slightly with some additional FACS shapes.

Here, the technical setup was quite simple, the main work was to create good, anatomically correct shapes, which also work well in combination with each other. Additionally, some more prominent combinations got some more combination shapes, which were faded in.

Grooming

Grooming the character was implemented in Houdini using the Groombear plug-in, which provides a relatively interactive workflow. However, partly due to a lack of experience and the complexity of Houdini, the grooming step was one of the most challenging aspects of the character. 

In addition to the hairstyle itself, we used other hair systems for the eyelashes and peach fuzz. The peach fuzz in particular, despite its subtlety, has a huge impact on the overall look of the character, providing softer light edges and blending all the elements further into one another.

Clothes

For the clothes, we primarily used Marvelous Designer. We created different versions for each outfit. The first mesh was quite detailed and then further elaborated, thickness added and remeshed in ZBrush. This version then served as the final, textured render geometry.

For the simulation itself, there was the second mesh in Marvelous Designer, which was kept much simpler and did not include details such as buttons, zippers, or smaller pockets. This had the advantage that it was much faster and also more stable to simulate. This simulated version was then exported as Alembic and brought into Houdini together with the render mesh. There, using IDs obtained from the individual Marvelous Designer patterns, the high poly rendermesh was wrapped to the simulated low poly mesh by a Point Deform to obtain the final simulation cache.

Character Simulation

To remove some of the stiffness that often arises from CG characters, the final step was to apply a fat and jiggle pass to the character using the Mush3D software. Mush3D allows you to sculpt on top of already cached alembic sequences and use various deformers such as Cloth and Jiggle. This effect was subtle but still added realism. While a true muscle or fat simulation using Houdini or Ziva would have been much more accurate here, considering the speed and simplicity of Mush3D, this approach paid off for us.

Motion Capture

Motion capture was an essential part of the project. In total, we did two different mocap shoots. The first shoot, which took place early in the project, was primarily about collecting animations for previz to work out the animatic. We imported these animations directly into Unreal Engine to quickly create blocking and cameras. The intent was not to get particularly beautiful animations but to test ideas and complete the edit of the video, on which the rest of the production can build.

The second shoot, on the other hand, was much more involved and spanned several days. During this shoot, Ayuko's and the extras' acting and the entire music performance of the band (including tracked instruments) were recorded.

This was important because we had to switch to the CG version of the band in various wide shots, and they had to fit seamlessly with the shots from the LEDcave.

In addition to the body, this time we also captured the facial expressions. For this, we used a GoPro camera mounted on a special helmet and the Dynamixyz software (which has since been sold to Take-Two Interactive and is therefore no longer available to the public). This can be used to train a so-called Actor Profile based on the GoPro footage, which makes it possible to retarget even long video sequences to the rig without additional work after the training process is complete and the face rig is connected.

Even though Dynamixyz does not require tracking markers on the face for retargeting, we decided to still apply markers to the face of our actress and animation supervisor Bea Höller so that we could better estimate the volume of the face movement and compare it to the rendered videos. To do this, we created a plaster mask so that we could apply the dots to the same anatomically important areas each day.

For this shoot, we used the Optitrack Mocap system with a total of 24 cameras. Since we had no real sets, we had to be creative with the requirements. For example, one scene was shot underwater using a rope rig that was actually designed for stunts, while simple markers on the ground had to act as walls.

Although the data captured in this way was a good starting point, more manual work was needed to get the most quality out of the mocap animations and also to allow a consistent character to shine through. This was done by some great animators under the direction of Bea Höller and resulted in a significant improvement in quality over the raw mocap data. 

Compositing

All renders were brought together in Nuke, the lights were split into different AOVs. In the more complex scenes, we split the light passes into directs and indirects to have full control in comp. With the help of compositing artist Lukas Löffler, a number of useful tools were developed to automate comp layouts, precomps, previews, and master exports. The BananaSplit tool, for example, creates a standardized comp layout in which all passes and highlights are split with a push of a button and the albedo pass is offset against the highlights in such a way that texture recoloring could still be done comfortably in Nuke without changing the raw lighting. Conversely, it was also possible to recolor highlights without recoloring the texture underneath. Due to the often long render times, this was an absolute must for NEOSHIN. It was also advantageous that each compositor roughly knew how the other's script was structured. 

Lukas Löffler also contributed a custom Nuke script that modified Nuke's ZDefocus to make the bokeh narrower and more turned in towards the edges to create the CatsEyeDefocus effect.

Because of the decision to do Depth of Field in comp, we saved render time but also had to set up the scenes in a way that didn't create unsightly edges on hair and transparent objects. After a lot of trial and error, the Deep workflow turned out to be the best in terms of quality. However, Deep rendering has a disadvantage: it produces extreme file sizes and thus slow comps. Since we were only interested in the possibility to create holdouts and especially the z-depth pass, it was sufficient to output a separate 32-bit float AOV in Deep instead of the full RGBA. This was then never much larger than 20MB per frame and gave us everything we needed.

Because of the small core team and the many shots, it was very important to know when 3D scenes were ready to render and what we could still "fudge" in comp. For example, we have two locations where the camera is looking through wet glass panes with drops running down. Since our 3D skills lay more in the area of environment and character, none of our own waterdrop simsetups could convince us. This is where Nuke came in: projecting filmed water elements onto the geo of the panes and displacing them via IDistort not only saved time compared to simulating but was also easier to control and, in our case, looked more realistic.

Current Status And Plans For The Future

A few months ago, we finished the first music video, which is currently in the festival evaluation and will be published online soon.

In addition to this first music video, which serves as a proof of concept, we have also developed a 50+ page world and the story bible, which includes world rules, lore, character descriptions, and treatments for the 12 episodes of the first season.

Until we have the budget or a suitable partner for the realization of the series, we will release more music videos. In addition, the band NEOSHIN will perform live and play real concerts as soon as the situation allows. Two singles are already finished, the full album is in development.

If you are interested, you can learn more about the project here, on Twitter, or Instagram. Additionally, there are already some making-ofs, which can be found on NEOSHIN's YouTube channel.

The Core Team

NEOSHIN was realized as a diploma project at the Filmakademie Baden-Württemberg by the core team consisting of four people.

However, the work of all the additional helpers should not be forgotten: from the great film crew on set to all the animators who brought our film to life, all the people who contributed various assets, and the lecturers and mentors who supported us during our studies. All in all, NEOSHIN's first sign of life in the form of "Episode 1: Cold Blood" was a gigantic team effort involving over 60 people.

Sebastian Selg – Director and part of the band NEOSHIN

Sebastian Selg is a 3D artist and camera enthusiast with a film, music, and fashion focus. After graduating from the Filmakademie Baden-Württemberg in animation in 2021, he is currently working as a freelancer.

Ramon Schauer – Co-Director/Technical Director

Ramon Schauer is a 3D Artist with a focus on VFX and games. In early 2018, he completed his bachelor studies in Animation & Game at the University of Applied Sciences in Darmstadt.

Meanwhile, he worked as a 3D Environment Artist at the Frankfurt-based game developer Deck13. Afterward, he started his postgraduate studies in Technical Directing at the Animation Institute of the Filmakademie Baden-Württemberg, which he completed in 2021.
He is currently working as a 3D Generalist in the VFX department of Mackevision in Stuttgart.

Jiayan Chen – Producer

Jiayan Chen works as an animation producer. In 2013, she completed her first Bachelor's degree in Journalism and Communication in China. In 2019, she followed up with a bachelor's degree in Audiovisual Media (focus on 3D animation) at Stuttgart Media University. During that time, she worked as a 3D Generalist on various animation projects. From 2019 to 2021 she studied Animation and Effects Producing at the Filmakademie Baden-Württemberg.

Roman Schneider – script, music, and part of the band NEOSHIN

Roman Schneider is a composer, currently, a research assistant, and doctoral candidate in media criminal law on the topic of modern forms of representation at the Eberhard Karls University of Tübingen. His research interests include the topic of protagonist representation of NEOSHIN (photogrammetry, AI, and ML systems). Before studying law, he was a drummer in the German Armed Forces (Army Music Corps 10 and 12) and studied music business at the Pop Academy in Mannheim.

NEOSHIN 2073X, Musical Project