Using Machine Learning for Terrain Creation

Introduction

My name is Francisco Múrias and I’m a Procedural Tech Artist from Porto, Portugal. I studied Computer Science at the University of Porto, and recently undertook my masters in 3D Games Art at Teesside University where I specialized in Technical Art. My work focuses on proceduralism and automating ways to produce content for games and 3D. Recently I’ve been exploring Houdini pipelines and the potential of machine learning in content generation.

Ever since I was a kid, I have had a fascination for computer technologies and videogames. My journey into the world of 3D Art started at 14 when I did a summer course on 3D modeling with 3Ds Max. After following an art pathway in high school and furthering my knowledge of 3D, I realized I needed to learn programming to turn my art into interactive experiences.

I went on to do my Bachelor’s in Computer Science at the University of Porto (Portugal) where I learned the principles of programming and algorithms. Together with other students, I started Porto Graphics, a group focused on Game development and computer graphics where I had the opportunity to work together with other people during Game Jams and make my first games. I then was lucky enough to do my masters in 3D Games Art at Teesside University (England) where I learned tools like Houdini, Unreal Engine and had the chance to explore some applications of Machine Learning in videogames during my thesis.

My aim is to combine my programming experience with the knowledge of 3D Art, to produce innovative ways to create game environments.

Machine Learning in Game Industry

Machine Learning is broadly used to describe computer algorithms that improve automatically through experience. In practice, we use sample data to “train” a computer algorithm to get us a “mapping function” that resolves a set input into a useful output.

The applications for games and CG are broad and diverse: From reducing the time it takes to render and compute different graphical effects, solving complex physics models in realtime, creating complex AI behaviors, automating facial animations to figuring out how to create 3D content faster.
ML is at the forefront of what is being done in this industry today and artists and programmers keep finding new and creative uses for it.

The main advantage of ML for content creation is to empower artists to work smarter and express their creativity in less restrictive ways. You can see this starting to be a focus of some major players such as Adobe with their new Photoshop Neural Filters, and Unity with the recently acquired ArtEngine from Artomatix. Emphasis is on how using these new tools can save time and allow you to focus on doing what you already wanted to do in the first place.

ML for the Terrain Tool

This project was an exploration into the uses of machine learning for content generation for video games, namely in the case of terrain generation. The intent was to be able to generate detailed and realistically eroded terrains in real-time, based on only a simple user sketch.

Realistic terrain generation poses a particularly difficult problem as real terrains are subjected to a wide variety of geomorphological processes that influence their shape and features, such as thousands of years of exposure to various erosion agents, different ranges of temperatures, vegetation, wind, and water sources that are hard to algorithmically define as usable editing tools. Because of this, workflows for designing terrains are particularly challenging. In the past three decades, researchers have investigated a variety of approaches to generate terrains, all with varying advantages and disadvantages.

The main inspiration for my tool was research done by Éric Guérin and other authors from the University of Lyon and Purdue University with support from Ubisoft. Their research paved the ground for this type of application. The terrain has the particular advantage of being represented as a heightmap which is ideal for a Machine learning approach, as it's essentially just an array of grayscale pixels. Not only that, but we have the real world as an excellent source of terrain data, with the USGS having accurate terrain Height Maps for almost the entirety of the US landmass. All these variables made it the perfect use case for a machine learning application experiment.

Some examples of the tool in action:

Working with GAN

A Conditional Generative Adversarial Network model was created and trained from automatically processed real-world terrain data, gathered with a Houdini pipeline. A user interface tool was developed to sketch lines defining the intended terrain, run the neural network model and showcase the results in a real-time 3D preview. The pipeline can generate any number of realistic virtual terrains using the learned features from the training data, without requiring a manual definition of any procedural rules or parameter properties. I’ve used Quadspinner’s Gaea to amplify the resulting heightmap and show that it can be integrated into any existing terrain pipeline. The result is then exported to any game engine, in my case Unreal Engine, where I applied a smart terrain material that applies different material types based on the height and normal direction of the terrain.

The cGAN system consists of 2 competing Networks: a generator that continuously synthesizes new terrains, and a discriminator that compares the generated ones with the real examples. The two networks compete with each other learning how to produce terrains that better match the training examples.

To gather the training data used to train the models, I used a Houdini pipeline to automate this task using the Procedural Dependency Graph.

The pipeline follows a specific set of steps: 

• Read intended coordinates
• Download Files from USGS servers
• Decompress files
• Split height information into smaller more manageable sections
• Process each tile to compute the Peak and Valley Lines
• Map result to matching images and save

The entire data gathering pipeline was based on the ML Data Preparation example from SideFX, which features an excellent approach to this and serves to teach how to use the PDG system.

Creating UI for the Tool

I built a custom tool to allow users the ability to sketch peak and valley lines that define their artistic intention. The machine learning algorithm automatically generates the resulting terrain in real-time, and you can switch between any of the trained models. The results are automatically exported on each change, so this could be integrated with other tools or pipelines.

A custom 3D previewer developed using ModernGL allows rapid iteration and visualization of the result.
This was my first major Python application, and I found that Python is an excellent language to start and build this kind of custom application. It has an excellent community and an active Discord server full of people willing to help you get started.

The terrain is then amplified using Gaea, and the resulting terrains are ready to be imported to any game engine and used as the basis for a 3D landscape.

Challenges

I believe this tool demonstrates that there is immense potential for ML applications for content creation in games and we are seeing that more and more. An intended goal for the future development of this project is to generalize the terrain generation of different features in the same network and to differentiate between them with different types of sketches in the authoring phase.

I want to continue working on ML applications and have several ideas on how to expand, not only this tool but also different uses of ML for games.

Francisco Múrias, Procedural Tech Artist

Interview conducted by Ellie Harisova