EchoForge Uses Spatial Sound to Build 3D Worlds in Unity

Editor's Note: Most 3D scene-generation systems begin with an explicit visual description, whether it is provided through text, reference images, or sketches. EchoForge explores a markedly different starting point: the sounds that already suggest an environment to the listener.

Developed by ESGI students Anamaria Buliga and Kamil Tebani, EchoForge analyzes spatial recordings to recognize sound events, estimate their approximate directions, and infer the broader environment they may describe. It organizes this evidence into an interpretable audio scene graph before producing layout instructions for a procedural 3D environment in Unity. The project was presented at SIGGRAPH 2026 and received first place in the undergraduate category of the ACM Student Research Competition.

Introduction

My name is Anamaria Buliga, and I am a Master’s student at ESGI in Paris, specializing in Computer Graphics, XR, and Interactive Systems. My work is centered around real-time 3D and immersive technologies, with a particular interest in procedural generation, simulation, spatial computing, and generative AI for 3D content. I enjoy building systems that connect technical research with interactive visual experiences, especially when they involve new ways of generating or understanding 3D environments. I currently work in R&D around XR, immersive 3D generation, and the evaluation of generative 3D methods.

I worked on EchoForge with Kamil Tebani, who is also studying 3D/VR engineering at ESGI in Paris. Before moving into engineering and real-time 3D, he spent two years studying Performing Arts, with a focus on theatre and cinema, at Université Paris Nanterre. His work now sits somewhere between research, engineering, and technical art, with a particular interest in turning ideas that can initially feel theoretical into systems that can actually be experienced inside a real-time engine.

EchoForge sits at the intersection of spatial audio, procedural world generation, and interpretable AI systems. The project was presented at SIGGRAPH 2026 Posters and received 1st place in the ACM Student Research Competition Undergraduate category.

EchoForge Inspiration

I have always had a very visual way of experiencing sound. When I listen to music or environmental audio, I naturally imagine scenes, places, movements, and visual elements associated with what I am hearing. Sound often creates a mental scene almost immediately.

That personal way of perceiving audio became part of the motivation behind EchoForge. It raised the question of whether a system could also use sound itself as the starting point for constructing a visual interpretation of an environment.

Sound already contains a kind of “hidden scene description.” When we hear ducks, wind, waves, birds, rustling leaves, or traffic, we naturally begin to imagine a place. Unlike text or sketches, sound also carries temporal and spatial information: something may be louder on one side, move over time, feel diffuse, or suggest distance and enclosure.

Most scene-generation tools begin from explicit visual or textual intent. With EchoForge, we wanted to explore whether the recording itself could become the starting point for generating a 3D scene.

The goal is not exact reconstruction, because audio alone cannot reveal the precise geometry of an environment. Instead, EchoForge creates an interpretable hypothesis of the scene: which sound sources are likely present, what kind of environment they suggest, where they may be located, and how confident the system is in those decisions.

How EchoForge Works

EchoForge starts with a spatial audio recording. The first part of the pipeline analyzes spatial cues in the recording to estimate coarse source directions, active time segments, and confidence values.

In parallel, YAMNet is used for semantic sound-event recognition through AudioSet classes. This gives us labels corresponding to events such as birds, ducks, insects, wind, rustling vegetation, or water.
The semantic and spatial information is then fused. Rather than treating every classifier label independently, related detections are grouped into higher-level semantic families such as forest, water, wind, bird vocalization, waterfowl, or insect ambience.

The central representation is what we call the audio scene graph. It contains the inferred environment, candidate sound sources, confidence values, spatial evidence, temporal direction information, relationships, and consistency checks.

This representation is important because it keeps the generation process interpretable. A visual decision can be traced back either to evidence found in the recording or to an explicit procedural rule.
Finally, the scene graph is converted into Unity-readable layout instructions. These determine elements such as the terrain, vegetation clusters, water regions, source zones, environmental ambience, and representative objects. The resulting scene can then be instantiated and explored in Unity.

One important distinction is that EchoForge does not claim to recover an object's exact position or physical scale. Instead, it estimates a plausible procedural placement from the available evidence.

Direction comes from the spatial information in the recording. When a semantic detection is repeatedly associated with a consistent direction, the system can turn that into a corresponding source zone in the generated environment.

Scale and visual representation are then determined by procedural rules rather than direct acoustic measurement. A bird vocalization, for example, might activate trees and an elevated perch region, while a duck or quacking sound might produce a ground-level waterfowl zone. Wind is treated as environmental ambience rather than as a localized object.

The global context is inferred by combining several detections. Wind, rustling leaves, birds, and other outdoor sounds can collectively support a forest-like environment, while waterfowl-related sounds can support a different scene configuration.

We deliberately designed this conservatively: strong and spatially consistent evidence can create localized source zones, while weak, diffuse, or ambiguous evidence remains part of the environmental context rather than being forced into a specific object.

Future EchoForge Plans

One of the main challenges is that different types of sounds require different kinds of representation. EchoForge already distinguishes between localized sources and more diffuse environmental sounds. Wind, for example, can influence environmental elements such as vegetation and ambience rather than being represented as a single object.

Future versions could extend this idea to more complex sounds. Crowds could be represented as density regions or groups of entities rather than individual sound sources, while reverberation could provide additional cues about whether an environment is open or enclosed and potentially about its scale.

Another important direction is improving sound-event localization. EchoForge currently combines semantic recognition with relatively coarse spatial cues. In future versions, we would like to explore SELD-style approaches that jointly perform Sound Event Localization and Detection, making it easier to distinguish and spatially organize several simultaneous sound sources. Source separation could also help when multiple sounds strongly overlap in the same recording.

We also want to considerably expand the semantic and procedural vocabulary of the system. The cricket/dusk case highlighted an important limitation: recognizing a sound is not sufficient if the generation stage does not yet know how to represent it. Adding more sound families, assets, environmental behaviors, and generation rules would allow EchoForge to interpret a much wider variety of soundscapes.

EchoForge already analyzes directional evidence over time through temporal direction events. A further step would be to use changes in this directional information to infer whether a sound source itself is moving, and then translate that estimated movement into animated entities inside the generated scene. This would allow the environment to evolve dynamically with the sound rather than remaining primarily static.

Longer term, we are also interested in reducing the reliance on manually defined rules, exploring more data-assisted mappings between audio evidence and visual scene elements, and moving toward faster or eventually real-time scene generation.

Audio Scene Generation's Value

We currently see the strongest potential in creative and assistive tools rather than exact reconstruction.

For game developers and environment artists, a field recording could become a rapid starting point for procedural worldbuilding. A sound designer could provide an ambience recording and immediately obtain a rough, explorable visualization of the soundscape.

It could also be useful for technical artists working with spatial audio or XR, allowing them to visualize where sounds are detected and how an audio environment is interpreted by the system.

Another direction we find particularly interesting is accessibility. For deaf or hard-of-hearing users, a system derived from EchoForge could potentially translate elements of a soundscape into visual and spatial information. Instead of only indicating that a sound occurred, it could communicate what kind of sound was detected, roughly where it originated, and whether it represents a localized source or a more diffuse environmental event. In immersive environments, this could provide an alternative way of perceiving spatial audio information.

EchoForge could also have applications in education, soundscape visualization, or the exploration of recordings from environments that are difficult to access directly.

For reconstruction of real environments, however, we would be much more cautious. EchoForge is not intended as a forensic or metric reconstruction system. Its strength is instead in making the interpretation process visible: what the audio suggests, where the evidence is strong, and where the system is making an uncertain procedural choice.

Anamaria Buliga and Kamil Tebani, ESGI Computer Graphics Master's Students