How ABCD Trains Large Gaussian Splat Scenes With Constant VRAM

Photogrammetry and neural rendering techniques are making it possible to reconstruct increasingly extensive real-world environments, but the size of those scenes can quickly exceed the memory available on even powerful GPUs. This became a practical obstacle for Research Scientist Ka Heng Shiu while working on The Lost Metropolis, an art and heritage project dedicated to archiving disappearing urban spaces in Hong Kong.

Shiu’s scans included entire villages and buildings with complete interiors and exteriors. When he attempted to reconstruct these environments using 3D Gaussian Splatting, he repeatedly ran out of GPU memory. That problem eventually inspired ABCD: Alpha-Composited Block Coordinate Descent, an out-of-core training framework developed with Kartic Subr and recently presented at SIGGRAPH.

ABCD divides a scene into spatial partitions and optimizes one partition at a time. Unlike approaches that train several isolated regions before merging them, ABCD keeps the remainder of the scene visually present through cached foreground and background RGBA images. This allows the active region to be trained within the context of the complete environment while keeping peak VRAM effectively independent of the scene’s total size. We spoke with Shiu about how ABCD works, its current RAM and training-time tradeoffs, and its potential applications in city-scale capture, game development, digital twins, and distributed rendering workflows.

First, can you please tell me a bit about you, your research, and your background as an introduction?

Ka Heng Shiu, Research Scientist: Hi I'm Heng! I’m a research scientist with a background in neural graphics, although I try not to stay too tightly inside one box. I’ve also worked on generative 3D and LLMs, and I generally enjoy taking ideas that are theoretically interesting and finding practical problems where they become useful.

ABCD is a good example of that. It actually began with an art and heritage project called The Lost Metropolis, where I was archiving disappearing urban spaces in Hong Kong. I was capturing photogrammetry scans of very large environments, including entire villages and buildings with both interiors and exteriors. I then explored whether those scans could be reconstructed as Gaussian splats.

The problem was simple: I kept running out of GPU memory.
I started investigating ways around that limitation, eventually turned the problem into my bachelor’s dissertation, and then shelved the project for about a year and a half while I was busy finding work. I later came back to it, developed the idea further, and recently published the resulting work at SIGGRAPH.

Could you explain ABCD’s block coordinate descent approach in accessible terms? How does training one spatial partition at a time differ from independently training several smaller scenes and merging them afterward?

Ka Heng Shiu: The obvious way to train a scene that is too large for GPU memory is to divide it into smaller regions, train each one separately, and merge them afterward. The problem is that each region is then optimized without knowing what the rest of the scene looks like.

Imagine splitting a room into left and right halves. While you are training the left half, some camera images still contain objects on the right. If the optimizer only has Gaussians from the left half available, it may incorrectly place those Gaussians in an attempt to explain what it sees on the other side of the image. When the independently trained halves are merged, these mistakes can appear as floating geometry, seams, or other inconsistencies.

ABCD instead trains one spatial chunk at a time while keeping the rest of the scene visible but frozen.

For each camera, the inactive Gaussians are pre-rendered into a foreground image and a background image. The active chunk is rendered between those two layers, and all three are alpha-composited together.

The important part is that only the active chunk needs to remain in GPU memory. The rest of the scene has effectively been collapsed into images whose size does not depend on how many Gaussians they represent.

That lets us optimize a very large scene one memory-bounded chunk at a time without treating those chunks as completely independent scenes.

A central innovation is collapsing the inactive parts of the scene into foreground and background RGBA images. How does alpha compositing allow the active partition to be optimized within the context of the complete scene while avoiding the seams and consistency errors common to partitioned training?

Ka Heng Shiu: The key idea is that removing something from GPU memory does not mean removing it from the rendered image.

The inactive parts of the scene are frozen, but they are still present visually through the pre-rendered foreground and background layers. When the active Gaussians are rendered between those layers, the final image still represents the complete scene.

So the optimizer is not learning a chunk in isolation. It still sees how that chunk should occlude, blend with, and sit behind or in front of the surrounding geometry.

That context is what prevents many of the inconsistencies you get from independently trained partitions. Instead of trying to stitch several separately optimized scenes together afterward, ABCD preserves their relationship throughout training.

ABCD greatly reduces GPU residency but currently increases system-memory usage and training time. How do you evaluate that tradeoff, and what optimizations could make the system faster and less dependent on a large cache of pre-rendered images?

Ka Heng Shiu: The current implementation essentially trades scarce GPU memory for cheaper resources: system RAM, disk storage, and additional training time.

VRAM is mainly determined by the size of the active chunk and the rendering resolution. RAM is used for cached images belonging to cameras that may observe that chunk, while disk stores a larger set of pre-rendered camera and chunk combinations.

As the overall scene becomes larger, the important property is that peak VRAM does not have to grow with it. You can keep the active chunk at a fixed size.

There is still a lot of room to improve the other costs. Our current visibility test is fairly conservative. It checks whether a chunk lies inside a camera frustum, but not whether that chunk is actually hidden behind other geometry. Occlusion-aware culling could eliminate many unnecessary cached views.

That becomes especially interesting for very large environments. Visibility is often local. A camera standing on one street cannot see most of a city. If we can exploit that property well, RAM usage could potentially remain much closer to a local working set rather than growing with total scene size.

The cache is also currently stored using lossless PNGs. More aggressive image compression, better caching strategies, and reducing how often partitions need to be refreshed could substantially reduce storage and training overhead. These are promising directions, although we have not evaluated them yet.

Your current experiments use relatively small scenes, where fixed overheads partly obscure the method’s expected memory advantage. What scale of real-world dataset would best demonstrate ABCD’s potential, and what new problems do you anticipate when moving to city-scale or similarly extensive captures?

Ka Heng Shiu: Testing ABCD on genuinely city-scale datasets would make its memory-scaling advantage much more visually obvious, and that is ultimately the kind of problem the method was motivated by.

At the same time, the experiments in the paper were deliberately a different kind of stress test. Mip-NeRF 360 scenes are relatively compact but difficult to divide cleanly. We split some scenes into almost 30 spatial chunks, which creates many opportunities for partitioning errors.

Despite that, the final reconstructions showed very little visible partitioning artifact and converged similarly to the baseline Gaussian splatting model.

The method can also already be useful at much smaller scales. If a scene barely exceeds the memory capacity of your GPU, simply dividing it into two chunks can roughly halve the Gaussian-related VRAM requirement.

For city-block-scale reconstruction, I expect the bigger challenge to become computation rather than VRAM. As the number of partitions grows, repeatedly switching between them, updating cached context, and revisiting chunks becomes increasingly expensive.

So at very large scales, the central problem shifts from “How do we fit this model into memory?” to “How do we schedule and update all of these local regions efficiently?”

Although the current implementation targets 3D Gaussian Splatting, you describe ABCD as a broader framework for alpha-composited radiance fields. Could it eventually support other representations, distributed training, or production workflows for scanning large game environments, digital twins, and virtual production sets?

Ka Heng Shiu: Yes. The core idea of ABCD is not inherently tied to 3D Gaussian Splatting.

What it needs is a representation that can be spatially partitioned and whose contributions can be recombined through alpha compositing. In principle, that could include NeRF-like representations, triangle-based splatting, or other radiance-field approaches.

One direction I find particularly interesting is distributed training.

The current implementation uses one GPU and moves through the chunks sequentially, but conceptually different chunks could be optimized by different workers. Because each worker spends most of its time working locally on one region, the system may not require the kind of constant high-bandwidth synchronization used by tightly coupled distributed training.

That could make heterogeneous hardware particularly useful. Instead of requiring a cluster of identical high-end GPUs, you could imagine assigning chunks according to the memory and compute available on different machines.

At a larger scale, this suggests a model where workers can be added or removed dynamically, potentially using a mix of local machines and cloud instances.

That is still a future research direction rather than something demonstrated by the current implementation, but I think it is one of the most exciting consequences of treating large-scene reconstruction as a collection of coordinated local optimization problems rather than one monolithic model.

Rather than asking, “What is the largest GPU that can fit this scene?”, the longer-term question becomes, “How can we use whatever compute is available to train the scene together?”

Ka Heng Shiu, Computer Graphics Research Scientist

You can read more about the work in the full research paper or check out the code on GitHub for yourself.