spark/docs/system-design.md

System Design

One of the biggest challenges in real-time Gaussian splat (Gsplat) rendering is sorting the splats so that they can be drawn and blended in back-to-front order, known as Painter's algorithm.

Rendering data flow cycle

ForgeRenderer is a key component in Forge that manages this process. It traverses the visible Three.js scene graph and compiles a complete list of all Gsplats across the scene, generated by instances of SplatMesh in the scene hierarchy.

Each ForgeRenderer has a default ForgeViewpoint, which reads back a list of all Gsplat distances from the viewpoint, the computes the Gsplat draw order using an efficient bucket sort algorithm, run in a background worker thread via SplatWorker. You can spawn additional ForgeViewpoints to create multiple simultaneous render viewpoints.

Finally, on the next Three.js render() call, ForgeRenderer invokes a single instanced geometry draw call to draw all the scene's Gsplats in the correct back-to-front order, merging with other opaque Three.js geometry using the Z buffer.

const forge = new ForgeRenderer({ renderer: webGlRenderer });
// Add it to the scene and it will manage rendering of SplatMeshes
scene.add(forge);

This design allows Gsplats from distinct scenes / object splat files to coexist in space and sort correctly w.r.t. each other's Gsplats. Gsplats from independent SplatMeshes are aggregated using a SplatAccumulator, which produces a PackedSplats, a collection Gsplats stored in a cache-efficient 16-byte/Gsplat format.

"Programmable Gsplats"

Forge also uses this opportunity to run a user-programmable data pipeline on each Gsplat on the GPU. The standard pipeline provide high-level controls, such as rigid transforms, adjusting RGB / opacity, and spherical harmonics, but also special effects (via SplatEdit) and a skeletal animation system (SplatSkinning). The standard pipeline also allows injecting arbitrary code to modify each Gsplat via dyno shader graph system.

SplatMesh derives from a more general base class SplatGenerator, which itself derives from THREE.Object3D. As such, it can be placed anywhere in the scene hierarchy and obeys expected local and global coordinate transforms. A SplatGenerator is the most general form of a "Gsplat object", whose Gsplats are produced programmatically via a dyno shader graph that maps { index: "int" } to { gsplat: "Gsplat" }. A SplatMesh is a higher-level object that implements such a mapping, reading source Gsplats from a template (loaded via a url constructor parameter or otherwise) at the given index, then applying functions such as transforming to world space.

In contrast, implementing a SplatGenerator gives you full control to write any function that programmatically computes a Gsplat's attributes (center, scales, quaternion, rgba). These could be stateless (relying only on index, random-number generators, etc), or could rely on a complex combination of textures and global parameters for real-time procedural generation, and can vary with time to produce real-time animations.

The dyno shader graph system allows you to create these programmatic pipelines with Javascript code, which is synthesized into GLSL code and compiled and run on the GPU. This dyno system powers other components of Forge as well, such as Readback (which can perform any computation and read back the resulting value), used to compute the sort distance metric for pairs of Gsplats and read them back for CPU sorting.