# System Design

One of the biggest challenges in real-time Gaussian splat (Gsplat) rendering is sorting the splats so that they can be drawn and blended in back-to-front order, known as Painter's algorithm.

## Rendering data flow cycle

`ForgeRenderer` is a key component in Forge that manages this process. It traverses the visible Three.js scene graph and compiles a complete list of all Gsplats across the scene, generated by instances of `SplatMesh` in the scene hierarchy.

Each `ForgeRenderer` has a default `ForgeViewpoint`, which reads back a list of all Gsplat distances from the viewpoint, the computes the Gsplat draw order using an efficient bucket sort algorithm, run in a background worker thread via `SplatWorker`. You can spawn additional `ForgeViewpoint`s to create multiple simultaneous render viewpoints.

Finally, on the next Three.js render() call, `ForgeRenderer` invokes a single instanced geometry draw call to draw all the scene's Gsplats in the correct back-to-front order, merging with other opaque Three.js geometry using the Z buffer.

```typescript
const forge = new ForgeRenderer({ renderer: webGlRenderer });
// Add it to the scene and it will manage rendering of SplatMeshes
scene.add(forge);
```

This design allows Gsplats from distinct scenes / object splat files to coexist in space and sort correctly w.r.t. each other's Gsplats. Gsplats from independent `SplatMesh`es are aggregated using a `SplatAccumulator`, which produces a `PackedSplats`, a collection Gsplats stored in a cache-efficient 16-byte/Gsplat format.

## "Programmable Gsplats"

Forge also uses this opportunity to run a user-programmable data pipeline on each Gsplat on the GPU. The standard pipeline provide high-level controls, such as rigid transforms, adjusting RGB / opacity, and spherical harmonics, but also special effects (via `SplatEdit`) and a skeletal animation system (`SplatSkinning`). The standard pipeline also allows injecting arbitrary code to modify each Gsplat via `dyno` shader graph system.

`SplatMesh` derives from a more general base class `SplatGenerator`, which itself derives from `THREE.Object3D`. As such, it can be placed anywhere in the scene hierarchy and obeys expected local and global coordinate transforms. A `SplatGenerator` is the most general form of a "Gsplat object", whose Gsplats are produced programmatically via a `dyno` shader graph that maps `{ index: "int" }` to `{ gsplat: "Gsplat" }`. A `SplatMesh` is a higher-level object that implements such a mapping, reading source Gsplats from a template (loaded via a `url` constructor parameter or otherwise) at the given `index`, then applying functions such as transforming to world space.

In contrast, implementing a `SplatGenerator` gives you full control to write any function that programmatically computes a Gsplat's attributes (center, scales, quaternion, rgba). These could be stateless (relying only on `index`, random-number generators, etc), or could rely on a complex combination of textures and global parameters for real-time procedural generation, and can vary with time to produce real-time animations.

The `dyno` shader graph system allows you to create these programmatic pipelines with Javascript code, which is synthesized into GLSL code and compiled and run on the GPU. This `dyno` system powers other components of Forge as well, such as `Readback` (which can perform any computation and read back the resulting value), used to compute the sort distance metric for pairs of Gsplats and read them back for CPU sorting.