spark/docs/packed-splats.md

PackedSplats

A PackedSplats is a collection of Gaussian splats, packed into a format that takes exactly 16 bytes per Gsplat to maximize memory and cache efficiency. The center xyz coordinates are encoded as float16 (3 x 2 bytes), scale xyz as 3 x uint8 that encode a log scale from e^-9 to e^9, rgba as 4 x uint8, and quaternion encoded via axis+angle using 2 x uint8 for octahedral encoding of the axis direction and a uint8 to encode rotation amount from 0..Pi.

Creating a PackedSplats

const packedSplats = new PackedSplats({
  // Fetch PLY/SPZ/SPLAT/KSPLAT file from URL
  url?: string;
  // Decode raw PLY/SPZ/SPLAT/KSPLAT file bytes
  fileBytes?: Uint8Array | ArrayBuffer;
  // Override file type
  fileType?: SplatFileType;
  // Reserve space for at least this many splats for construction
  maxSplats?: number;
  // Use provided packed data array, 4 words per splat
  packedArray?: Uint32Array;
  // Override number of splats in packed array to a subset
  numSplats?: number;
  // Constructor callback to create splats
  construct?: (splats: PackedSplats) => Promise<void> | void;
  // Extra splat data, such as sh1..3 components
  extra?: Record<string, unknown>;
});

Optional parameters

Like for SplatMesh you can create a new PackedSplats() with no options, will create a new empty instance with 0 Gsplats. Similarly, you can provide an input url, fileBytes, fileType to decode from a file source. You can also create a PackedSplats from a raw Uint32Array where each successive 4 Uint32 values encodes one "packed" Gsplat. Finally, a construct(splats) callback provides an ergonomic way to create Gsplats procedurally with an in-line callback closure.

Parameter	Description
url	URL to fetch a Gaussian splat file from (supports .ply, .splat, .ksplat, .spz formats). (default: undefined)
fileBytes	Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: undefined)
fileType	Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: undefined auto-detects other formats from file contents)
maxSplats	Reserve space for at least this many splats when constructing the collection initially. The array will automatically resize past maxSplats so setting it is an optional optimization. (default: 0)
packedArray	Use provided packed data array, where each 4 consecutive uint32 values encode one "packed" Gsplat. (default: undefined)
numSplats	Override number of splats in packed array to use only a subset. (default: length of packed array / 4)
construct	Callback function to programmatically create splats at initialization. (default: undefined)
extra	Additional splat data, such as spherical harmonics components (sh1, sh2, sh3). (default: {})

Encoding / Decoding

Utility functions are provided in Javascript to pack/unpack these encodings:

// Set via packedSplats interface
packedSplats.setSplat(index, center, scales, quaternion, opacity, color);

// Set underlying Uint32 array directly
import { utils } from "@forge-gfx/forge";
utils.setPackedSplat(packedSplats.packedArray, index, x, y, z, scaleX, scaleY, ...);

// Set rotation components of underlying Uint32 array directly
utils.setPackedSplatQuat(packedSplats.packedArray, index, quatX, quatY, quatZ, quatW);

// Unpack all Gsplat components from the Uint32 array
const { center, scales, quaternion, color, opacity } = utils.unpackSplat(packedSplats.packedArray, index);

// Unpack all Gsplats with callback
packedSplats.forEachSplat((index, center, scales, quaternion, opacity, color) => {
    // Use unpacked Gsplat data. Changing the inputs directly has no effect.
    // Update just the scales component
    utils.setPackedSplatScales(packedSplat.packedArray, index, 0.005, 0.01, 0.015);
    // Update the entire splat
    packedSplat.setSplat(index, center, scales, quaternion, opacity, color);
});

In GLSL / dyno shader contexts you can use the following utility functions that are available via splatDefines.glsl:

// Pack a Gsplat into a uvec4
uvec4 packSplat(vec3 center, vec3 scales, vec4 quaternion, vec4 rgba);

// Unpack a Gsplat from a uvec4
void unpackSplat(uvec4 packed, out vec3 center, out vec3 scales, out vec4 quaternion, out vec4 rgba);

// Fetch and unpack a particular index from a PackedSplats.
const gsplat = dyno.readPackedSplat(packedSplats.dyno, index);

Byte Layout

Each PackedSplat occupies 16 bytes (4 × uint32), with the following layout of fields by byte offset:

Offset (bytes)	Field	Size (bytes)	Description
0	R	1	Red color channel (uint8 0–255 → 0.0–1.0)
1	G	1	Green color channel (uint8 0–255 → 0.0–1.0)
2	B	1	Blue color channel (uint8 0–255 → 0.0–1.0)
3	A	1	Alpha (opacity) channel (uint8 0–255 → 0.0–1.0)
4–5	center.x	2	X coordinate of splat center (float16)
6–7	center.y	2	Y coordinate of splat center (float16)
8–9	center.z	2	Z coordinate of splat center (float16)
10	quat oct.U	1	Octahedral quaternion U component (uint8)
11	quat oct.V	1	Octahedral quaternion V component (uint8)
12	scale.x	1	X scale, log-encoded to uint8
13	scale.y	1	Y scale, log-encoded to uint8
14	scale.z	1	Z scale, log-encoded to uint8
15	quat angle (θ)	1	Encoded quaternion rotation angle (uint8, θ/π·255)

Gsplat RGBA encoding

RGB values are encoded are uint8 sRGB values with 0..255 mapping to 0..1. When loading from a PLY file these values are derived by calculating ply[f_dc_0] * SH_C0 + 0.5.

Opacity is encoded on a linear scale where 0..255 maps to 0..1.

Gsplat center encoding

The center x/y/z components are encoded as float16, which provides 10 bits of mantissa, or approximately 1K steps (0.1%) of resolution each power of 2 exponent, with a range of up to 32K in distance. If most of the Gsplats are positioned relative to the origin this provides enough positional resolution. Gsplats that are transformed far from the origin, however (for example when bringing multiple SplatMeshes together in a scene that are far apart) may lose precision when mapped to the space of ForgeRenderer. For scenes where the user camera may move far from the origin, you may want to tie the ForgeRenderer origin to your camera by adding it as a child of the camera.

Gsplat scales encoding

The XYZ scales are encoded independently using the following mapping: Any scale values below e^-20 are interpreted as "true zero" scale, and encoded as uint8(0). Any other values quantized by computing ln(scale_xyz), mapping the range e^-9..e^9 to uint8 values 1..254, rounding, and clamping. This logarithmic scale range provides values from 0.0001 up to 8K in scale, with approximately 7% steps between discrete values. Seems to have minimal impact on perceptible visual quality.

Gsplat orientation encoding

We encode a Gsplat's quaternion/orientation by encoding it explicitly in an axis + angle representation: 8 bits for each U/V coordinate for octahedral encoding of the axis direction, and 8 bits to encode the rotation angle range 0..Pi.

This representation was chosen oven other internal rotation representations because it provided a good mix of speed/simplicity, uniformity of representable orientations, and especially its ability to handle rotations "near identity". Other encodings such as the more common "3 quaternion components" tend to have poor rotational resolution around I(1), which is a particularly important area of that parameter.

extra Gsplat data

Each instance of PackedSplats also has a property extra: Record<string, unknown> that is used to attach additional Gsplat-related data to the PackedSplats container. For example, spherical harmonics degrees 1..3 are stored in sh1: Uint32Array(numSplats * 2), sh2: Uint32Array(numSplats * 4), sh3: Uint32Array(numSplats * 4), and intermediate textures are generated by SplatMesh and stored as sh1Texture etc.

This structure can be used to extend and store additional data in a PackedSplats, but there is no specific convention yet to prevent collisions.

sh1 stores each of 3 x 3 RGB signed components as Sint7 (mapping -1..1) in 63 bits (8 bytes) per Gsplat. sh2 stores each of 5 x 3 RGB signed components as Sint8 in 120 bits (16 bytes). sh3 stores each of 7 x 3 RGB signed components as Sint6 in 126 bits (16 bytes) to improve memory bandwidth efficiency.

PackedSplats instance methods

dispose()

Call this when you are finished with the PackedSplats and want to free any buffers it holds.

ensureSplats(numSplats)

Ensures that this.packedArray can fit numSplats Gsplats. If it's too small, resize exponentially and copy over the original data.

Typically you don't need to call this, because calling this.setSplat(index, ...) and this.pushSplat(...) will automatically call ensureSplats() so we have enough splats.

getSplat(index): { center, scales, quaternion, opacity, color }

Unpack the 16-byte Gsplat data at index into the Three.js components center: THREE.Vector3, scales: THREE.Vector3, quaternion: THREE.Quaternion, opacity: number 0..1, color: THREE.Color 0..1.

setSplat(index, center, scales, quaternion, opacity, color)

Set all PackedSplat components at index with the provided Gsplat attributes (can be the same objects returned by getSplat). Ensures there is capacity for at least index+1 Gsplats.

pushSplat(center, scales, quaternion, opacity, color)

Effectively calls this.setSplat(this.numSplats++, center, ...), useful on construction where you just want to iterate and create a collection of Gsplats.

forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)

Iterate over Gsplats index 0..=(this.numSplats-1), unpack each Gsplat and invoke the callback function with the Gsplat attributes.

getTexture()

Returns a THREE.DataArrayTexture representing the PackedSplats content as a Uint32x4 data array texture (2048 x 2048 x depth in size)

getEmpty()

Can be used where you need an uninitialized THREE.DataArrayTexture like a uniform you will update with the result of this.getTexture() later.

Generating Gsplats on the GPU

To generate a large number of Gsplats we can use the dyno shader graph system, which allows to create a computation graph mapping { index: DynoVal<"int"> } to { gsplat: DynoVal<Gsplat> } via Javascript code, then have that synthesize GLSL code, which is finally compiled and executed in parallel on the GPU.

This building block is used by ForgeRenderer to traverse each visible SplatMesh/SplatGenerator and have it "generate" its Gsplats into the global PackedSplats array managed by a SplatAccumulator. At its core a PackedSplats has the ability to run dyno computation graphs to produce its contents using the following methods, which are typically managed by ForgeRenderer:

generateMapping(splatCounts: number[]): { maxSplats, mapping[] }

Given an array of splatCounts (.numSplats for each SplatGenerator/SplatMesh in the scene), compute a "mapping layout" in the composite array of generated outputs.

ensureGenerate(maxSplats)

Ensures our PackedSplats.target render target has enough space to generate maxSplats total Gsplats, and reallocate if not large enough.

generate({ generator, base, count, ... })

Executes a dyno program specified by generator which is any DynoBlock that maps { index: "int" } to { gsplat: Gsplat }. This is called in ForgeRenderer.updateInternal() to re-generate Gsplats in the scene for SplatGenerator instances whose version is newer than what was generated for it last time.

Using dynamic PackedSplats inputs in dyno

You can use a PackedSplats as a dyno block using the function dyno.readPackedSplats(packedSplats.dyno, dynoIndex) where dynoIndex is of type DynoVal<"int"> If you need to be able to change the input PackedSplats dynamically, however, you should create a DynoPackedSplats, whose property packedSplats you can change to any PackedSplats and that will be used in the dyno shader program.