spark/docs/splat-mesh.md

SplatMesh

A SplatMesh is a high-level interface for displaying and manipulating a "Splat mesh", a collection of Gaussian splats that serves as an "object" of sorts. It is analagous to a traditional polygon THREE.Mesh, which consists of geometry (points and triangles) and materials (color and lighting). Similarly, a SplatMesh contains geometry (Gsplat centers, quaternion, and xyz scales) and materials (RGB color, opacity, spherical harmonics for directional lighting), and can be added anywhere in the scene hierarchy.

The usual Three.js properties position, quaternion, rotation behave as you would expect, however scale only allows uniform scaling and averages the x/y/z scales. Additional properties recolor and opacity are multiplied in with the final Gsplat color and opacity.

SplatMesh is a subclass of the more fundamental SplatGenerator, which itself is a subclass of THREE.Object3D. Any methods and properties on Object3D are also available in SplatMesh. SplatGenerator gives you more control over Gsplat generation and modification, but SplatMesh has an easier and higher-level API.

Creating a SplatMesh

const splats = new SplatMesh({
  // Fetch PLY/SPZ/SPLAT/KSPLAT file from URL
  url?: string;
  // Decode raw PLY/SPZ/SPLAT/KSPLAT file bytes
  fileBytes?: Uint8Array | ArrayBuffer;
  // Override file type
  fileType?: SplatFileType;
  // Use PackedSplats object as source
  packedSplats?: PackedSplats;
  // Reserve space for at least this many splats for construction
  maxSplats?: number;
  // Constructor callback to create splats
  constructSplats?: (splats: PackedSplats) => Promise<void> | void;
  // Callback for when mesh initialization is complete
  onLoad?: (mesh: SplatMesh) => Promise<void> | void;
  // Toggle controls whether SplatEdits have an effect, default true
  editable?: boolean;
  // Frame callback to update mesh. Call mesh.updateVersion() if we need to re-generate
  onFrame?: ({
    mesh,
    time,
    deltaTime,
  }: { mesh: SplatMesh; time: number; deltaTime: number }) => void;
  // Object-space and world-space Gsplat modifiers to apply
  objectModifier?: GsplatModifier;
  worldModifier?: GsplatModifier;
});
// Add to scene to show Gsplats (requires ForgeRenderer as well)
scene.add(splats);

Optional parameters

You can create a new SplatMesh() with no options, which will create a new default instance with .numSplats=0. Alternatively, you can provide an input url to fetch and decode, fileBytes, packedSplats (an existing collection of tightly "packed" Gsplats). Forge supports most Gsplat file types, including .ply, .splat, .ksplat, .spz. To load filetypes .splat and .ksplat (which can't be reliably auto-detected), use the optional fileType argument.

Constructor argument callbacks can be used like constructSplats to create a collection of Gsplat programmatically at initialization, onLoad when loading and initialization completes, onFrame to update state on every frame. Gsplat effects can be injected into the standard Gsplat processinng pipeline that operate in object-space and world-space via objectModifier and worldModifier respectively.

Parameter	Description
url	URL to fetch a Gaussian splat file from(supports .ply, .splat, .ksplat, .spz formats). (default: undefined)
fileBytes	Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: undefined)
fileType	Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: undefined auto-detects other formats from file contents)
packedSplats	Use an existing PackedSplats object as the source instead of loading from a file. Can be used to share a collection of Gsplats among multiple SplatMeshes (default: undefined creates a new empty PackedSplats or decoded from a data source above)
maxSplats	Reserve space for at least this many splats when constructing the mesh initially. (default: determined by file)
constructSplats	Callback function to programmatically create splats at initialization in provided PackedSplats. (default: undefined)
onLoad	Callback function that is called when mesh initialization is complete. (default: undefined)
editable	Controls whether SplatEdits have any effect on this mesh. (default: true)
onFrame	Callback function that is called every frame to update the mesh. Call mesh.updateVersion() if splats need to be regenerated due to some change. Calling updateVersion() is not necessary for object transformations, recoloring, or opacity adjustments as these are auto-detected. (default: undefined)
objectModifier	Gsplat modifier to apply in object-space before any transformations. A GsplatModifier is a dyno shader-graph block that transforms an input gsplat: DynoVal<Gsplat> to an output gsplat: DynoVal<Gsplat> with gsplat.center coordinate in object-space. (default: undefined)
worldModifier	Gsplat modifier to apply in world-space after transformations. (default: undefined)

Instance properties

The constructor argument options packedSplats, editable, onFrame, objectModifier, and worldModifier can be modified directly on the SplatMesh.

If you modify packedSplats you should set splatMesh.packedSplats.needsUpdate = true to signal to Three.js that it should re-upload the data to the underlying texture. Use this sparingly with objects with smaller Gsplat counts as it requires a CPU-GPU data transfer for each frame. Thousands to tens of thousands of Gsplats ir fine. (See hands.ts for an example of rendering "Gsplat hands" in WebXR using this technique.)

If you modify objectModifier or worldModifier you should call splatMesh.updateGenerator() to update the pipeline and have it compile to run efficiently on the GPU.

Additional properties you can modify on a SplatMesh instance:

Property	Description
initialized	A Promise<SplatMesh> you can await to ensure fetching, parsing, and initialization has completed
isInitialized	A boolean indicating whether initialization is complete
recolor	A THREE.Color that can be used to tint all splats in the mesh. (default: new THREE.Color(1, 1, 1))
opacity	Global opacity multiplier for all splats in the mesh. (default: 1)
context	A SplatMeshContext consisting of useful scene and object dyno uniforms that can be used to in the Gsplat processing pipeline, for example via objectModifier and worldModifier. (created on construction)
enableViewToObject	Set to true to have the viewToObject property in context be updated each frame. If the mesh has extra.sh1 (first order spherical harmonics directional lighting) this property will always be updated. (default: false )
enableViewToWorld	Set to true to have context.viewToWorld updated each frame. (default: false)
enableWorldToView	Set to true to have context.worldToView updated each frame. (default: false)
skinning	Optional SplatSkinning instance for animating splats with dual-quaternion skeletal animation. (default: null)
edits	Optional list of SplatEdits to apply to the mesh. If null, any SplatEdit children in the scene graph will be added automatically. (default: null)
splatRgba	Optional RgbaArray to overwrite splat RGBA values with custom values. Useful for "baking" RGB and opacity edits into the SplatMesh. (default: null)
maxSh	Maximum Spherical Harmonics level to use. Call updateGenerator() after changing. (default: 3)

dispose()

Call this when you are finished with the SplatMesh and want to free any buffers it holds (via packedSplats).

pushSplat(center, scales, quaternion, opacity, color)

Creates a new Gsplat with the provided parameters (all values in "float" space, i.e. 0-1 for opacity and color) and adds it to the end of the packedSplats, increasing numSplats by 1. If necessary, reallocates the buffer with an exponential doubling strategy to fit the new data, so it's fairly efficient to just pushSplat(...) each Gsplat you want to create in a loop.

forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)

This method iterates over all Gsplats in this instance's packedSplats, invoking the provided callback with index: number in 0..=(this.numSplats-1) and center: THREE.Vector3, scales: THREE.Vector3, quaternion: THREE.Quaternion, opacity: number (0..1), and color: THREE.Color (rgb values in 0..1). Note that the objects passed in as center etc. are the same for every callback invocation: these objects are reused for efficiency. Changing these values has no effect as they are decoded/unpacked copies of the underlying data. To update the packedSplats, call .packedSplats.setSplat(index, center, scales, quaternion, opacity, color).

updateGenerator()

Call this whenever something changes in the Gsplat processing pipeline, for example changing maxSh or updating objectModifier or worldModifier. Compiled generators are cached for efficiency and re-use when the same pipeline structure emerges after successive changes.

update(...)

This is called automatically by ForgeRenderer and you should not have to call it. It updates parameters for the generated pipeline and calls updateGenerator() if the pipeline needs to change.

raycast(raycaster, intersects: { distance, point, object}[])

This method conforms to the standard THREE.Raycaster API, performing object-ray intersections using this method to populate the provided intersects[] array with each intersection point.

Usage example:

const raycaster = new THREE.Raycaster();
canvas.addEventListener("click", (event) => {
  raycaster.setFromCamera(new THREE.Vector2(
    (event.clientX / canvas.width) * 2 - 1,
    -(event.clientY / canvas.height) * 2 + 1,
  ), camera);
  const intersects = raycaster.intersectObjects(scene.children);
  const splatIndex = intersects.findIndex((i) => i.object instanceof SplatMesh);
});