# SplatMesh

A `SplatMesh` is a high-level interface for displaying and manipulating a "Splat mesh", a collection of Gaussian splats that serves as an "object" of sorts. It is analagous to a traditional polygon `THREE.Mesh`, which consists of geometry (points and triangles) and materials (color and lighting). Similarly, a `SplatMesh` contains geometry (Gsplat centers, quaternion, and xyz scales) and materials (RGB color, opacity, spherical harmonics for directional lighting), and can be added anywhere in the scene hierarchy.

The usual Three.js properties `position`, `quaternion`, `rotation` behave as you would expect, however `scale` only allows uniform scaling and averages the x/y/z scales. Additional properties `recolor` and `opacity` are multiplied in with the final Gsplat color and opacity.

`SplatMesh` is a subclass of the more fundamental `SplatGenerator`, which itself is a subclass of `THREE.Object3D`. Any methods and properties on `Object3D` are also available in `SplatMesh`. `SplatGenerator` gives you more control over Gsplat generation and modification, but `SplatMesh` has an easier and higher-level API.

## Creating a `SplatMesh`

```typescript
const splats = new SplatMesh({
  // Fetch PLY/WLG/SPZ/SPLAT/KSPLAT file from URL
  url?: string;
  // Decode raw PLY/WLG/SPZ/SPLAT/KSPLAT file bytes
  fileBytes?: Uint8Array | ArrayBuffer;
  // Override file type
  fileType?: SplatFileType;
  // Use PackedSplats object as source
  packedSplats?: PackedSplats;
  // Reserve space for at least this many splats for construction
  maxSplats?: number;
  // Constructor callback to create splats
  constructSplats?: (splats: PackedSplats) => Promise<void> | void;
  // Callback for when mesh initialization is complete
  onLoad?: (mesh: SplatMesh) => Promise<void> | void;
  // Toggle controls whether SplatEdits have an effect, default true
  editable?: boolean;
  // Frame callback to update mesh. Call mesh.updateVersion() if we need to re-generate
  onFrame?: ({
    mesh,
    time,
    deltaTime,
  }: { mesh: SplatMesh; time: number; deltaTime: number }) => void;
  // Object-space and world-space Gsplat modifiers to apply
  objectModifier?: GsplatModifier;
  worldModifier?: GsplatModifier;
});
// Add to scene to show Gsplats (requires ForgeRenderer as well)
scene.add(splats);
```

### Optional parameters

You can create a `new SplatMesh()` with no options, which will create a new default instance with `.numSplats=0`. Alternatively, you can provide an input `url` to fetch and decode, `fileBytes`, `packedSplats` (an existing collection of tightly "packed" Gsplats). Forge supports most Gsplat file types, including .ply, .splat, .ksplat, .spz. To load filetypes .splat and .ksplat (which can't be reliably auto-detected), use the optional `fileType` argument.

Constructor argument callbacks can be used like `constructSplats` to create a collection of Gsplat programmatically at initialization, `onLoad` when loading and initialization completes, `onFrame` to update state on every frame. Gsplat effects can be injected into the standard Gsplat processinng pipeline that operate in object-space and world-space via `objectModifier` and `worldModifier` respectively.

| **Parameter**     | Description |
| ----------------- | ----------- |
| **url**           | URL to fetch a Gaussian splat file from(supports .ply, .splat, .ksplat, .spz formats). (default: `undefined`)
| **fileBytes**     | Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: `undefined`)
| **fileType**      | Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: `undefined` auto-detects other formats from file contents)
| **packedSplats**  | Use an existing PackedSplats object as the source instead of loading from a file. Can be used to share a collection of Gsplats among multiple `SplatMesh`es (default: `undefined` creates a new empty `PackedSplats` or decoded from a data source above)
| **maxSplats**     | Reserve space for at least this many splats when constructing the mesh initially. (default: determined by file)
| **constructSplats** | Callback function to programmatically create splats at initialization in provided `PackedSplats`. (default: `undefined`)
| **onLoad**        | Callback function that is called when mesh initialization is complete. (default: `undefined`)
| **editable**      | Controls whether SplatEdits have any effect on this mesh. (default: `true`)
| **onFrame**       | Callback function that is called every frame to update the mesh. Call `mesh.updateVersion()` if splats need to be regenerated due to some change. Calling `updateVersion()` is not necessary for object transformations, recoloring, or opacity adjustments as these are auto-detected. (default: `undefined`)
| **objectModifier** | Gsplat modifier to apply in object-space before any transformations. A `GsplatModifier` is a `dyno` shader-graph block that transforms an input `gsplat: DynoVal<Gsplat>` to an output `gsplat: DynoVal<Gsplat>` with `gsplat.center` coordinate in object-space. (default: `undefined`)
| **worldModifier** | Gsplat modifier to apply in world-space after transformations. (default: `undefined`)

## Instance properties

The constructor argument options `packedSplats`, `editable`, `onFrame`, `objectModifier`, and `worldModifier` can be modified directly on the `SplatMesh`.

If you modify `packedSplats` you should set `splatMesh.packedSplats.needsUpdate = true` to signal to Three.js that it should re-upload the data to the underlying texture. Use this sparingly with objects with smaller Gsplat counts as it requires a CPU-GPU data transfer for each frame. Thousands to tens of thousands of Gsplats ir fine. (See `hands.ts` for an example of rendering "Gsplat hands" in WebXR using this technique.)

If you modify `objectModifier` or `worldModifier` you should call `splatMesh.updateGenerator()` to update the pipeline and have it compile to run efficiently on the GPU.

Additional properties you can modify on a `SplatMesh` instance:

| **Property**      | Description |
| ----------------- | ----------- |
| **initialized**   | A `Promise<SplatMesh>` you can await to ensure fetching, parsing, and initialization has completed
| **isInitialized** | A `boolean` indicating whether initialization is complete
| **recolor**       | A `THREE.Color` that can be used to tint all splats in the mesh. (default: `new THREE.Color(1, 1, 1)`)
| **opacity**       | Global opacity multiplier for all splats in the mesh. (default: `1`)
| **context**       | A `SplatMeshContext` consisting of useful scene and object `dyno` uniforms that can be used to in the Gsplat processing pipeline, for example via `objectModifier` and `worldModifier`. (created on construction)
| **enableViewToObject** | Set to `true` to have the `viewToObject` property in `context` be updated each frame. If the mesh has `extra.sh1` (first order spherical harmonics directional lighting) this property will always be updated. (default: `false` )
| **enableViewToWorld** | Set to `true` to have `context.viewToWorld` updated each frame. (default: `false`)
| **enableWorldToView** | Set to `true` to have `context.worldToView` updated each frame. (default: `false`)
| **skinning**      | Optional `SplatSkinning` instance for animating splats with dual-quaternion skeletal animation. (default: `null`)
| **edits**         | Optional list of `SplatEdit`s to apply to the mesh. If `null`, any `SplatEdit` children in the scene graph will be added automatically. (default: `null`)
| **splatRgba**     | Optional `RgbaArray` to overwrite splat RGBA values with custom values. Useful for "baking" RGB and opacity edits into the `SplatMesh`. (default: `null`)
| **maxSh**         | Maximum Spherical Harmonics level to use. Call `updateGenerator()` after changing. (default: `3`)

## `dispose()`

Call this when you are finished with the `SplatMesh` and want to free any buffers it holds (via `packedSplats`).

## `pushSplat(center, scales, quaternion, opacity, color)`

Creates a new Gsplat with the provided parameters (all values in "float" space, i.e. 0-1 for opacity and color) and adds it to the end of the `packedSplats`, increasing `numSplats` by 1. If necessary, reallocates the buffer with an exponential doubling strategy to fit the new data, so it's fairly efficient to just `pushSplat(...)` each Gsplat you want to create in a loop.

## `forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)`

This method iterates over all Gsplats in this instance's `packedSplats`, invoking the provided callback with `index: number` in `0..=(this.numSplats-1)` and `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number` (0..1), and `color: THREE.Color` (rgb values in 0..1). Note that the objects passed in as `center` etc. are the same for every callback invocation: these objects are reused for efficiency. *Changing these values has no effect* as they are decoded/unpacked copies of the underlying data. To update the `packedSplats`, call `.packedSplats.setSplat(index, center, scales, quaternion, opacity, color)`.

## `updateGenerator()`

Call this whenever something changes in the Gsplat processing pipeline, for example changing `maxSh` or updating `objectModifier` or `worldModifier`. Compiled generators are cached for efficiency and re-use when the same pipeline structure emerges after successive changes.

## `update(...)`

This is called automatically by `ForgeRenderer` and you should not have to call it. It updates parameters for the generated pipeline and calls `updateGenerator()` if the pipeline needs to change.

## `raycast(raycaster, intersects: { distance, point, object}[])`

This method conforms to the standard `THREE.Raycaster` API, performing object-ray intersections using this method to populate the provided `intersects[]` array with each intersection point.

Usage example:
```javascript
const raycaster = new THREE.Raycaster();
canvas.addEventListener("click", (event) => {
  raycaster.setFromCamera(new THREE.Vector2(
    (event.clientX / canvas.width) * 2 - 1,
    -(event.clientY / canvas.height) * 2 + 1,
  ), camera);
  const intersects = raycaster.intersectObjects(scene.children);
  const splatIndex = intersects.findIndex((i) => i.object instanceof SplatMesh);
});
```