mirror of
https://github.com/storytold/spark.git
synced 2026-10-12 23:49:01 +00:00
Diego Marcos
124 lines
10 KiB
Markdown
124 lines
10 KiB
Markdown
# SplatMesh
|
|
|
|
A `SplatMesh` is a high-level interface for displaying and manipulating a "Splat mesh", a collection of Gaussian splats that serves as an "object" of sorts. It is analagous to a traditional polygon `THREE.Mesh`, which consists of geometry (points and triangles) and materials (color and lighting). Similarly, a `SplatMesh` contains geometry (Gsplat centers, quaternion, and xyz scales) and materials (RGB color, opacity, spherical harmonics for directional lighting), and can be added anywhere in the scene hierarchy.
|
|
|
|
The usual Three.js properties `position`, `quaternion`, `rotation` behave as you would expect, however `scale` only allows uniform scaling and averages the x/y/z scales. Additional properties `recolor` and `opacity` are multiplied in with the final Gsplat color and opacity.
|
|
|
|
`SplatMesh` is a subclass of the more fundamental `SplatGenerator`, which itself is a subclass of `THREE.Object3D`. Any methods and properties on `Object3D` are also available in `SplatMesh`. `SplatGenerator` gives you more control over Gsplat generation and modification, but `SplatMesh` has an easier and higher-level API.
|
|
|
|
## Creating a `SplatMesh`
|
|
|
|
```typescript
|
|
const splats = new SplatMesh({
|
|
// Fetch PLY/WLG/SPZ/SPLAT/KSPLAT file from URL
|
|
url?: string;
|
|
// Decode raw PLY/WLG/SPZ/SPLAT/KSPLAT file bytes
|
|
fileBytes?: Uint8Array | ArrayBuffer;
|
|
// Override file type
|
|
fileType?: SplatFileType;
|
|
// Use PackedSplats object as source
|
|
packedSplats?: PackedSplats;
|
|
// Reserve space for at least this many splats for construction
|
|
maxSplats?: number;
|
|
// Constructor callback to create splats
|
|
constructSplats?: (splats: PackedSplats) => Promise<void> | void;
|
|
// Callback for when mesh initialization is complete
|
|
onLoad?: (mesh: SplatMesh) => Promise<void> | void;
|
|
// Toggle controls whether SplatEdits have an effect, default true
|
|
editable?: boolean;
|
|
// Frame callback to update mesh. Call mesh.updateVersion() if we need to re-generate
|
|
onFrame?: ({
|
|
mesh,
|
|
time,
|
|
deltaTime,
|
|
}: { mesh: SplatMesh; time: number; deltaTime: number }) => void;
|
|
// Object-space and world-space Gsplat modifiers to apply
|
|
objectModifier?: GsplatModifier;
|
|
worldModifier?: GsplatModifier;
|
|
});
|
|
// Add to scene to show Gsplats (requires ForgeRenderer as well)
|
|
scene.add(splats);
|
|
```
|
|
|
|
### Optional parameters
|
|
|
|
You can create a `new SplatMesh()` with no options, which will create a new default instance with `.numSplats=0`. Alternatively, you can provide an input `url` to fetch and decode, `fileBytes`, `packedSplats` (an existing collection of tightly "packed" Gsplats). Forge supports most Gsplat file types, including .ply, .splat, .ksplat, .spz. To load filetypes .splat and .ksplat (which can't be reliably auto-detected), use the optional `fileType` argument.
|
|
|
|
Constructor argument callbacks can be used like `constructSplats` to create a collection of Gsplat programmatically at initialization, `onLoad` when loading and initialization completes, `onFrame` to update state on every frame. Gsplat effects can be injected into the standard Gsplat processinng pipeline that operate in object-space and world-space via `objectModifier` and `worldModifier` respectively.
|
|
|
|
| **Parameter** | Description |
|
|
| ----------------- | ----------- |
|
|
| **url** | URL to fetch a Gaussian splat file from(supports .ply, .splat, .ksplat, .spz formats). (default: `undefined`)
|
|
| **fileBytes** | Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: `undefined`)
|
|
| **fileType** | Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: `undefined` auto-detects other formats from file contents)
|
|
| **packedSplats** | Use an existing PackedSplats object as the source instead of loading from a file. Can be used to share a collection of Gsplats among multiple `SplatMesh`es (default: `undefined` creates a new empty `PackedSplats` or decoded from a data source above)
|
|
| **maxSplats** | Reserve space for at least this many splats when constructing the mesh initially. (default: determined by file)
|
|
| **constructSplats** | Callback function to programmatically create splats at initialization in provided `PackedSplats`. (default: `undefined`)
|
|
| **onLoad** | Callback function that is called when mesh initialization is complete. (default: `undefined`)
|
|
| **editable** | Controls whether SplatEdits have any effect on this mesh. (default: `true`)
|
|
| **onFrame** | Callback function that is called every frame to update the mesh. Call `mesh.updateVersion()` if splats need to be regenerated due to some change. Calling `updateVersion()` is not necessary for object transformations, recoloring, or opacity adjustments as these are auto-detected. (default: `undefined`)
|
|
| **objectModifier** | Gsplat modifier to apply in object-space before any transformations. A `GsplatModifier` is a `dyno` shader-graph block that transforms an input `gsplat: DynoVal<Gsplat>` to an output `gsplat: DynoVal<Gsplat>` with `gsplat.center` coordinate in object-space. (default: `undefined`)
|
|
| **worldModifier** | Gsplat modifier to apply in world-space after transformations. (default: `undefined`)
|
|
|
|
## Instance properties
|
|
|
|
The constructor argument options `packedSplats`, `editable`, `onFrame`, `objectModifier`, and `worldModifier` can be modified directly on the `SplatMesh`.
|
|
|
|
If you modify `packedSplats` you should set `splatMesh.packedSplats.needsUpdate = true` to signal to Three.js that it should re-upload the data to the underlying texture. Use this sparingly with objects with smaller Gsplat counts as it requires a CPU-GPU data transfer for each frame. Thousands to tens of thousands of Gsplats ir fine. (See `hands.ts` for an example of rendering "Gsplat hands" in WebXR using this technique.)
|
|
|
|
If you modify `objectModifier` or `worldModifier` you should call `splatMesh.updateGenerator()` to update the pipeline and have it compile to run efficiently on the GPU.
|
|
|
|
Additional properties you can modify on a `SplatMesh` instance:
|
|
|
|
| **Property** | Description |
|
|
| ----------------- | ----------- |
|
|
| **initialized** | A `Promise<SplatMesh>` you can await to ensure fetching, parsing, and initialization has completed
|
|
| **isInitialized** | A `boolean` indicating whether initialization is complete
|
|
| **recolor** | A `THREE.Color` that can be used to tint all splats in the mesh. (default: `new THREE.Color(1, 1, 1)`)
|
|
| **opacity** | Global opacity multiplier for all splats in the mesh. (default: `1`)
|
|
| **context** | A `SplatMeshContext` consisting of useful scene and object `dyno` uniforms that can be used to in the Gsplat processing pipeline, for example via `objectModifier` and `worldModifier`. (created on construction)
|
|
| **enableViewToObject** | Set to `true` to have the `viewToObject` property in `context` be updated each frame. If the mesh has `extra.sh1` (first order spherical harmonics directional lighting) this property will always be updated. (default: `false` )
|
|
| **enableViewToWorld** | Set to `true` to have `context.viewToWorld` updated each frame. (default: `false`)
|
|
| **enableWorldToView** | Set to `true` to have `context.worldToView` updated each frame. (default: `false`)
|
|
| **skinning** | Optional `SplatSkinning` instance for animating splats with dual-quaternion skeletal animation. (default: `null`)
|
|
| **edits** | Optional list of `SplatEdit`s to apply to the mesh. If `null`, any `SplatEdit` children in the scene graph will be added automatically. (default: `null`)
|
|
| **splatRgba** | Optional `RgbaArray` to overwrite splat RGBA values with custom values. Useful for "baking" RGB and opacity edits into the `SplatMesh`. (default: `null`)
|
|
| **maxSh** | Maximum Spherical Harmonics level to use. Call `updateGenerator()` after changing. (default: `3`)
|
|
|
|
## `dispose()`
|
|
|
|
Call this when you are finished with the `SplatMesh` and want to free any buffers it holds (via `packedSplats`).
|
|
|
|
## `pushSplat(center, scales, quaternion, opacity, color)`
|
|
|
|
Creates a new Gsplat with the provided parameters (all values in "float" space, i.e. 0-1 for opacity and color) and adds it to the end of the `packedSplats`, increasing `numSplats` by 1. If necessary, reallocates the buffer with an exponential doubling strategy to fit the new data, so it's fairly efficient to just `pushSplat(...)` each Gsplat you want to create in a loop.
|
|
|
|
## `forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)`
|
|
|
|
This method iterates over all Gsplats in this instance's `packedSplats`, invoking the provided callback with `index: number` in `0..=(this.numSplats-1)` and `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number` (0..1), and `color: THREE.Color` (rgb values in 0..1). Note that the objects passed in as `center` etc. are the same for every callback invocation: these objects are reused for efficiency. *Changing these values has no effect* as they are decoded/unpacked copies of the underlying data. To update the `packedSplats`, call `.packedSplats.setSplat(index, center, scales, quaternion, opacity, color)`.
|
|
|
|
## `updateGenerator()`
|
|
|
|
Call this whenever something changes in the Gsplat processing pipeline, for example changing `maxSh` or updating `objectModifier` or `worldModifier`. Compiled generators are cached for efficiency and re-use when the same pipeline structure emerges after successive changes.
|
|
|
|
## `update(...)`
|
|
|
|
This is called automatically by `ForgeRenderer` and you should not have to call it. It updates parameters for the generated pipeline and calls `updateGenerator()` if the pipeline needs to change.
|
|
|
|
## `raycast(raycaster, intersects: { distance, point, object}[])`
|
|
|
|
This method conforms to the standard `THREE.Raycaster` API, performing object-ray intersections using this method to populate the provided `intersects[]` array with each intersection point.
|
|
|
|
Usage example:
|
|
```javascript
|
|
const raycaster = new THREE.Raycaster();
|
|
canvas.addEventListener("click", (event) => {
|
|
raycaster.setFromCamera(new THREE.Vector2(
|
|
(event.clientX / canvas.width) * 2 - 1,
|
|
-(event.clientY / canvas.height) * 2 + 1,
|
|
), camera);
|
|
const intersects = raycaster.intersectObjects(scene.children);
|
|
const splatIndex = intersects.findIndex((i) => i.object instanceof SplatMesh);
|
|
});
|
|
```
|