mirror of
https://github.com/storytold/spark.git
synced 2026-10-09 00:09:53 +00:00
Initial commit
This commit is contained in:
Diego Marcos
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
@@ -0,0 +1,8 @@
|
||||
<svg width="501" height="94" viewBox="0 0 501 94" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M0.618225 2.26904H137.346L132.185 20.4418C88.172 20.805 78.5674 74.9667 120.859 79.8385C120.859 79.8385 120.859 79.8386 118.408 91.5049L20.8455 91.5634L20.8455 69.2102C20.8455 69.2102 58.0664 68.1922 51.5113 49.0215C44.9561 29.8507 13.0163 42.7018 0.618225 2.26904Z" fill="white"/>
|
||||
<path d="M230.17 1.91944L226.42 21.9194H203.42V38.9194H224.17L220.42 58.9194H203.42V91.9194H179.045V19.0444L172.17 1.91944H230.17Z" fill="white"/>
|
||||
<path d="M264.215 93.1694C256.048 93.1694 249.465 91.7944 244.465 89.0444C239.465 86.2111 235.798 81.4611 233.465 74.7944C231.215 68.0444 230.09 58.7944 230.09 47.0444C230.09 35.2944 231.215 26.0861 233.465 19.4194C235.798 12.6694 239.465 7.91944 244.465 5.16944C249.465 2.3361 256.048 0.919434 264.215 0.919434C272.381 0.919434 278.965 2.3361 283.965 5.16944C288.965 7.91944 292.59 12.6694 294.84 19.4194C297.173 26.0861 298.34 35.2944 298.34 47.0444C298.34 58.7944 297.173 68.0444 294.84 74.7944C292.59 81.4611 288.965 86.2111 283.965 89.0444C278.965 91.7944 272.381 93.1694 264.215 93.1694ZM272.09 46.7944C272.09 39.6278 271.84 34.1694 271.34 30.4194C270.923 26.5861 270.131 23.9611 268.965 22.5444C267.881 21.0444 266.298 20.2944 264.215 20.2944C262.215 20.2944 260.631 21.0444 259.465 22.5444C258.298 23.9611 257.465 26.5861 256.965 30.4194C256.548 34.1694 256.34 39.6278 256.34 46.7944C256.34 53.8778 256.548 59.3778 256.965 63.2944C257.465 67.2111 258.298 69.9611 259.465 71.5444C260.631 73.0444 262.215 73.7944 264.215 73.7944C266.298 73.7944 267.881 73.0444 268.965 71.5444C270.131 69.9611 270.923 67.2111 271.34 63.2944C271.84 59.3778 272.09 53.8778 272.09 46.7944Z" fill="white"/>
|
||||
<path d="M335.355 1.91944C342.689 1.91944 348.814 2.87777 353.73 4.79444C358.647 6.62777 362.314 9.75277 364.73 14.1694C367.23 18.5028 368.48 24.4611 368.48 32.0444C368.48 38.3778 367.522 43.5861 365.605 47.6694C363.689 51.6694 360.772 54.8778 356.855 57.2944L372.105 91.9194H344.98L335.105 62.6694H331.105V91.9194H306.73V19.0444L299.855 1.91944H335.355ZM331.105 45.9194C334.939 45.9194 337.772 44.9611 339.605 43.0444C341.439 41.1278 342.355 37.7528 342.355 32.9194C342.355 29.0861 342.022 26.2528 341.355 24.4194C340.689 22.5028 339.522 21.2111 337.855 20.5444C336.272 19.8778 334.022 19.5444 331.105 19.5444V45.9194Z" fill="white"/>
|
||||
<path d="M400.261 47.4194C400.261 53.5861 400.719 58.5028 401.636 62.1694C402.552 65.8361 404.094 68.5028 406.261 70.1694C408.511 71.8361 411.594 72.8361 415.511 73.1694V64.5444L408.636 47.4194H438.011V87.6694C435.344 89.4194 431.886 90.7944 427.636 91.7944C423.386 92.7111 418.511 93.1694 413.011 93.1694C405.011 93.1694 398.094 91.6694 392.261 88.6694C386.427 85.6694 381.927 80.8361 378.761 74.1694C375.594 67.4194 374.011 58.5444 374.011 47.5444C374.011 36.4611 375.719 27.5028 379.136 20.6694C382.552 13.8361 387.344 8.8361 393.511 5.66943C399.677 2.50277 406.844 0.919434 415.011 0.919434C417.927 0.919434 420.969 1.16943 424.136 1.66943C427.302 2.16943 430.552 2.75277 433.886 3.41944L431.636 23.9194C428.886 23.4194 426.511 23.0444 424.511 22.7944C422.594 22.4611 420.636 22.2944 418.636 22.2944C414.302 22.2944 410.761 23.0028 408.011 24.4194C405.344 25.8361 403.386 28.3778 402.136 32.0444C400.886 35.6278 400.261 40.7528 400.261 47.4194Z" fill="white"/>
|
||||
<path d="M439.87 1.91944H499.745L495.995 21.9194H471.12V35.6694H494.495L490.745 55.6694H471.12V71.9194H500.62L496.87 91.9194H446.745V19.0444L439.87 1.91944Z" fill="white"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.4 KiB |
@@ -0,0 +1,152 @@
|
||||
# Controls
|
||||
|
||||
A program using `Forge` can use any camera control scheme they want that is compatible with Three.js and will typically manipulate a `THREE.Camera` object's transform. `Forge` ships with simple, intuitive controls for navigating 3D space that use the keyboard + mouse, game pad, or mobile multi-touch. To add these controls, you can add:
|
||||
|
||||
```typescript
|
||||
const controls = new ForgeControls({
|
||||
canvas: HTMLCanvasElement;
|
||||
});
|
||||
|
||||
renderer.setAnimationLoop((time) => {
|
||||
renderer.render(scene, camera);)
|
||||
controls.update(camera);
|
||||
});
|
||||
```
|
||||
|
||||
`ForgeControls` instantiates two classes `FpsMovement` and `PointerControls` that it updates internally. You can also instantiate and use these two classes separately:
|
||||
|
||||
## `class FpsMovement`
|
||||
|
||||
`FpsMovement` implements controls that will be familiar to anyone who plays First Person Shooters using keyboard + mouse or a gamepad. Creating a `FpsMovement` instance provides many parameters:
|
||||
|
||||
```typescript
|
||||
const fpsMovement = new FpsMovement({
|
||||
moveSpeed?: number;
|
||||
rollSpeed?: number;
|
||||
stickThreshold?: number;
|
||||
rotateSpeed?: number;
|
||||
keycodeMoveMapping?: { [key: string]: THREE.Vector3 };
|
||||
keycodeRotateMapping?: { [key: string]: THREE.Vector3 };
|
||||
gamepadMapping?: {
|
||||
[button: number]: "shift" | "ctrl" | "rollLeft" | "rollRight";
|
||||
};
|
||||
capsMultiplier?: number;
|
||||
shiftMultiplier?: number;
|
||||
ctrlMultiplier?: number;
|
||||
xr?: THREE.WebXRManager;
|
||||
});
|
||||
```
|
||||
When gamepads are connected, `FpsMovement` will always use gamepad index 0 for twin-stick movement and rotation.
|
||||
|
||||
If `xr` is passed in, the WebXR controllers can be used as a split gamepad to control movement and rotation. (tested on Quest 3)
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `moveSpeed` | `1.0` | Base movement speed |
|
||||
| `rollSpeed` | `2.0` | Speed of roll rotation |
|
||||
| `stickThreshold` | `0.1` | Deadzone for gamepad analog sticks |
|
||||
| `rotateSpeed` | `2.0` | Speed of rotation when using gamepad or keys |
|
||||
| `keycodeMoveMapping` | `{...WASD_KEYCODE_MOVE, ...ARROW_KEYCODE_MOVE}` | Maps keyboard keys to movement directions |
|
||||
| `keycodeRotateMapping` | `{...QE_KEYCODE_ROTATE, ...ARROW_KEYCODE_ROTATE}` | Maps keyboard keys to rotation directions |
|
||||
| `gamepadMapping` | `{4: "rollLeft", 5: "rollRight", 6: "ctrl", 7: "shift"}` | Maps gamepad buttons to actions |
|
||||
| `capsMultiplier` | `10.0` | Speed multiplier when Caps Lock is active |
|
||||
| `shiftMultiplier` | `5.0` | Speed multiplier when Shift is held |
|
||||
| `ctrlMultiplier` | `0.2` | Speed multiplier when Ctrl is held |
|
||||
| `xr` | `undefined` | Optional WebXR manager for XR controller stick support
|
||||
|
||||
### `update(deltaTime, control)`
|
||||
|
||||
Call this method in your render loop with `control` set to the object to control (`THREE.Camera` or a `THREE.Object3D` that contains it), with `deltaTime` in seconds since the last update.
|
||||
The update method handles:
|
||||
|
||||
- Processing keyboard input for movement and rotation
|
||||
- Processing gamepad input for movement and rotation
|
||||
- Applying speed multipliers based on modifier keys and gamepad buttons
|
||||
- Applying movement and rotation to the controlled object
|
||||
|
||||
|
||||
## `class PointerControls`
|
||||
|
||||
`PointerControls` implements pointer/mouse/touch controls on the canvas, for both desktop and mobile web applications. Creating a new control:
|
||||
```typescript
|
||||
const pointerControls = new PointerControls({
|
||||
canvas: HTMLCanvasElement;
|
||||
rotateSpeed?: number;
|
||||
slideSpeed?: number;
|
||||
scrollSpeed?: number;
|
||||
reverseRotate?: boolean;
|
||||
reverseSlide?: boolean;
|
||||
reverseSwipe?: boolean;
|
||||
reverseScroll?: boolean;
|
||||
moveInertia?: number;
|
||||
rotateInertia?: number;
|
||||
doublePress?: ({
|
||||
position,
|
||||
intervalMs,
|
||||
}: { position: THREE.Vector2; intervalMs: number }) => void;
|
||||
})
|
||||
```
|
||||
|
||||
### Require parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `canvas` | The HTML canvas element to attach pointer events to |
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `rotateSpeed` | `0.002` | Speed of rotation when dragging |
|
||||
| `slideSpeed` | `0.006` | Speed of sliding when dragging with right button or two fingers |
|
||||
| `scrollSpeed` | `0.0015` | Speed of movement when using mouse wheel |
|
||||
| `reverseRotate` | `false` | Reverse the direction of rotation |
|
||||
| `reverseSlide` | `false` | Reverse the direction of sliding |
|
||||
| `reverseSwipe` | `false` | Reverse the direction of swipe gestures |
|
||||
| `reverseScroll` | `false` | Reverse the direction of scroll wheel movement |
|
||||
| `moveInertia` | `0.15` | Inertia factor for movement |
|
||||
| `rotateInertia` | `0.15` | Inertia factor for rotation |
|
||||
| `doublePress` | `undefined` | Callback function for double-press/double-tap events |
|
||||
|
||||
### `update(deltaTime, control)`
|
||||
|
||||
Call this method in your render loop with `control` set to the object to control (`THREE.Camera` or a `THREE.Object3D` that contains it), with `deltaTime` in seconds since the last update.
|
||||
|
||||
The update method handles:
|
||||
|
||||
- Processing pointer/mouse/touch movements for rotation
|
||||
- Handling dual-press touch gestures for camera movement
|
||||
- Applying scroll wheel input
|
||||
- Calculating and applying inertia for smooth motion
|
||||
- Updating the camera position and orientation
|
||||
|
||||
|
||||
## Adding a simple GUI to configure controls
|
||||
|
||||
Add `lil-gui` to your package (`npm add lil-gui`) to provide a simple configurable GUI.
|
||||
|
||||
```typescript
|
||||
import GUI from "lil-gui";
|
||||
|
||||
const gui = new GUI({ title: "Settings + Controls" }).close();
|
||||
const controlOptions = {
|
||||
reversePointerFps: false,
|
||||
reversePointerPan: false,
|
||||
};
|
||||
gui
|
||||
.add(controlOptions, "reversePointerFps")
|
||||
.name("Reverse Pointer FPS")
|
||||
.onChange((value: boolean) => {
|
||||
pointerControls.reverseRotate = value;
|
||||
pointerControls.reverseScroll = value;
|
||||
});
|
||||
gui
|
||||
.add(controlOptions, "reversePointerPan")
|
||||
.name("Reverse Pointer Pan")
|
||||
.onChange((value: boolean) => {
|
||||
pointerControls.reverseSlide = value;
|
||||
pointerControls.reverseSwipe = value;
|
||||
});
|
||||
```
|
||||
@@ -0,0 +1,86 @@
|
||||
# ForgeRenderer
|
||||
|
||||
## Adding to your `THREE.Scene`
|
||||
|
||||
Using Forge begins with creating a `ForgeRenderer` object and adding it to your `THREE.Scene`. You can add it anywhere in the scene, for example at the root:
|
||||
```typescript
|
||||
const forge = new ForgeRenderer({
|
||||
renderer: myThreeWebGlRenderer,
|
||||
});
|
||||
const scene = new THREE.Scene();
|
||||
scene.add(forge);
|
||||
```
|
||||
|
||||
## Larger scenes
|
||||
|
||||
All scene Gsplats are accumulated by SplatAccumulator into a single global PackedSplats, whose coordinates are relative to the ForgeRenderer's origin. Gsplats that are far away from this origin may exhibit float16 quantization artifacts, so if you plan on moving the camera large distances you can instead add the renderer as a child of your `THREE.Camera`, ensuring that coordinates near the camera viewpoint have higher precision:
|
||||
```javascript
|
||||
const aspect = canvas.width / canvas.height;
|
||||
const camera = new THREE.PerspectiveCamera(75, aspect, 0.1, 1000);
|
||||
scene.add(camera);
|
||||
// Add ForgeRenderer as a child of camera to follow it
|
||||
camera.add(forge);
|
||||
```
|
||||
|
||||
## Creating a `ForgeRenderer`
|
||||
|
||||
```typescript
|
||||
const forge = new ForgeRenderer({
|
||||
renderer: THREE.WebGLRenderer;
|
||||
clock?: THREE.Clock;
|
||||
autoUpdate?: boolean;
|
||||
preUpdate?: boolean;
|
||||
originDistance?: number;
|
||||
maxStdDev?: number;
|
||||
enable2DGS?: boolean;
|
||||
preBlurAmount?: number;
|
||||
blurAmount?: number;
|
||||
falloff?: number;
|
||||
clipXY?: number;
|
||||
view?: ForgeViewpointOptions;
|
||||
});
|
||||
```
|
||||
### Required parameters
|
||||
| **Parameter** | Description |
|
||||
| ------------- | ----------- |
|
||||
| **renderer** | Pass in your `THREE.WebGLRenderer` instance so Forge can perform work outside the usual render loop. Should be created with `antialias: false` (default setting) as WebGL anti-aliasing doesn't improve Gaussian Splatting rendering and significantly reduces performance.
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **clock** | Pass in a `THREE.Clock` to synchronize time-based effects across different systems. Alternatively, you can set the `ForgeRenderer` properties `time` and `deltaTime` directly. (default: `new THREE.Clock`)
|
||||
| **autoUpdate** | Controls whether to check and automatically update Gsplat collection after each frame render. (default: `true`)
|
||||
| **preUpdate** | Controls whether to update the Gsplats before or after rendering. For WebXR this *must* be false in order to complete rendering as soon as possible. (default: `false`)
|
||||
| **originDistance** | Distance threshold for ForgeRenderer movement triggering a Gsplat update at the new origin. (default: `1.0`)
|
||||
| **maxStdDev** | Maximum standard deviations from the center to render Gaussians. Values `Math.sqrt(5)`..`Math.sqrt(8)` produce good results and can be tweaked for performance. (default: `Math.sqrt(8)`)
|
||||
| **enable2DGS** | Enable 2D Gaussian splatting rendering ability. When this mode is enabled, any `scale` x/y/z component that is exactly `0` (minimum quantized value) results in the other two non-0 axis being interpreted as an oriented 2D Gaussian Splat, rather instead of the usual projected 3DGS Z-slice. When reading PLY files, scale values less than e^-20 will be interpreted as `0`. (default: `true`)
|
||||
| **preBlurAmount** | Scalar value to add to 2D splat covariance diagonal, effectively blurring + enlarging splats. In scenes trained without the Gsplat anti-aliasing tweak this value was typically 0.3, but with anti-aliasing it is 0.0 (default: `0.0`)
|
||||
| **blurAmount** | Scalar value to add to 2D splat covarianve diagonal, with opacity adjustment to correctly account for "blurring" when anti-aliasing. Typically 0.3 (equivalent to approx 0.5 pixel radius) in scenes trained with anti-aliasing.
|
||||
| **falloff** | Modulate Gaussian kernel falloff. 0 means "no falloff, flat shading", while 1 is the normal Gaussian kernel. (default: `1.0`)
|
||||
| **clipXY** | X/Y clipping boundary factor for Gsplat centers against view frustum. 1.0 clips any centers that are exactly out of bounds, while 1.4 clips centers that are 40% beyond the bounds. (default: `1.4`)
|
||||
| **view** | Configures the `ForgeViewpointOptions` for the default `ForgeViewpoint` associated with this `ForgeRenderer`. Notable option: `sortRadial` (sort by radial distance or Z-depth)
|
||||
|
||||
## `newViewpoint(options: ForgeViewpointOptions)`
|
||||
|
||||
Create a new `ForgeViewpoint` for this `ForgeRenderer`. Note that every `ForgeRenderer` has an initial `forge.defaultView: ForgeViewpoint` created during construction, which is used for default canvas rendering. Calling this method allows you to create additional viewpoints, which can be updated automatically each frame (performing Gsplat sorting every time there is an update), or updated on-demand for controlled rendering for video frame rendering or similar applications.
|
||||
|
||||
## `update({ scene })`
|
||||
|
||||
If `forge.autoUpdate` is `false` then you must manually call `forge.update({ scene })` to have the scene Gsplats re-generated.
|
||||
|
||||
## `renderEnvMap({ renderer, scene, worldCenter, ... })`
|
||||
|
||||
Renders out the scene to an environment map that can be used for image-based lighting or similar applications. First updates Gsplats, sorts them with respect to the provided `worldCenter`, renders 6 cube faces, then pre-filters them using `THREE.PMREMGenerator` and returns a `THREE.Texture` that can assigned directly to a `THREE.MeshStandardMaterial.envMap` property.
|
||||
|
||||
## `recurseSetEnvMap(root, envMap)`
|
||||
|
||||
Utility function to recursively set the `envMap` property for any `THREE.MeshStandardMaterial` within the subtree of `root`.
|
||||
|
||||
## `getRgba({ generator, ... })`
|
||||
|
||||
Utility function that helps extract the Gsplat RGBA values from a `SplatGenerator`, including the result of any real-time RGBA SDF edits applied to a `SplatMesh`. This effectively "bakes" any computed RGBA values, which can now be used as a pipeline input via `SplatMesh.splatRgba` to inject these baked values into the Gsplat data.
|
||||
|
||||
## `readRgba({ generator, ...})`
|
||||
|
||||
Utility function that builds on `getRgba({ generator })` and additionally reads back the RGBA values to the CPU in a Uint8Array with packed RGBA in that byte order.
|
||||
@@ -0,0 +1,81 @@
|
||||
# ForgeViewpoint
|
||||
|
||||
A `ForgeViewpoint` is created from and tied to a `ForgeRenderer`, and represents an independent viewpoint of all the scene Gsplats and their sort order. Making these viewpoints explicit allows us to have multiple, simultaneous viewpoint renders, for example for camera preview panes or overhead map views.
|
||||
|
||||
When creating a `ForgeRenderer` it automatically creates a default viewpoint `.defaultView` that is used in the normal render loop when drawing to the canvas, and is automatically updated whenever the camera moves. Additional viewpoints can be created and configured separately:
|
||||
|
||||
## Creating a `ForgeViewpoint`
|
||||
|
||||
```typescript
|
||||
const viewpoint = forge.newViewpoint({
|
||||
autoUpdate?: boolean;
|
||||
camera?: THREE.Camera;
|
||||
viewToWorld?: THREE.Matrix4;
|
||||
target?: {
|
||||
width: number;
|
||||
height: number;
|
||||
doubleBuffer?: boolean;
|
||||
superXY?: number;
|
||||
};
|
||||
onTextureUpdated?: (texture: THREE.Texture) => void;
|
||||
sortRadial?: boolean;
|
||||
sortDistance?: number;
|
||||
sortCoorient?: boolean;
|
||||
depthBias?: number;
|
||||
sort360?: boolean;
|
||||
});
|
||||
```
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **autoUpdate** | Controls whether to auto-update its sort order whenever the ForgeRenderer updates the Gsplats. If you expect to render/display from this viewpoint most frames, set this to `true`. (default: `false`)
|
||||
| **camera** | Set a `THREE.Camera` for this viewpoint to follow. (default: `undefined`)
|
||||
| **viewToWorld** | Set an explicit view-to-world transformation matrix for this viewpoint (equivalent to `camera.matrixWorld`), overrides any `camera` setting. (default: `undefined`)
|
||||
| **target** | Configure viewpoint with an off-screen render target. (default: `undefined`)
|
||||
| **target.width** | Width of the render target in pixels.
|
||||
| **target.height** | Height of the render target in pixels.
|
||||
| **target.doubleBuffer** | If you want to be able to render a scene that depends on this target's output (for example, a recursive viewport), set this to `true` to enable double buffering. (default: `false`)
|
||||
| **target.superXY** | Super-sampling factor for the render target. Values 1-4 are supported. Note that re-sampling back down to `.width` x `.height` is done on the CPU with simple averaging only when calling `readTarget()`. (default: `1`)
|
||||
| **onTextureUpdated** | Callback function that is called when the render target texture is updated. Receives the texture as a parameter. Use this to update a viewport with the latest viewpoint render each frame. (default: `undefined`)
|
||||
| **sortRadial** | Whether to sort splats radially (geometric distance) from the viewpoint (true) or by Z-depth (false). Most scenes are trained with the Z-depth sort metric and will render more accurately at certain viewpoints. However, radial sorting is more stable under viewpoint rotations. (default: `true`)
|
||||
| **sortDistance** | Distance threshold for re-sorting splats. If the viewpoint moves more than this distance, splats will be re-sorted. (default: `0.01` units)
|
||||
| **sortCoorient** | View direction dot product threshold for re-sorting splats. For `sortRadial: true` we use 0.99 while `sortRadial: false` uses 0.999 because it is more sensitive to view direction. (default: `0.99` if `sortRadial` else `0.999`)
|
||||
| **depthBias** | Constant added to Z-depth to bias values into the positive range for `sortRadial: false`, but also used for culling Gsplats "well behind" the viewpoint origin (default: `1.0`)
|
||||
| **sort360** | Set this to true if rendering a 360 to disable "behind the viewpoint" culling during sorting. This is set automatically when rendering 360 envMaps using the `ForgeRenderer.renderEnvMap()` utility function. (default: `false`)
|
||||
|
||||
## `dispose()`
|
||||
|
||||
Call this when you are done with the `ForgeViewpoint` and want to free up its resources (GPU targets, pixel buffers, etc.)
|
||||
|
||||
## `setAutoUpdate(autoUpdate: boolean)`
|
||||
|
||||
Use this function to change whether this viewpoint will auto-update its sort order whenever the attached `ForgeRenderer` updates the Gsplats. Turn this on or off depending on whether you expect to do renders from this viewpoint most frames.
|
||||
|
||||
## `async prepareRenderPixels({ scene, camera?, viewToWOrld?, update?, forceOrigin? })`
|
||||
|
||||
Render out a viewpoint as a Uint8Array of RGBA values for the provided scene and any `camera`/`viewToWorld` viewpoint overrides. By default `update` is `true`, which triggers its `ForgeRenderer` to check and potentially update the Gsplats. Setting `update` to `false` disables this and sorts the Gsplats as they are. Setting `forceOrigin` (default: `false`) to `true` forces the view update to recalculate the splats with this view origin, potentially altering any view-dependent effects. If you expect view-dependent effects to play a role in the rendering quality, enable this.
|
||||
|
||||
Underneath, `prepareRenderPixels()` simply calls `await this.prepare(...)`, `this.renderTarget(...)`, and finally returns the result `this.readTarget()`, a Promise to a Uint8Array with RGBA values for all the pixels (potentially downsampled if the `superXY` parameter was used). These steps can also be called manually, for example if you need to alter the scene before and after `this.renderTarget(...)` to hide UI elements from being rendered.
|
||||
|
||||
## `async prepare({ scene, camera?, viewToWorld?, update?, forceOrigin? })`
|
||||
|
||||
See above `async prepareRenderPixels()` for explanation of parameters. Awaiting this method updates the Gsplats in the scene and performs a sort of the Gsplats from this viewpoint, preparing it for a subsequent `this.renderTarget()` call in the same tick.
|
||||
|
||||
## `renderTarget({ scene, camera? })`
|
||||
|
||||
Render out the viewpoint to the view target RGBA buffer. Swaps buffers if `doubleBuffer: true` was set. Calls `onTextureUpdated(texture)` with the resulting texture.
|
||||
|
||||
## `async readTarget()`
|
||||
|
||||
Read back the previously rendered target image as a Uint8Array of packed RGBA values (in that order). If `superXY` was set greater than `1` than downsampling is performed in the target pixel array with simple averaging to derive the returned pixel values. Subsequent calls to `this.readTarget()` will reuse the same buffers to minimize memory allocations.
|
||||
|
||||
## `autoPoll()`
|
||||
|
||||
This is called automatically by `ForgeRenderer`, there is no need to call it! The method cannot be private because then ForgeRenderer would not be able to call it.
|
||||
|
||||
## `ForgeViewpoint.EMPTY_TEXTURE`
|
||||
|
||||
If you need an empty `THREE.Texture` to use to initialize a uniform that is updated via `onTextureUpdated(texture)`, this static texture can be handy.
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# Getting Started
|
||||
|
||||
## Quick Start
|
||||
|
||||
Copy and paste code below in an `index.html` file or remix in the [Web Playground](https://glitch.com/edit/#!/forge-dev)
|
||||
|
||||
```html
|
||||
<style> body {margin: 0;} </style>
|
||||
<script type="importmap">
|
||||
{
|
||||
"imports": {
|
||||
"three": "/node_modules/three/build/three.module.js",
|
||||
"@worldlabsai/forge": "/dist/forge.module.js"
|
||||
}
|
||||
}
|
||||
</script>
|
||||
<script type="module">
|
||||
import * as THREE from "three";
|
||||
import { SplatMesh } from "@worldlabsai/forge";
|
||||
|
||||
const scene = new THREE.Scene();
|
||||
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
|
||||
const renderer = new THREE.WebGLRenderer();
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
document.body.appendChild(renderer.domElement)
|
||||
|
||||
const butterfly = new SplatMesh({ url: "../assets/basic/butterfly.wlg"});
|
||||
butterfly.quaternion.set(1, 0, 0, 0);
|
||||
butterfly.position.set(0, 0, -1);
|
||||
scene.add(butterfly);
|
||||
|
||||
renderer.setAnimationLoop(function animate(time) {
|
||||
renderer.render(scene, camera);
|
||||
butterfly.rotation.y += 0.01;
|
||||
});
|
||||
</script>
|
||||
```
|
||||
## Install with NPM
|
||||
|
||||
```shell
|
||||
npm install forge-dev
|
||||
```
|
||||
## Develop and contribute to Forge
|
||||
|
||||
Build Forge (It requires [Rust](https://www.rust-lang.org/tools/install) installed in your machine)
|
||||
```
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
This will run a Web server at [http://localhost:8080/](http://localhost:8080/) with the examples.
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
title: Home
|
||||
hide:
|
||||
- navigation
|
||||
- toc
|
||||
---
|
||||
|
||||
<div class="hero">
|
||||
<h1><img src="/assets/images/logo-hero.png"/></h1>
|
||||
<h2>An advanced 3D Gaussian Splatting renderer for THREE.js</h2>
|
||||
<a href="/getting-started/" class="md-button md-button--primary">Get started →</a>
|
||||
<img class="hero-image" src="/assets/images/hero-image.png"/>
|
||||
</div>
|
||||
@@ -0,0 +1,86 @@
|
||||
# Loading Gsplats
|
||||
|
||||
Forge provides loaders for most popular Gsplat file formats, including `.ply` (original "gsplat" format, some compressed variants, and plain x/y/z/r/g/b point clouds) and `.spz` (Niantic open source compressed format), both auto-detected from the file contents.
|
||||
|
||||
Forge can also load popular formats `.splat` (from `antimatter15/splat`) and `.ksplat` (from `mkkellogg/GaussianSplats3D`) if the file type can be inferred from the URL/path extension, or set explicitly using the `fileType` property when creating a `SplatMesh` or `PackedSplats`.
|
||||
|
||||
## Loading auto-detectable formats `.ply` and `.spz`
|
||||
|
||||
Adding an individual `SplatMesh` from an auto-detectable format is easy and can be done as simply as below:
|
||||
|
||||
```javascript
|
||||
// Load and create SplatMesh in one go
|
||||
const splats = new SplatMesh({ url: "./butterfly.ply" });
|
||||
scene.add(splats);
|
||||
|
||||
scene.add(new SplatMesh({ url: "plyBin/0123456789abcdef" }));
|
||||
scene.add(new SplatMesh({ url: "spzBin/fedcba9876543210" }));
|
||||
```
|
||||
|
||||
### Load via `PackedSplats`
|
||||
|
||||
Alternatively, you can load a `.ply` or `.spz` into a `PackedSplats`, which can then be used as an input source for multiple `SplatMesh` instances in the scene.
|
||||
|
||||
```javascript
|
||||
const packedSplats = new PackedSplats({ url: "./clone.ply" });
|
||||
|
||||
const splats1 = new SplatMesh({ packedSplats });
|
||||
scene.add(splats1);
|
||||
|
||||
const splats2 = new SplatMesh({ packedSplats });
|
||||
scene.add(splats2);
|
||||
```
|
||||
|
||||
### Load via `SplatLoader`
|
||||
|
||||
Finally, you can make use of the `THREE.Loader` infrastructure via the `SplatLoader` class. A `Loader` has a synchronous and asynchronous interface. You can provide progress meters during downloads and invoke completion callbacks as follows:
|
||||
|
||||
```javascript
|
||||
const loader = new SplatLoader();
|
||||
loader.loadAsync(url, (event) => {
|
||||
if (event.type === "progress") {
|
||||
const progress = event.lengthComputable
|
||||
? `${((event.loaded / event.total) * 100).toFixed(2)}%`
|
||||
: `${event.loaded} bytes`;
|
||||
console.log(`Background download progress: ${progress}`);
|
||||
}
|
||||
})
|
||||
.then((packedSplats) => {
|
||||
const splatMesh = new SplatMesh({ packedSplats });
|
||||
// Re-orient from OpenCV to OpenGL coordinates
|
||||
splatMesh.quaternion.set(1, 0, 0, 0);
|
||||
splatMesh.position.set(0, 0, -1);
|
||||
splatMesh.scale.setScalar(0.5);
|
||||
scene.add(splatMesh);
|
||||
})
|
||||
.catch((error) => {
|
||||
console.warn(error);
|
||||
});
|
||||
```
|
||||
|
||||
## Loading additional formats `.splat` and `.ksplat`
|
||||
|
||||
These formats are reliably auto-detected from the file contents, so we use two fall-back mechanism to enable support for these popular formats.
|
||||
|
||||
First, the auto-detection fails on these files, which triggers file type inference via URL/path file extension. If the URL contains the `.splat` or `.ksplat` extensions (stripping out query parameters etc.), we set the corresponding file type.
|
||||
|
||||
```javascript
|
||||
const splats = new SplatMesh({ url: "./butterfly.splats" });
|
||||
scene.add(splats);
|
||||
|
||||
const ksplats = new SplatMesh({ url: "./butterfly.ksplats" });
|
||||
scene.add(ksplats);
|
||||
```
|
||||
If the URL contains a path with no obvious file extension, you can set the field `fileType` when constructing a `SplatMesh` or `PackedSplats`.
|
||||
|
||||
```javascript
|
||||
scene.add(new SplatMesh({
|
||||
url: "splatBin/0123456789abcdef",
|
||||
fileType: SplatFileType.SPLAT,
|
||||
}));
|
||||
|
||||
scene.add(new SplatMesh({
|
||||
url: "ksplatBin/fedcba9876543210",
|
||||
fileType: SplatFileType.KSPLAT,
|
||||
}));
|
||||
```
|
||||
@@ -0,0 +1,45 @@
|
||||
{% extends "base.html" %}
|
||||
{% block extrahead %}
|
||||
{{ super() }}
|
||||
<style>
|
||||
.hero {
|
||||
text-align: center;
|
||||
}
|
||||
.hero h1 {
|
||||
font-size: 3rem;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
.hero h2 {
|
||||
margin-top: 1rem;
|
||||
margin-bottom: 2rem;
|
||||
}
|
||||
.hero p {
|
||||
font-size: 1.2rem;
|
||||
margin-bottom: 0rem;
|
||||
}
|
||||
|
||||
.hero img.hero-image {
|
||||
width: 100%;
|
||||
margin-top: 3rem;
|
||||
max-width: 800px;
|
||||
border-radius: 15px;
|
||||
}
|
||||
|
||||
.hero img {
|
||||
display: block;
|
||||
max-width: 550px;
|
||||
margin:auto;
|
||||
}
|
||||
|
||||
@media screen and (max-width: 60em) {
|
||||
.hero img {
|
||||
max-width: 400px;
|
||||
margin:auto;
|
||||
}
|
||||
.hero h2 {
|
||||
font-size: 22px;
|
||||
}
|
||||
}
|
||||
|
||||
</style>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,30 @@
|
||||
<header class="md-header" data-md-component="header">
|
||||
<nav class="md-header__inner md-grid md-custom-header" aria-label="Header">
|
||||
|
||||
<!-- Left: Logo -->
|
||||
<div class="md-header__title">
|
||||
<a href="/" class="md-header__button md-logo">
|
||||
<img src="/assets/images/logo.svg" alt="{{ config.site_name }}">
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<label class="md-header__button md-icon" for="__search">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5"></path></svg>
|
||||
</label>
|
||||
|
||||
{% include "partials/search.html" %}
|
||||
<!-- Right: Custom links + search -->
|
||||
<div class="md-custom-header-right">
|
||||
<a href="/examples/" class="md-nav__link">Examples</a>
|
||||
<a href="/getting-started/" class="md-nav__link">Docs</a>
|
||||
<a href="https://glitch.com/edit/#!/forge-dev" class="md-nav__link">Playground</a>
|
||||
<a href="/viewer/" class="md-nav__link">Viewer</a>
|
||||
<a rel="noopener noreferrer" class="contrast" aria-label="GitHub repository" href="https://github.com/forge-gfx/forge" target="_blank">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" height="24" width="24" viewBox="0 0 496 475" class="icon-github">
|
||||
<path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path>
|
||||
</svg>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</nav>
|
||||
</header>
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
*Forge aims to expand what's possible with Gaussian Splatting and help 3D/4D creators bring their visions to life and share it with others.*
|
||||
|
||||
# Overview
|
||||
|
||||
Forge is a dynamic Gaussian splat (Gsplat) renderer built on Three.js. It renders Gsplat-based worlds and objects into Three.js scenes, giving you the ability to fuse AI-generated with photogrammetry and regular triangle meshes. Forge is also programmable and fully dynamic, giving you unprecedented control over how Gsplat elements are generated and rendered into the scene.
|
||||
|
||||
## Features
|
||||
- Render multiple splat objects together with correct sorting
|
||||
- Integrates with Three.js rendering pipeline to fuse Gsplat + mesh-based objects
|
||||
- Portable: Works across almost all devices, targeting 98%+ WebGL2 support
|
||||
- Fast! Renders fast even on low-powered mobile devices
|
||||
- File format support: all major formats supported including .PLY (including compressed), .SPZ, .SPLAT, .KSPLAT
|
||||
- Render multiple viewpoints simultaneously
|
||||
- Fully dynamic: each Gsplat can be transformed and edited for animation
|
||||
- Real-time Gsplat color editing and skeletal animation
|
||||
- Shader graph system to dynamically create/edit Gsplats on the GPU
|
||||
@@ -0,0 +1,188 @@
|
||||
# PackedSplats
|
||||
|
||||
A `PackedSplats` is a collection of Gaussian splats, packed into a format that takes exactly 16 bytes per Gsplat to maximize memory and cache efficiency. The `center` xyz coordinates are encoded as float16 (3 x 2 bytes), `scale` xyz as 3 x uint8 that encode a log scale from e^-9 to e^9, `rgba` as 4 x uint8, and quaternion encoded via axis+angle using 2 x uint8 for octahedral encoding of the axis direction and a uint8 to encode rotation amount from 0..Pi.
|
||||
|
||||
## Creating a `PackedSplats`
|
||||
|
||||
```typescript
|
||||
const packedSplats = new PackedSplats({
|
||||
// Fetch PLY/WLG/SPZ/SPLAT/KSPLAT file from URL
|
||||
url?: string;
|
||||
// Decode raw PLY/WLG/SPZ/SPLAT/KSPLAT file bytes
|
||||
fileBytes?: Uint8Array | ArrayBuffer;
|
||||
// Override file type
|
||||
fileType?: SplatFileType;
|
||||
// Reserve space for at least this many splats for construction
|
||||
maxSplats?: number;
|
||||
// Use provided packed data array, 4 words per splat
|
||||
packedArray?: Uint32Array;
|
||||
// Override number of splats in packed array to a subset
|
||||
numSplats?: number;
|
||||
// Constructor callback to create splats
|
||||
construct?: (splats: PackedSplats) => Promise<void> | void;
|
||||
// Extra splat data, such as sh1..3 components
|
||||
extra?: Record<string, unknown>;
|
||||
});
|
||||
```
|
||||
|
||||
### Optional parameters
|
||||
|
||||
Like for `SplatMesh` you can create a `new PackedSplats()` with no options, will create a new empty instance with 0 Gsplats. Similarly, you can provide an input `url`, `fileBytes`, `fileType` to decode from a file source. You can also create a `PackedSplats` from a raw `Uint32Array` where each successive 4 Uint32 values encodes one "packed" Gsplat. Finally, a `construct(splats)` callback provides an ergonomic way to create Gsplats procedurally with an in-line callback closure.
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **url** | URL to fetch a Gaussian splat file from (supports .ply, .splat, .ksplat, .spz formats). (default: `undefined`)
|
||||
| **fileBytes** | Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: `undefined`)
|
||||
| **fileType** | Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: `undefined` auto-detects other formats from file contents)
|
||||
| **maxSplats** | Reserve space for at least this many splats when constructing the collection initially. The array will automatically resize past maxSplats so setting it is an optional optimization. (default: `0`)
|
||||
| **packedArray** | Use provided packed data array, where each 4 consecutive uint32 values encode one "packed" Gsplat. (default: `undefined`)
|
||||
| **numSplats** | Override number of splats in packed array to use only a subset. (default: length of packed array / 4)
|
||||
| **construct** | Callback function to programmatically create splats at initialization. (default: `undefined`)
|
||||
| **extra** | Additional splat data, such as spherical harmonics components (sh1, sh2, sh3). (default: `{}`)
|
||||
|
||||
## Encoding / Decoding
|
||||
|
||||
Utility functions are provided in Javascript to pack/unpack these encodings:
|
||||
```javascript
|
||||
|
||||
// Set via packedSplats interface
|
||||
packedSplats.setSplat(index, center, scales, quaternion, opacity, color);
|
||||
|
||||
// Set underlying Uint32 array directly
|
||||
import { utils } from "@forge-gfx/forge";
|
||||
utils.setPackedSplat(packedSplats.packedArray, index, x, y, z, scaleX, scaleY, ...);
|
||||
|
||||
// Set rotation components of underlying Uint32 array directly
|
||||
utils.setPackedSplatQuat(packedSplats.packedArray, index, quatX, quatY, quatZ, quatW);
|
||||
|
||||
// Unpack all Gsplat components from the Uint32 array
|
||||
const { center, scales, quaternion, color, opacity } = utils.unpackSplat(packedSplats.packedArray, index);
|
||||
|
||||
// Unpack all Gsplats with callback
|
||||
packedSplats.forEachSplat((index, center, scales, quaternion, opacity, color) => {
|
||||
// Use unpacked Gsplat data. Changing the inputs directly has no effect.
|
||||
// Update just the scales component
|
||||
utils.setPackedSplatScales(packedSplat.packedArray, index, 0.005, 0.01, 0.015);
|
||||
// Update the entire splat
|
||||
packedSplat.setSplat(index, center, scales, quaternion, opacity, color);
|
||||
});
|
||||
```
|
||||
|
||||
In GLSL / `dyno` shader contexts you can use the following utility functions that are available via `splatDefines.glsl`:
|
||||
|
||||
```glsl
|
||||
// Pack a Gsplat into a uvec4
|
||||
uvec4 packSplat(vec3 center, vec3 scales, vec4 quaternion, vec4 rgba);
|
||||
|
||||
// Unpack a Gsplat from a uvec4
|
||||
void unpackSplat(uvec4 packed, out vec3 center, out vec3 scales, out vec4 quaternion, out vec4 rgba);
|
||||
|
||||
// Fetch and unpack a particular index from a PackedSplats.
|
||||
const gsplat = dyno.readPackedSplat(packedSplats.dyno, index);
|
||||
```
|
||||
|
||||
### Byte Layout
|
||||
|
||||
Each `PackedSplat` occupies 16 bytes (4 × `uint32`), with the following layout of fields by byte offset:
|
||||
|
||||
| Offset (bytes) | Field | Size (bytes) | Description |
|
||||
|----------------|-----------------|--------------|------------------------------------------------------------|
|
||||
| 0 | R | 1 | Red color channel (uint8 0–255 → 0.0–1.0) |
|
||||
| 1 | G | 1 | Green color channel (uint8 0–255 → 0.0–1.0) |
|
||||
| 2 | B | 1 | Blue color channel (uint8 0–255 → 0.0–1.0) |
|
||||
| 3 | A | 1 | Alpha (opacity) channel (uint8 0–255 → 0.0–1.0) |
|
||||
| 4–5 | center.x | 2 | X coordinate of splat center (float16) |
|
||||
| 6–7 | center.y | 2 | Y coordinate of splat center (float16) |
|
||||
| 8–9 | center.z | 2 | Z coordinate of splat center (float16) |
|
||||
| 10 | quat oct.U | 1 | Octahedral quaternion U component (uint8) |
|
||||
| 11 | quat oct.V | 1 | Octahedral quaternion V component (uint8) |
|
||||
| 12 | scale.x | 1 | X scale, log-encoded to uint8 |
|
||||
| 13 | scale.y | 1 | Y scale, log-encoded to uint8 |
|
||||
| 14 | scale.z | 1 | Z scale, log-encoded to uint8 |
|
||||
| 15 | quat angle (θ) | 1 | Encoded quaternion rotation angle (uint8, θ/π·255) |
|
||||
|
||||
### Gsplat RGBA encoding
|
||||
|
||||
RGB values are encoded are uint8 sRGB values with 0..255 mapping to 0..1. When loading from a PLY file these values are derived by calculating `ply[f_dc_0] * SH_C0 + 0.5`.
|
||||
|
||||
Opacity is encoded on a linear scale where 0..255 maps to 0..1.
|
||||
|
||||
### Gsplat center encoding
|
||||
|
||||
The center x/y/z components are encoded as float16, which provides 10 bits of mantissa, or approximately 1K steps (0.1%) of resolution each power of 2 exponent, with a range of up to 32K in distance. If most of the Gsplats are positioned relative to the origin this provides enough positional resolution. Gsplats that are transformed far from the origin, however (for example when bringing multiple `SplatMesh`es together in a scene that are far apart) may lose precision when mapped to the space of `ForgeRenderer`. For scenes where the user camera may move far from the origin, you may want to tie the `ForgeRenderer` origin to your camera by adding it as a child of the camera.
|
||||
|
||||
### Gsplat scales encoding
|
||||
|
||||
The XYZ scales are encoded independently using the following mapping: Any scale values below e^-20 are interpreted as "true zero" scale, and encoded as `uint8(0)`. Any other values quantized by computing `ln(scale_xyz)`, mapping the range e^-9..e^9 to uint8 values 1..254, rounding, and clamping. This logarithmic scale range provides values from 0.0001 up to 8K in scale, with approximately 7% steps between discrete values. Seems to have minimal impact on perceptible visual quality.
|
||||
|
||||
### Gsplat orientation encoding
|
||||
|
||||
We encode a Gsplat's quaternion/orientation by encoding it explicitly in an axis + angle representation: 8 bits for each U/V coordinate for octahedral encoding of the axis direction, and 8 bits to encode the rotation angle range 0..Pi.
|
||||
|
||||
This representation was chosen oven other internal rotation representations because it provided a good mix of speed/simplicity, uniformity of representable orientations, and especially its ability to handle rotations "near identity". Other encodings such as the more common "3 quaternion components" tend to have poor rotational resolution around I(1), which is a particularly important area of that parameter.
|
||||
|
||||
### `extra` Gsplat data
|
||||
|
||||
Each instance of `PackedSplats` also has a property `extra: Record<string, unknown>` that is used to attach additional Gsplat-related data to the `PackedSplats` container. For example, spherical harmonics degrees 1..3 are stored in `sh1: Uint32Array(numSplats * 2)`, `sh2: Uint32Array(numSplats * 4)`, `sh3: Uint32Array(numSplats * 4)`, and intermediate textures are generated by `SplatMesh` and stored as `sh1Texture` etc.
|
||||
|
||||
This structure can be used to extend and store additional data in a `PackedSplats`, but there is no specific convention yet to prevent collisions.
|
||||
|
||||
`sh1` stores each of 3 x 3 RGB signed components as Sint7 (mapping -1..1) in 63 bits (8 bytes) per Gsplat. `sh2` stores each of 5 x 3 RGB signed components as Sint8 in 120 bits (16 bytes). `sh3` stores each of 7 x 3 RGB signed components as Sint6 in 126 bits (16 bytes) to improve memory bandwidth efficiency.
|
||||
|
||||
## `PackedSplats` instance methods
|
||||
|
||||
### `dispose()`
|
||||
|
||||
Call this when you are finished with the PackedSplats and want to free any buffers it holds.
|
||||
|
||||
### `ensureSplats(numSplats)`
|
||||
|
||||
Ensures that `this.packedArray` can fit `numSplats` Gsplats. If it's too small, resize exponentially and copy over the original data.
|
||||
|
||||
Typically you don't need to call this, because calling `this.setSplat(index, ...)` and `this.pushSplat(...)` will automatically call `ensureSplats()` so we have enough splats.
|
||||
|
||||
### `getSplat(index): { center, scales, quaternion, opacity, color }`
|
||||
|
||||
Unpack the 16-byte Gsplat data at `index` into the Three.js components `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number 0..1`, `color: THREE.Color 0..1`.
|
||||
|
||||
### `setSplat(index, center, scales, quaternion, opacity, color)`
|
||||
|
||||
Set all PackedSplat components at `index` with the provided Gsplat attributes (can be the same objects returned by `getSplat`). Ensures there is capacity for at least `index+1` Gsplats.
|
||||
|
||||
### `pushSplat(center, scales, quaternion, opacity, color)`
|
||||
|
||||
Effectively calls `this.setSplat(this.numSplats++, center, ...)`, useful on construction where you just want to iterate and create a collection of Gsplats.
|
||||
|
||||
### `forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)`
|
||||
|
||||
Iterate over Gsplats index `0..=(this.numSplats-1)`, unpack each Gsplat and invoke the callback function with the Gsplat attributes.
|
||||
|
||||
### `getTexture()`
|
||||
|
||||
Returns a `THREE.DataArrayTexture` representing the PackedSplats content as a Uint32x4 data array texture (2048 x 2048 x depth in size)
|
||||
|
||||
### `getEmpty()`
|
||||
|
||||
Can be used where you need an uninitialized `THREE.DataArrayTexture` like a uniform you will update with the result of `this.getTexture()` later.
|
||||
|
||||
## Generating Gsplats on the GPU
|
||||
|
||||
To generate a large number of Gsplats we can use the `dyno` shader graph system, which allows to create a computation graph mapping `{ index: DynoVal<"int"> }` to `{ gsplat: DynoVal<Gsplat> }` via Javascript code, then have that synthesize GLSL code, which is finally compiled and executed in parallel on the GPU.
|
||||
|
||||
This building block is used by `ForgeRenderer` to traverse each visible `SplatMesh`/`SplatGenerator` and have it "generate" its Gsplats into the global `PackedSplats` array managed by a `SplatAccumulator`. At its core a `PackedSplats` has the ability to run `dyno` computation graphs to produce its contents using the following methods, which are typically managed by `ForgeRenderer`:
|
||||
|
||||
### `generateMapping(splatCounts: number[]): { maxSplats, mapping[] }`
|
||||
|
||||
Given an array of splatCounts (`.numSplats` for each `SplatGenerator`/`SplatMesh` in the scene), compute a "mapping layout" in the composite array of generated outputs.
|
||||
|
||||
### `ensureGenerate(maxSplats)`
|
||||
|
||||
Ensures our `PackedSplats.target` render target has enough space to generate `maxSplats` total Gsplats, and reallocate if not large enough.
|
||||
|
||||
### `generate({ generator, base, count, ... })`
|
||||
|
||||
Executes a `dyno` program specified by `generator` which is any `DynoBlock` that maps `{ index: "int" }` to `{ gsplat: Gsplat }`. This is called in `ForgeRenderer.updateInternal()` to re-generate Gsplats in the scene for `SplatGenerator` instances whose version is newer than what was generated for it last time.
|
||||
|
||||
## Using dynamic PackedSplats inputs in `dyno`
|
||||
|
||||
You can use a `PackedSplats` as a `dyno` block using the function `dyno.readPackedSplats(packedSplats.dyno, dynoIndex)` where `dynoIndex` is of type `DynoVal<"int">` If you need to be able to change the input `PackedSplats` dynamically, however, you should create a `DynoPackedSplats`, whose property `packedSplats` you can change to any `PackedSplats` and that will be used in the `dyno` shader program.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Performance Tuning
|
||||
|
||||
Rendering millions of Gsplats at 60+ fps can be a demanding task, especially for mobile-class GPUs. Each Gsplat is rendered as two triangles that span the footprint of a Gaussian up to `sqrt(8)` standard deviations (default value) from the center. Each Gsplat is rendered as a transparent object and must be blended back-to-front.
|
||||
|
||||
As a quick rule-of-thumb, the following "Gsplat budgets" are recommended:
|
||||
|
||||
- Quest 3: 1 million Gsplats or less, not too many Gsplats concentrated in a small area
|
||||
- Android phone: 1-2 million Gsplats
|
||||
- iPhone: 1-3 million Gsplats
|
||||
- Computer: 1-5 million Gsplats (10-20+ million on some desktops)
|
||||
|
||||
Each Gsplat incurs overhead in transforming it via SplatAccumulator for sorting and rendering, and at around 1 million Gsplats this becomes a bottleneck on some systems. Unintuitively, when a large number of Gsplats are concentrated in a small area (for example 500K Gsplats from a Trellis object at a small screen scale) they can bottleneck the GPU's rendering and blending ability.
|
||||
|
||||
## maxStdDev
|
||||
|
||||
Adjust `ForgeRenderer.maxStdDev` (either directly on `ForgeRenderer` or via constructor options) to a value less than the default `Math.sqrt(8)`. This limits the extent of the Gaussian fall-off, which by default is approx 2.8. For VR a good value is `Math.sqrt(5)`, which is perceptually very similar to the default.
|
||||
@@ -0,0 +1,286 @@
|
||||
# Procedural Splats
|
||||
|
||||
Forge makes it easy to create Gaussian splat collections procedurally, and includes some splat constructors that may be useful for tasks like creating a grid or text made of splats. The example "Procedural Splats" puts some of these to use in a scene.
|
||||
|
||||
## Adding splats to a collection
|
||||
|
||||
To create a `PackedSplats` with custom, procedurally-derived splats use the methods `pushSplat` or `setSplat`:
|
||||
```javascript
|
||||
const splats = new PackedSplats();
|
||||
const center = new THREE.Vector3(0, 0, 0);
|
||||
const scales = new THREE.Vector3(0.1, 0.1, 0.1);
|
||||
const quaternion = new THREE.Quaternion();
|
||||
const opacity = 1.0;
|
||||
const color = new THREE.Color()
|
||||
splats.pushSplat(center, scales, quaternion, opacity, color);
|
||||
...
|
||||
```
|
||||
|
||||
The array in `PackedSplats` will be resized automatically to fit any splats you add. Alternatively, you can use the `construct` initializer callback:
|
||||
```javascript
|
||||
const splats = new PackedSplats({
|
||||
construct: (splats) => {
|
||||
...
|
||||
for (let i = 0; i < NUM_SPLATS; ++i) {
|
||||
// Compute splat #i
|
||||
...
|
||||
splats.pushSplat(center, scales, quaternion, opacity, color);
|
||||
}
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Once you've created your `PackedSplats`, render it to the scene via a `SplatMesh`:
|
||||
```javascript
|
||||
const mesh = new SplatMesh({ packedSplats: splats });
|
||||
scene.add(mesh);
|
||||
```
|
||||
|
||||
Alternatively, you can create the `SplatMesh` and its splats in the initializer, which internally passes your constructor to its contained `PackedSplats`:
|
||||
```javascript
|
||||
const mesh = new SplatMesh({
|
||||
constructSplats: (splats) => {
|
||||
for (let i = 0; i < NUM_SPLATS; ++i) {
|
||||
// Compute splat #i
|
||||
...
|
||||
splats.pushSplat(center, scales, quaternion, opacity, color);
|
||||
}
|
||||
},
|
||||
});
|
||||
scene.add(mesh);
|
||||
```
|
||||
|
||||
## Grid
|
||||
|
||||
```javascript
|
||||
import { constructGrid } from "@worldlabsai/forge";
|
||||
|
||||
const grid = new SplatMesh({
|
||||
constructSplats: (splats) => constructGrid({
|
||||
splats,
|
||||
extents: new THREE.Box3(
|
||||
new THREE.Vector3(-10, -10, -10),
|
||||
new THREE.Vector3(10, 10, 10),
|
||||
),
|
||||
}),
|
||||
});
|
||||
scene.add(grid);
|
||||
```
|
||||
|
||||
### Required parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `splats` | PackedSplats object to add splats to |
|
||||
| `extents` | min and max box extents of the grid |
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `stepSize` | `1` | step size along each grid axis |
|
||||
| `pointRadius` | `0.01` | spherical radius of each Gsplat |
|
||||
| `pointShadowScale` | `2.0` | relative size of the "shadow copy" of each Gsplat placed behind it |
|
||||
| `opacity` | `1.0` | Gsplat opacity |
|
||||
| `color` | RGB-modulated grid | Gsplat color (THREE.Color) or function to set color for position: ((THREE.Color, THREE.Vector3) => void) |
|
||||
|
||||
|
||||
## XYZ axis
|
||||
|
||||
```javascript
|
||||
import { constructAxes } from "@worldlabsai/forge";
|
||||
|
||||
const axes = new SplatMesh({
|
||||
constructSplats: (splats) => constructAxes({ splats }),
|
||||
});
|
||||
axes.position.set(0, 0, -1);
|
||||
scene.add(axes);
|
||||
```
|
||||
|
||||
### Required parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `splats` | PackedSplats object to add splats to |
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `scale` | `0.25` | scale (Gsplat scale along axis) |
|
||||
| `axisRadius` | `0.0075` | radius of the axes (Gsplat scale orthogonal to axis) |
|
||||
| `axisShadowScale` | `2.0` | relative size of the "shadow copy" of each Gsplat placed behind it |
|
||||
| `origins` | `[new THREE.Vector3()]` | origins of the axes (default single axis at origin) |
|
||||
|
||||
|
||||
## Gsplat sphere
|
||||
|
||||
```javascript
|
||||
import { constructSpherePoints } from "@worldlabsai/forge";
|
||||
|
||||
const sphere = new SplatMesh({
|
||||
constructSplats: (splats) => constructSpherePoints({
|
||||
splats,
|
||||
maxDepth: 4,
|
||||
}),
|
||||
});
|
||||
scene.add(sphere);
|
||||
```
|
||||
|
||||
### Required parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `splats` | PackedSplats object to add splats to |
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `origin` | `new THREE.Vector3()` | center of the sphere (default: origin) |
|
||||
| `radius` | `1.0` | radius of the sphere |
|
||||
| `maxDepth` | `3` | maximum depth of recursion for subdividing the sphere. Warning: Gsplat count grows exponentially with depth |
|
||||
| `filter` | `null` | filter function to apply to each point, for example to select points in a certain direction or other function ((THREE.Vector3) => boolean) |
|
||||
| `pointRadius` | `0.02` | radius of each oriented Gsplat |
|
||||
| `pointThickness` | `0.001` | flatness of each oriented Gsplat |
|
||||
| `color` | `new THREE.Color(1, 1, 1)` | color of each Gsplat (THREE.Color) or function to set color for point: ((THREE.Color, THREE.Vector3) => void) |
|
||||
|
||||
|
||||
## Rasterizing Text
|
||||
|
||||
```typescript
|
||||
const splats = textSplats({
|
||||
text: string;
|
||||
font?: string;
|
||||
fontSize?: number;
|
||||
color?: THREE.Color;
|
||||
rgb?: THREE.Color;
|
||||
dotRadius?: number;
|
||||
textAlign?: "left" | "center" | "right" | "start" | "end";
|
||||
lineHeight?: number;
|
||||
});
|
||||
scene.add(splats);
|
||||
```
|
||||
|
||||
### Required parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `text` | text string to display |
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `font` | `"Arial"` | browser font to render text with |
|
||||
| `fontSize` | `32` | font size in pixels/Gsplats |
|
||||
| `color` | `new THREE.Color(1, 1, 1)` | SplatMesh.recolor tint assuming white Gsplats |
|
||||
| `rgb` | `new THREE.Color(1, 1, 1)` | Individual Gsplat color |
|
||||
| `dotRadius` | `0.8` | Gsplat radius (0.8 covers 1-unit spacing well) |
|
||||
| `textAlign` | `"start"` | text alignment: "left", "center", "right", "start", "end" |
|
||||
| `lineHeight` | `1.0` | line spacing multiplier, lines delimited by "\n" |
|
||||
|
||||
## Turning images into Gsplats
|
||||
|
||||
```typescript
|
||||
const image = imageSplats({
|
||||
url: string;
|
||||
dotRadius?: number;
|
||||
subXY?: number;
|
||||
forEachSplat?: (
|
||||
width: number,
|
||||
height: number,
|
||||
index: number,
|
||||
center: THREE.Vector3,
|
||||
scales: THREE.Vector3,
|
||||
quaternion: THREE.Quaternion,
|
||||
opacity: number,
|
||||
color: THREE.Color,
|
||||
) => number | null;
|
||||
});
|
||||
scene.add(image);
|
||||
```
|
||||
|
||||
### Required parameters
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `url` | URL of the image to convert to splats (example: `url: "./image.png"`) |
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `dotRadius` | `0.8` | Radius of each Gsplat, default covers 1-unit spacing well |
|
||||
| `subXY` | `1` | Subsampling factor for the image. Higher values reduce resolution, for example 2 will halve the width and height by averaging |
|
||||
| `forEachSplat` | `undefined` | Optional callback function to modify each Gsplat before it's added. Return null to skip adding the Gsplat, or a number to set the opacity and add the Gsplat with parameter values in the objects center, rgba etc. were passed into the forEachSplat callback. Ending the callback in `return opacity;` will retain the original opacity. |
|
||||
|
||||
### Example
|
||||
|
||||
```javascript
|
||||
// Load RGBA image from image.png, subsample it 2x
|
||||
// horizontally and vertically, and create Gsplats for
|
||||
// the resulting pixels that have at least 10% opacity.
|
||||
const image = imageSplats({
|
||||
url: "./image.png",
|
||||
subXY: 2,
|
||||
forEachSplat: (width, height, index, center, scales, quaternion, opacity, color) => {
|
||||
// Only keep Gsplats with opacity 10% or higher
|
||||
return (opacity >= 0.1) ? opacity : null;
|
||||
},
|
||||
});
|
||||
scene.add(image);
|
||||
```
|
||||
|
||||
## Particle Effects
|
||||
|
||||
Some of the building blocks are meant to serve as "code inspiration", showing how a particle effect animation can be achieved using a "stateless" `dyno` computation graph that uses only the Gsplat `index` input to produce pseudo-random numbers that drive various randomized particle effects.
|
||||
|
||||
Note that Gsplat sorting takes a little bit of time and can "lag behind" the Gsplat updates each frame, so it's important that there is a reasonably stable correspondence between each frame's Gsplat output for the same `index`.
|
||||
|
||||
In `staticBox`, the AABB is sliced up into a 3D grid of cells, and random X/Y/Z points within those cells are sampled each frame, retaining consistency in Gsplat position between successive frames. See `examples/gsplat-sweets-garden/main.js` for an example that creates a `staticBox`.
|
||||
|
||||
### `generators.snowBox(...options)`
|
||||
|
||||
Similarly, `snowBox` produces Gsplat trajectories that move in a deterministic fashion over time, with high similarity between adjacent frames. See `examples/atmospheric/main.js` for an example that creates a `snowBox`.
|
||||
|
||||
A snowBox instance has a collection of properties that can be tuned to achieve different particle effects. DEFAULT_SNOW and DEFAULT_RAIN are example parameter sets that look a lot like snow and rain, and can be used as a starting point for further tweaking: `const mySnow = { ...DEFAULT_SNOW, density: 500 };`
|
||||
|
||||
```typescript
|
||||
const snowControls = generators.snowBox({
|
||||
box,
|
||||
minY,
|
||||
numSplats,
|
||||
density,
|
||||
anisoScale,
|
||||
minScale,
|
||||
maxScale,
|
||||
fallDirection,
|
||||
fallVelocity,
|
||||
wanderScale,
|
||||
wanderVariance,
|
||||
color1,
|
||||
color2,
|
||||
opacity,
|
||||
onFrame,
|
||||
});
|
||||
scene.add(snowControls.snow);
|
||||
```
|
||||
|
||||
| Parameter | Default Value | Description |
|
||||
|-----------|---------------|-------------|
|
||||
| `box` | `new THREE.Box3(new THREE.Vector3(-1, -1, -1), new THREE.Vector3(1, 1, 1))` | min and max box extents of the snowBox |
|
||||
| `minY` | `Number.NEGATIVE_INFINITY` | minimum y-coordinate to clamp particle position, which can be used to fake hitting a ground plane and lingering there for a bit |
|
||||
| `numSplats` | calculated from box and density | number of Gsplats to generate |
|
||||
| `density` | `100` | density of Gsplats per unit volume |
|
||||
| `anisoScale` | `new THREE.Vector3(1, 1, 1)` | The xyz anisotropic scale of the Gsplat, which can be used for example to elongate rain particles |
|
||||
| `minScale` | `0.001` | Minimum Gsplat particle scale |
|
||||
| `maxScale` | `0.005` | Maximum Gsplat particle scale |
|
||||
| `fallDirection` | `new THREE.Vector3(0, -1, 0)` | The average direction of fall |
|
||||
| `fallVelocity` | `0.02` | The average speed of the fall (multiplied with fallDirection) |
|
||||
| `wanderScale` | `0.01` | The world scale of wandering overlay motion |
|
||||
| `wanderVariance` | `2` | Controls how uniformly the particles wander in sync, more variance means more randomness in the motion |
|
||||
| `color1` | `new THREE.Color(1, 1, 1)` | Color 1 of the two colors interpolated between |
|
||||
| `color2` | `new THREE.Color(0.5, 0.5, 1)` | Color 2 of the two colors interpolated between |
|
||||
| `opacity` | `1` | The base opacity of the Gsplats |
|
||||
| `onFrame` | `undefined` | Optional callback function to call each frame |
|
||||
@@ -0,0 +1,88 @@
|
||||
# Gsplat Editing
|
||||
|
||||
Forge provides the ability to apply "edits" to Gsplats as part of the standard `SplatMesh` pipeline. These edits take the form of a sequence of operations, applied one at a time to the set of Gsplats in its `packedSplats`. Each operation evaluates a 7-dimensional field (RGBA and XYZ displacement) at each point in space that derives from N=1 or more Signed Distance Field shapes (such as spheres, boxes, planes, etc.), blended together and across inside-outisde boundaries.
|
||||
|
||||
The result is a an RGBA,XYZ value for each point in space, which combined with SplatEditRgbaBlendMode.MULTIPLY/SET_RGB/ADD_RGBA can be used to create special effects, for example simulating simple lighting or applying deformations in space, whose parameters can be updated each frame to create animated effects.
|
||||
|
||||
## RGBA blend modes
|
||||
|
||||
When creating a `SplatEdit` you can specify a `rgbaBlendMode?: SplatEditRgbaBlendMode` value from the enum to choose between 3 blend modes:
|
||||
|
||||
| Blend Mode | Description |
|
||||
|-------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
|
||||
| `SplatEditRgbaBlendMode.MULTIPLY` | The RGBA of the splat is multiplied component-wise by the SDF’s RGBA value at that point in space. |
|
||||
| `SplatEditRgbaBlendMode.SET_RGB` | Ignore the Alpha value in the SDF, but set the splat’s RGB to equal the SDF’s RGB value at that point. |
|
||||
| `SplatEditRgbaBlendMode.ADD_RGBA` | Add the SDF’s RGBA value at that point to the RGBA value of the Gsplat. This can produce hyper-saturated results, but is useful to easily “light up” areas. |
|
||||
|
||||
## SDF Shapes
|
||||
|
||||
The following SDF shapes are available in the `SplatEditSdfType` enum:
|
||||
|
||||
| SDF Type | Description | Parameters |
|
||||
|----------|-------------|------------|
|
||||
| `ALL` | Affects all points in space | None |
|
||||
| `PLANE` | Infinite plane | position, rotation |
|
||||
| `SPHERE` | Sphere | position, radius |
|
||||
| `BOX` | Box (with optional corner rounding) | position, rotation, sizes, radius |
|
||||
| `ELLIPSOID` | Ellipsoid | position, rotation, sizes |
|
||||
| `CYLINDER` | Cylinder | position, rotation, size_y |
|
||||
| `CAPSULE` | Capsule | position, rotation, size_y |
|
||||
| `INFINITE_CONE` | Infinite cone | position, rotation, radius=angle |
|
||||
|
||||
## Creating a Gsplat edit operation
|
||||
|
||||
A `SplatEdit` operation can be assigned to a particular `SplatMesh` through its `.edits[]` property or by adding the `SplatEdit` as a child of the `SplatMesh` in the scene hierarchy. If the `SplatEdit` has no `SplatMesh` ancestor, its edits will apply globally to all `SplatMesh`es whose `editable` property is set to default true.
|
||||
|
||||
```typescript
|
||||
const edit = new SplatEdit({
|
||||
name?: string;
|
||||
rgbaBlendMode?: SplatEditRgbaBlendMode;
|
||||
sdfSmooth?: number;
|
||||
softEdge?: number;
|
||||
invert?: boolean;
|
||||
sdfs?: SplatEditSdf[];
|
||||
});
|
||||
scene.add(edit);
|
||||
```
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------------|------------------------------------------|--------------------------------------------------------------------------------------------------------------|
|
||||
| `name` | `undefined` (auto‐generated to `Edit <n>`) | Name of this edit operation. If you omit it, a default `"Edit 1"`, `"Edit 2"`, … is assigned. |
|
||||
| `rgbaBlendMode` | `SplatEditRgbaBlendMode.MULTIPLY` | How the SDF’s RGBA modifies each splat’s RGBA: multiply, overwrite RGB, or add RGBA. |
|
||||
| `sdfSmooth` | `0.0` | Smoothing (in world‐space units) for blending between multiple SDF shapes at their boundaries. |
|
||||
| `softEdge` | `0.0` | Soft‐edge falloff radius (in world‐space units) around each SDF shape’s surface. |
|
||||
| `invert` | `false` | Invert the SDF evaluation (inside/outside swap). |
|
||||
| `sdfs` | `null` | Explicit array of `SplatEditSdf` objects to include. If `null`, any child `SplatEditSdf` instances are used. |
|
||||
|
||||
## Adding an SDF RGBA-XYZ shape to the edit operation
|
||||
|
||||
```typescript
|
||||
const shape1 = new SplatEditSdf({
|
||||
type?: SplatEditSdfType;
|
||||
invert?: boolean;
|
||||
opacity?: number;
|
||||
color?: THREE.Color;
|
||||
displace?: THREE.Vector3;
|
||||
radius?: number;
|
||||
});
|
||||
edit.add(shape1);
|
||||
```
|
||||
|
||||
### Optional parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `type` | `SplatEditSdfType.SPHERE` | The SDF shape type: `ALL`, `PLANE`, `SPHERE`, `BOX`, `ELLIPSOID`, `CYLINDER`, `CAPSULE`, or `INFINITE_CONE`. |
|
||||
| `invert` | `false` | Invert the SDF evaluation, swapping inside and outside regions. |
|
||||
| `opacity` | `1.0` | Opacity / "alpha" value used differently by blending modes |
|
||||
| `color` | `new THREE.Color(1.0, 1.0, 1.0)` | RGB color applied within the shape. |
|
||||
| `displace` | `new THREE.Vector3(0.0, 0.0, 0.0)` | XYZ displacement applied to splat positions inside the shape. |
|
||||
| `radius` | `0.0` | Shape-specific size parameter: sphere radius, box corner rounding, cylinder/capsule radius, or for the infinite cone the angle factor (opening half-angle = π/4 × `radius`). |
|
||||
|
||||
## Adding multiple SDF RGBA-XYZ shapes to the edit operation
|
||||
|
||||
RGBA-XYZ values are computed by blending together values from all SDF shapes using the exponential "softmax" function, which is commutative (so blending order within a `SplatEdit` operation doesn't matter). The parameter `SplatEdit.sdfSmooth` controls the blending scale *between* SDF shapes, while `SplatEdit.softEdge` controls the scale of soft inside-outside shape edit blending. Their default values start at `0.0` and should be increased to soften the effect.
|
||||
|
||||
Note that XYZ displacement values are blended in the same way as RGBA, with a resulting displacement field that can be quite complex but "softly" blending between shapes. These RGBA-XYZ edits, along with time-based and overlapping fields can create many interesting animations and special effects, such as rippling leaves in the wind, an angry fire, or a looping water effects. Simply update the `SplatEdit` and `SplatEditSdf` objects and the operations will be applied immediately to the Gsplats in the scene.
|
||||
@@ -0,0 +1,123 @@
|
||||
# SplatMesh
|
||||
|
||||
A `SplatMesh` is a high-level interface for displaying and manipulating a "Splat mesh", a collection of Gaussian splats that serves as an "object" of sorts. It is analagous to a traditional polygon `THREE.Mesh`, which consists of geometry (points and triangles) and materials (color and lighting). Similarly, a `SplatMesh` contains geometry (Gsplat centers, quaternion, and xyz scales) and materials (RGB color, opacity, spherical harmonics for directional lighting), and can be added anywhere in the scene hierarchy.
|
||||
|
||||
The usual Three.js properties `position`, `quaternion`, `rotation` behave as you would expect, however `scale` only allows uniform scaling and averages the x/y/z scales. Additional properties `recolor` and `opacity` are multiplied in with the final Gsplat color and opacity.
|
||||
|
||||
`SplatMesh` is a subclass of the more fundamental `SplatGenerator`, which itself is a subclass of `THREE.Object3D`. Any methods and properties on `Object3D` are also available in `SplatMesh`. `SplatGenerator` gives you more control over Gsplat generation and modification, but `SplatMesh` has an easier and higher-level API.
|
||||
|
||||
## Creating a `SplatMesh`
|
||||
|
||||
```typescript
|
||||
const splats = new SplatMesh({
|
||||
// Fetch PLY/WLG/SPZ/SPLAT/KSPLAT file from URL
|
||||
url?: string;
|
||||
// Decode raw PLY/WLG/SPZ/SPLAT/KSPLAT file bytes
|
||||
fileBytes?: Uint8Array | ArrayBuffer;
|
||||
// Override file type
|
||||
fileType?: SplatFileType;
|
||||
// Use PackedSplats object as source
|
||||
packedSplats?: PackedSplats;
|
||||
// Reserve space for at least this many splats for construction
|
||||
maxSplats?: number;
|
||||
// Constructor callback to create splats
|
||||
constructSplats?: (splats: PackedSplats) => Promise<void> | void;
|
||||
// Callback for when mesh initialization is complete
|
||||
onLoad?: (mesh: SplatMesh) => Promise<void> | void;
|
||||
// Toggle controls whether SplatEdits have an effect, default true
|
||||
editable?: boolean;
|
||||
// Frame callback to update mesh. Call mesh.updateVersion() if we need to re-generate
|
||||
onFrame?: ({
|
||||
mesh,
|
||||
time,
|
||||
deltaTime,
|
||||
}: { mesh: SplatMesh; time: number; deltaTime: number }) => void;
|
||||
// Object-space and world-space Gsplat modifiers to apply
|
||||
objectModifier?: GsplatModifier;
|
||||
worldModifier?: GsplatModifier;
|
||||
});
|
||||
// Add to scene to show Gsplats (requires ForgeRenderer as well)
|
||||
scene.add(splats);
|
||||
```
|
||||
|
||||
### Optional parameters
|
||||
|
||||
You can create a `new SplatMesh()` with no options, which will create a new default instance with `.numSplats=0`. Alternatively, you can provide an input `url` to fetch and decode, `fileBytes`, `packedSplats` (an existing collection of tightly "packed" Gsplats). Forge supports most Gsplat file types, including .ply, .splat, .ksplat, .spz. To load filetypes .splat and .ksplat (which can't be reliably auto-detected), use the optional `fileType` argument.
|
||||
|
||||
Constructor argument callbacks can be used like `constructSplats` to create a collection of Gsplat programmatically at initialization, `onLoad` when loading and initialization completes, `onFrame` to update state on every frame. Gsplat effects can be injected into the standard Gsplat processinng pipeline that operate in object-space and world-space via `objectModifier` and `worldModifier` respectively.
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **url** | URL to fetch a Gaussian splat file from(supports .ply, .splat, .ksplat, .spz formats). (default: `undefined`)
|
||||
| **fileBytes** | Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: `undefined`)
|
||||
| **fileType** | Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: `undefined` auto-detects other formats from file contents)
|
||||
| **packedSplats** | Use an existing PackedSplats object as the source instead of loading from a file. Can be used to share a collection of Gsplats among multiple `SplatMesh`es (default: `undefined` creates a new empty `PackedSplats` or decoded from a data source above)
|
||||
| **maxSplats** | Reserve space for at least this many splats when constructing the mesh initially. (default: determined by file)
|
||||
| **constructSplats** | Callback function to programmatically create splats at initialization in provided `PackedSplats`. (default: `undefined`)
|
||||
| **onLoad** | Callback function that is called when mesh initialization is complete. (default: `undefined`)
|
||||
| **editable** | Controls whether SplatEdits have any effect on this mesh. (default: `true`)
|
||||
| **onFrame** | Callback function that is called every frame to update the mesh. Call `mesh.updateVersion()` if splats need to be regenerated due to some change. Calling `updateVersion()` is not necessary for object transformations, recoloring, or opacity adjustments as these are auto-detected. (default: `undefined`)
|
||||
| **objectModifier** | Gsplat modifier to apply in object-space before any transformations. A `GsplatModifier` is a `dyno` shader-graph block that transforms an input `gsplat: DynoVal<Gsplat>` to an output `gsplat: DynoVal<Gsplat>` with `gsplat.center` coordinate in object-space. (default: `undefined`)
|
||||
| **worldModifier** | Gsplat modifier to apply in world-space after transformations. (default: `undefined`)
|
||||
|
||||
## Instance properties
|
||||
|
||||
The constructor argument options `packedSplats`, `editable`, `onFrame`, `objectModifier`, and `worldModifier` can be modified directly on the `SplatMesh`.
|
||||
|
||||
If you modify `packedSplats` you should set `splatMesh.packedSplats.needsUpdate = true` to signal to Three.js that it should re-upload the data to the underlying texture. Use this sparingly with objects with smaller Gsplat counts as it requires a CPU-GPU data transfer for each frame. Thousands to tens of thousands of Gsplats ir fine. (See `hands.ts` for an example of rendering "Gsplat hands" in WebXR using this technique.)
|
||||
|
||||
If you modify `objectModifier` or `worldModifier` you should call `splatMesh.updateGenerator()` to update the pipeline and have it compile to run efficiently on the GPU.
|
||||
|
||||
Additional properties you can modify on a `SplatMesh` instance:
|
||||
|
||||
| **Property** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **initialized** | A `Promise<SplatMesh>` you can await to ensure fetching, parsing, and initialization has completed
|
||||
| **isInitialized** | A `boolean` indicating whether initialization is complete
|
||||
| **recolor** | A `THREE.Color` that can be used to tint all splats in the mesh. (default: `new THREE.Color(1, 1, 1)`)
|
||||
| **opacity** | Global opacity multiplier for all splats in the mesh. (default: `1`)
|
||||
| **context** | A `SplatMeshContext` consisting of useful scene and object `dyno` uniforms that can be used to in the Gsplat processing pipeline, for example via `objectModifier` and `worldModifier`. (created on construction)
|
||||
| **enableViewToObject** | Set to `true` to have the `viewToObject` property in `context` be updated each frame. If the mesh has `extra.sh1` (first order spherical harmonics directional lighting) this property will always be updated. (default: `false` )
|
||||
| **enableViewToWorld** | Set to `true` to have `context.viewToWorld` updated each frame. (default: `false`)
|
||||
| **enableWorldToView** | Set to `true` to have `context.worldToView` updated each frame. (default: `false`)
|
||||
| **skinning** | Optional `SplatSkinning` instance for animating splats with dual-quaternion skeletal animation. (default: `null`)
|
||||
| **edits** | Optional list of `SplatEdit`s to apply to the mesh. If `null`, any `SplatEdit` children in the scene graph will be added automatically. (default: `null`)
|
||||
| **splatRgba** | Optional `RgbaArray` to overwrite splat RGBA values with custom values. Useful for "baking" RGB and opacity edits into the `SplatMesh`. (default: `null`)
|
||||
| **maxSh** | Maximum Spherical Harmonics level to use. Call `updateGenerator()` after changing. (default: `3`)
|
||||
|
||||
## `dispose()`
|
||||
|
||||
Call this when you are finished with the `SplatMesh` and want to free any buffers it holds (via `packedSplats`).
|
||||
|
||||
## `pushSplat(center, scales, quaternion, opacity, color)`
|
||||
|
||||
Creates a new Gsplat with the provided parameters (all values in "float" space, i.e. 0-1 for opacity and color) and adds it to the end of the `packedSplats`, increasing `numSplats` by 1. If necessary, reallocates the buffer with an exponential doubling strategy to fit the new data, so it's fairly efficient to just `pushSplat(...)` each Gsplat you want to create in a loop.
|
||||
|
||||
## `forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)`
|
||||
|
||||
This method iterates over all Gsplats in this instance's `packedSplats`, invoking the provided callback with `index: number` in `0..=(this.numSplats-1)` and `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number` (0..1), and `color: THREE.Color` (rgb values in 0..1). Note that the objects passed in as `center` etc. are the same for every callback invocation: these objects are reused for efficiency. *Changing these values has no effect* as they are decoded/unpacked copies of the underlying data. To update the `packedSplats`, call `.packedSplats.setSplat(index, center, scales, quaternion, opacity, color)`.
|
||||
|
||||
## `updateGenerator()`
|
||||
|
||||
Call this whenever something changes in the Gsplat processing pipeline, for example changing `maxSh` or updating `objectModifier` or `worldModifier`. Compiled generators are cached for efficiency and re-use when the same pipeline structure emerges after successive changes.
|
||||
|
||||
## `update(...)`
|
||||
|
||||
This is called automatically by `ForgeRenderer` and you should not have to call it. It updates parameters for the generated pipeline and calls `updateGenerator()` if the pipeline needs to change.
|
||||
|
||||
## `raycast(raycaster, intersects: { distance, point, object}[])`
|
||||
|
||||
This method conforms to the standard `THREE.Raycaster` API, performing object-ray intersections using this method to populate the provided `intersects[]` array with each intersection point.
|
||||
|
||||
Usage example:
|
||||
```javascript
|
||||
const raycaster = new THREE.Raycaster();
|
||||
canvas.addEventListener("click", (event) => {
|
||||
raycaster.setFromCamera(new THREE.Vector2(
|
||||
(event.clientX / canvas.width) * 2 - 1,
|
||||
-(event.clientY / canvas.height) * 2 + 1,
|
||||
), camera);
|
||||
const intersects = raycaster.intersectObjects(scene.children);
|
||||
const splatIndex = intersects.findIndex((i) => i.object instanceof SplatMesh);
|
||||
});
|
||||
```
|
||||
@@ -0,0 +1,120 @@
|
||||
@font-face {
|
||||
font-family: "Inter";
|
||||
src: url(https://fonts.googleapis.com/css2?family=Cinzel:wght@400..900&family=Inter:ital,opsz,wght@0,14..32,100..900;1,14..32,100..900&family=Limelight&display=swap);
|
||||
}
|
||||
|
||||
:root {
|
||||
--color-dark: white;
|
||||
--color-light: black;
|
||||
--md-primary-fg-color: #d43e4c;
|
||||
--md-accent-fg-color: #cb6065;
|
||||
--md-text-font: "Inter";
|
||||
}
|
||||
|
||||
.md-typeset .md-button.md-button--primary {
|
||||
background-color: --md-primary-fg-color;
|
||||
border-color: --md-primary-fg-color;
|
||||
color: white;
|
||||
padding: 0.75em 2em;
|
||||
font-size: 1rem;
|
||||
border-radius: .75rem;
|
||||
}
|
||||
|
||||
.md-button.md-button--primary:hover {
|
||||
opacity: 1.0;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.md-header__button.md-logo {
|
||||
color: white;
|
||||
}
|
||||
|
||||
.md-search__form {
|
||||
border-radius: 5px;
|
||||
}
|
||||
|
||||
.icon-github {
|
||||
width: 30px;
|
||||
height: 30px;
|
||||
}
|
||||
|
||||
.icon-github path {
|
||||
fill: white;
|
||||
}
|
||||
|
||||
.md-logo {
|
||||
height: 40px;
|
||||
}
|
||||
|
||||
.md-logo:hover {
|
||||
text-decoration: none;
|
||||
opacity: 1.0;
|
||||
}
|
||||
|
||||
.md-header {
|
||||
padding: 5px;
|
||||
padding-top: 15px;
|
||||
padding-bottom: 15px;
|
||||
}
|
||||
|
||||
.md-header__inner {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.md-header__links {
|
||||
display: flex;
|
||||
gap: 1rem;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.md-nav {
|
||||
line-height: 2;
|
||||
}
|
||||
|
||||
.md-nav__link {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
.md-nav__link:hover,
|
||||
.md-nav__link:focus {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.md-header__option {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 1rem;
|
||||
}
|
||||
|
||||
.md-search {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.md-header__topic {
|
||||
display: flex;
|
||||
gap: 1.25rem;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.md-custom-header,
|
||||
.md-header__title {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
.md-custom-header-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 1rem;
|
||||
margin-left: 20px;
|
||||
margin-right: 20px;
|
||||
}
|
||||
|
||||
.md-custom-header-right .md-nav__link:hover {
|
||||
color: white;
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
# System Design
|
||||
|
||||
One of the biggest challenges in real-time Gaussian splat (Gsplat) rendering is sorting the splats so that they can be drawn and blended in back-to-front order, known as Painter's algorithm.
|
||||
|
||||
## Rendering data flow cycle
|
||||
|
||||
`ForgeRenderer` is a key component in Forge that manages this process. It traverses the visible Three.js scene graph and compiles a complete list of all Gsplats across the scene, generated by instances of `SplatMesh` in the scene hierarchy.
|
||||
|
||||
Each `ForgeRenderer` has a default `ForgeViewpoint`, which reads back a list of all Gsplat distances from the viewpoint, the computes the Gsplat draw order using an efficient bucket sort algorithm, run in a background worker thread via `SplatWorker`. You can spawn additional `ForgeViewpoint`s to create multiple simultaneous render viewpoints.
|
||||
|
||||
Finally, on the next Three.js render() call, `ForgeRenderer` invokes a single instanced geometry draw call to draw all the scene's Gsplats in the correct back-to-front order, merging with other opaque Three.js geometry using the Z buffer.
|
||||
|
||||
```typescript
|
||||
const forge = new ForgeRenderer({ renderer: webGlRenderer });
|
||||
// Add it to the scene and it will manage rendering of SplatMeshes
|
||||
scene.add(forge);
|
||||
```
|
||||
|
||||
This design allows Gsplats from distinct scenes / object splat files to coexist in space and sort correctly w.r.t. each other's Gsplats. Gsplats from independent `SplatMesh`es are aggregated using a `SplatAccumulator`, which produces a `PackedSplats`, a collection Gsplats stored in a cache-efficient 16-byte/Gsplat format.
|
||||
|
||||
## "Programmable Gsplats"
|
||||
|
||||
Forge also uses this opportunity to run a user-programmable data pipeline on each Gsplat on the GPU. The standard pipeline provide high-level controls, such as rigid transforms, adjusting RGB / opacity, and spherical harmonics, but also special effects (via `SplatEdit`) and a skeletal animation system (`SplatSkinning`). The standard pipeline also allows injecting arbitrary code to modify each Gsplat via `dyno` shader graph system.
|
||||
|
||||
`SplatMesh` derives from a more general base class `SplatGenerator`, which itself derives from `THREE.Object3D`. As such, it can be placed anywhere in the scene hierarchy and obeys expected local and global coordinate transforms. A `SplatGenerator` is the most general form of a "Gsplat object", whose Gsplats are produced programmatically via a `dyno` shader graph that maps `{ index: "int" }` to `{ gsplat: "Gsplat" }`. A `SplatMesh` is a higher-level object that implements such a mapping, reading source Gsplats from a template (loaded via a `url` constructor parameter or otherwise) at the given `index`, then applying functions such as transforming to world space.
|
||||
|
||||
In contrast, implementing a `SplatGenerator` gives you full control to write any function that programmatically computes a Gsplat's attributes (center, scales, quaternion, rgba). These could be stateless (relying only on `index`, random-number generators, etc), or could rely on a complex combination of textures and global parameters for real-time procedural generation, and can vary with time to produce real-time animations.
|
||||
|
||||
The `dyno` shader graph system allows you to create these programmatic pipelines with Javascript code, which is synthesized into GLSL code and compiled and run on the GPU. This `dyno` system powers other components of Forge as well, such as `Readback` (which can perform any computation and read back the resulting value), used to compute the sort distance metric for pairs of Gsplats and read them back for CPU sorting.
|
||||
Reference in New Issue
Block a user