mirror of
https://github.com/storytold/spark.git
synced 2026-10-09 00:09:53 +00:00
Improve all docs. Add feature bullets to main forge.dev page. Rename Gsplat to splat in docs. Expand on Forge Overview. Add Next Steps section to Getting Started for people who don't see left pane links. Added more tips to Performance page. Updated docs re: optional ForgeRenderer. (#25)
This commit is contained in:
Andreas Sundquist
committed by
GitHub
GitHub
parent
0c8e5bd3b4
commit
7d62dd1a8a
@@ -10,7 +10,7 @@ Join the [Forge Discord](https://discord.gg/W39qmSKemS) to connect with other us
|
||||
|
||||
Forge can be used alongside or within React for declarative scene management, dynamic rendering, and state management between your user interface and the 3D scene. See the following examples for how to use Forge with React.
|
||||
|
||||
- [`forge-react-basic`](https://github.com/forge-gfx/forge-react-basic) - A basic example of creating a `<canvas>` and Three.js scene with Forge.
|
||||
- [`forge-react-r3f`](https://github.com/forge-gfx/forge-react-r3f) - Use Forge declaratively in React with [React Three Fiber](https://r3f.docs.pmnd.rs).
|
||||
- [`forge-react-router`](https://github.com/forge-gfx/forge-react-router) - An example of using Forge and React Three Fiber with [React Router](https://reactrouter.com) v7 framework mode with SSR.
|
||||
- [`forge-react-nextjs`](https://github.com/forge-gfx/forge-react-nextjs) - An example of using Forge and React Three Fiber with Next.js App Router.
|
||||
- [`forge-react-basic`](https://github.com/forge-gfx/forge-react-basic): A basic example of creating a `<canvas>` and Three.js scene with Forge.
|
||||
- [`forge-react-r3f`](https://github.com/forge-gfx/forge-react-r3f): Use Forge declaratively in React with [React Three Fiber](https://r3f.docs.pmnd.rs).
|
||||
- [`forge-react-router`](https://github.com/forge-gfx/forge-react-router): An example of using Forge and React Three Fiber with [React Router](https://reactrouter.com) v7 framework mode with SSR.
|
||||
- [`forge-react-nextjs`](https://github.com/forge-gfx/forge-react-nextjs): An example of using Forge and React Three Fiber with Next.js App Router.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Controls
|
||||
|
||||
A program using `Forge` can use any camera control scheme they want that is compatible with Three.js and will typically manipulate a `THREE.Camera` object's transform. `Forge` ships with simple, intuitive controls for navigating 3D space that use the keyboard + mouse, game pad, or mobile multi-touch. To add these controls, you can add:
|
||||
A program using `Forge` can use any camera control scheme that is compatible with Three.js and will typically manipulate a `THREE.Camera` object's transform. `Forge` also ships with simple, intuitive controls for navigating 3D space that use the keyboard + mouse, game pad, or mobile multi-touch. To add these controls, you can create a `ForgeControls` instance:
|
||||
|
||||
```typescript
|
||||
const controls = new ForgeControls({
|
||||
@@ -8,7 +8,7 @@ const controls = new ForgeControls({
|
||||
});
|
||||
|
||||
renderer.setAnimationLoop((time) => {
|
||||
renderer.render(scene, camera);)
|
||||
renderer.render(scene, camera);
|
||||
controls.update(camera);
|
||||
});
|
||||
```
|
||||
|
||||
+16
-16
@@ -1,11 +1,11 @@
|
||||
# ForgeRenderer
|
||||
|
||||
## Adding to your `THREE.Scene`
|
||||
## Optionally adding to your `THREE.Scene`
|
||||
|
||||
Using Forge begins with creating a `ForgeRenderer` object and adding it to your `THREE.Scene`. You can add it anywhere in the scene, for example at the root:
|
||||
Forge internally uses a `ForgeRenderer` object in your `THREE.Scene` to perform splat rendering. Forge will automatically create a `ForgeRenderer` and add it to your scene if you don't create one yourself. For more advanced use cases such as multiple viewpoints or rendering environment maps, you can create your own `ForgeRenderer` and add it anywhere in the scene, for example at the root:
|
||||
```typescript
|
||||
const forge = new ForgeRenderer({
|
||||
renderer: myThreeWebGlRenderer,
|
||||
renderer: myThreeJsWebGlRenderer,
|
||||
});
|
||||
const scene = new THREE.Scene();
|
||||
scene.add(forge);
|
||||
@@ -13,7 +13,7 @@ scene.add(forge);
|
||||
|
||||
## Larger scenes
|
||||
|
||||
All scene Gsplats are accumulated by SplatAccumulator into a single global PackedSplats, whose coordinates are relative to the ForgeRenderer's origin. Gsplats that are far away from this origin may exhibit float16 quantization artifacts, so if you plan on moving the camera large distances you can instead add the renderer as a child of your `THREE.Camera`, ensuring that coordinates near the camera viewpoint have higher precision:
|
||||
All scene splats are accumulated by SplatAccumulator into a single global PackedSplats, whose coordinates are relative to the ForgeRenderer's origin. Splats that are far away from this origin may exhibit float16 quantization artifacts, so if you plan on moving the camera large distances you can instead add `ForgeRenderer` as a child of your `THREE.Camera`, ensuring that coordinates will have higher precision near the camera viewpoint:
|
||||
```javascript
|
||||
const aspect = canvas.width / canvas.height;
|
||||
const camera = new THREE.PerspectiveCamera(75, aspect, 0.1, 1000);
|
||||
@@ -50,28 +50,28 @@ const forge = new ForgeRenderer({
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **clock** | Pass in a `THREE.Clock` to synchronize time-based effects across different systems. Alternatively, you can set the `ForgeRenderer` properties `time` and `deltaTime` directly. (default: `new THREE.Clock`)
|
||||
| **autoUpdate** | Controls whether to check and automatically update Gsplat collection after each frame render. (default: `true`)
|
||||
| **preUpdate** | Controls whether to update the Gsplats before or after rendering. For WebXR this *must* be false in order to complete rendering as soon as possible. (default: `false`)
|
||||
| **originDistance** | Distance threshold for ForgeRenderer movement triggering a Gsplat update at the new origin. (default: `1.0`)
|
||||
| **maxStdDev** | Maximum standard deviations from the center to render Gaussians. Values `Math.sqrt(5)`..`Math.sqrt(8)` produce good results and can be tweaked for performance. (default: `Math.sqrt(8)`)
|
||||
| **enable2DGS** | Enable 2D Gaussian splatting rendering ability. When this mode is enabled, any `scale` x/y/z component that is exactly `0` (minimum quantized value) results in the other two non-0 axis being interpreted as an oriented 2D Gaussian Splat, rather instead of the usual projected 3DGS Z-slice. When reading PLY files, scale values less than e^-20 will be interpreted as `0`. (default: `true`)
|
||||
| **preBlurAmount** | Scalar value to add to 2D splat covariance diagonal, effectively blurring + enlarging splats. In scenes trained without the Gsplat anti-aliasing tweak this value was typically 0.3, but with anti-aliasing it is 0.0 (default: `0.0`)
|
||||
| **blurAmount** | Scalar value to add to 2D splat covarianve diagonal, with opacity adjustment to correctly account for "blurring" when anti-aliasing. Typically 0.3 (equivalent to approx 0.5 pixel radius) in scenes trained with anti-aliasing.
|
||||
| **autoUpdate** | Controls whether to check and automatically update splat collection after each frame render. (default: `true`)
|
||||
| **preUpdate** | Controls whether to update the splats before or after rendering. For WebXR this *must* be false in order to complete rendering as soon as possible. (default: `false`)
|
||||
| **originDistance** | Distance threshold for `ForgeRenderer` movement triggering a splat update at the new origin. (default: `1.0`) This can be useful when your `ForgeRenderer` is a child of your camera and you want to retain high precision coordinates near the camera.
|
||||
| **maxStdDev** | Maximum standard deviations from the center to render Gaussians. Values `Math.sqrt(5)`..`Math.sqrt(9)` produce good results and can be tweaked for performance. (default: `Math.sqrt(8)`)
|
||||
| **enable2DGS** | Enable 2D Gaussian splatting rendering ability. When this mode is enabled, any `scale` x/y/z component that is exactly `0` (minimum quantized value) results in the other two non-zero axes being interpreted as an oriented 2D Gaussian Splat instead of the usual approximate projected 3DGS Z-slice. When reading PLY files, scale values less than e^-20 will be interpreted as `0`. (default: `true`)
|
||||
| **preBlurAmount** | Scalar value to add to 2D splat covariance diagonal, effectively blurring + enlarging splats. In scenes trained without the splat anti-aliasing tweak this value was typically 0.3, but with anti-aliasing it is 0.0 (default: `0.0`)
|
||||
| **blurAmount** | Scalar value to add to 2D splat covariance diagonal, with opacity adjustment to correctly account for "blurring" when anti-aliasing. Typically 0.3 (equivalent to approx 0.5 pixel radius) in scenes trained with anti-aliasing.
|
||||
| **falloff** | Modulate Gaussian kernel falloff. 0 means "no falloff, flat shading", while 1 is the normal Gaussian kernel. (default: `1.0`)
|
||||
| **clipXY** | X/Y clipping boundary factor for Gsplat centers against view frustum. 1.0 clips any centers that are exactly out of bounds, while 1.4 clips centers that are 40% beyond the bounds. (default: `1.4`)
|
||||
| **clipXY** | X/Y clipping boundary factor for splat centers against view frustum. 1.0 clips any centers that are exactly out of bounds (but the splat's entire projection may still be in bounds), while 1.4 clips centers that are 40% beyond the bounds. (default: `1.4`)
|
||||
| **view** | Configures the `ForgeViewpointOptions` for the default `ForgeViewpoint` associated with this `ForgeRenderer`. Notable option: `sortRadial` (sort by radial distance or Z-depth)
|
||||
|
||||
## `newViewpoint(options: ForgeViewpointOptions)`
|
||||
|
||||
Create a new `ForgeViewpoint` for this `ForgeRenderer`. Note that every `ForgeRenderer` has an initial `forge.defaultView: ForgeViewpoint` created during construction, which is used for default canvas rendering. Calling this method allows you to create additional viewpoints, which can be updated automatically each frame (performing Gsplat sorting every time there is an update), or updated on-demand for controlled rendering for video frame rendering or similar applications.
|
||||
Create a new `ForgeViewpoint` for this `ForgeRenderer`. Note that every `ForgeRenderer` has an initial `forge.defaultView: ForgeViewpoint` created during construction, which is used for default canvas rendering. Calling this method allows you to create additional viewpoints, which can be updated automatically each frame (performing splat sorting every time there is an update), or updated on-demand for controlled rendering for video frame rendering or similar applications.
|
||||
|
||||
## `update({ scene })`
|
||||
|
||||
If `forge.autoUpdate` is `false` then you must manually call `forge.update({ scene })` to have the scene Gsplats re-generated.
|
||||
If `forge.autoUpdate` is `false` then you must manually call `forge.update({ scene })` to have the scene splats re-generated.
|
||||
|
||||
## `renderEnvMap({ renderer, scene, worldCenter, ... })`
|
||||
|
||||
Renders out the scene to an environment map that can be used for image-based lighting or similar applications. First updates Gsplats, sorts them with respect to the provided `worldCenter`, renders 6 cube faces, then pre-filters them using `THREE.PMREMGenerator` and returns a `THREE.Texture` that can assigned directly to a `THREE.MeshStandardMaterial.envMap` property.
|
||||
Renders out the scene to an environment map that can be used for image-based lighting or similar applications. First updates splats, sorts them with respect to the provided `worldCenter`, renders 6 cube faces, then pre-filters them using `THREE.PMREMGenerator` and returns a `THREE.Texture` that can assigned directly to a `THREE.MeshStandardMaterial.envMap` property.
|
||||
|
||||
## `recurseSetEnvMap(root, envMap)`
|
||||
|
||||
@@ -79,7 +79,7 @@ Utility function to recursively set the `envMap` property for any `THREE.MeshSta
|
||||
|
||||
## `getRgba({ generator, ... })`
|
||||
|
||||
Utility function that helps extract the Gsplat RGBA values from a `SplatGenerator`, including the result of any real-time RGBA SDF edits applied to a `SplatMesh`. This effectively "bakes" any computed RGBA values, which can now be used as a pipeline input via `SplatMesh.splatRgba` to inject these baked values into the Gsplat data.
|
||||
Utility function that helps extract the splat RGBA values from a `SplatGenerator`, including the result of any real-time RGBA SDF edits applied to a `SplatMesh`. This effectively "bakes" any computed RGBA values, which can now be used as a pipeline input via `SplatMesh.splatRgba` to inject these baked values into the splat data.
|
||||
|
||||
## `readRgba({ generator, ...})`
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# ForgeViewpoint
|
||||
|
||||
A `ForgeViewpoint` is created from and tied to a `ForgeRenderer`, and represents an independent viewpoint of all the scene Gsplats and their sort order. Making these viewpoints explicit allows us to have multiple, simultaneous viewpoint renders, for example for camera preview panes or overhead map views.
|
||||
A `ForgeViewpoint` is created from and tied to a `ForgeRenderer`, and represents an independent viewpoint of all the scene splats and their sort order. Making these viewpoints explicit allows us to have multiple, simultaneous viewpoint renders, for example for camera preview panes or overhead map views.
|
||||
|
||||
When creating a `ForgeRenderer` it automatically creates a default viewpoint `.defaultView` that is used in the normal render loop when drawing to the canvas, and is automatically updated whenever the camera moves. Additional viewpoints can be created and configured separately:
|
||||
|
||||
@@ -30,7 +30,7 @@ const viewpoint = forge.newViewpoint({
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **autoUpdate** | Controls whether to auto-update its sort order whenever the ForgeRenderer updates the Gsplats. If you expect to render/display from this viewpoint most frames, set this to `true`. (default: `false`)
|
||||
| **autoUpdate** | Controls whether to auto-update its sort order whenever the ForgeRenderer updates the splats. If you expect to render/display from this viewpoint most frames, set this to `true`. (default: `false`)
|
||||
| **camera** | Set a `THREE.Camera` for this viewpoint to follow. (default: `undefined`)
|
||||
| **viewToWorld** | Set an explicit view-to-world transformation matrix for this viewpoint (equivalent to `camera.matrixWorld`), overrides any `camera` setting. (default: `undefined`)
|
||||
| **target** | Configure viewpoint with an off-screen render target. (default: `undefined`)
|
||||
@@ -41,8 +41,8 @@ const viewpoint = forge.newViewpoint({
|
||||
| **onTextureUpdated** | Callback function that is called when the render target texture is updated. Receives the texture as a parameter. Use this to update a viewport with the latest viewpoint render each frame. (default: `undefined`)
|
||||
| **sortRadial** | Whether to sort splats radially (geometric distance) from the viewpoint (true) or by Z-depth (false). Most scenes are trained with the Z-depth sort metric and will render more accurately at certain viewpoints. However, radial sorting is more stable under viewpoint rotations. (default: `true`)
|
||||
| **sortDistance** | Distance threshold for re-sorting splats. If the viewpoint moves more than this distance, splats will be re-sorted. (default: `0.01` units)
|
||||
| **sortCoorient** | View direction dot product threshold for re-sorting splats. For `sortRadial: true` we use 0.99 while `sortRadial: false` uses 0.999 because it is more sensitive to view direction. (default: `0.99` if `sortRadial` else `0.999`)
|
||||
| **depthBias** | Constant added to Z-depth to bias values into the positive range for `sortRadial: false`, but also used for culling Gsplats "well behind" the viewpoint origin (default: `1.0`)
|
||||
| **sortCoorient** | View direction dot product threshold for re-sorting splats. For `sortRadial: true` it defaults to 0.99 while `sortRadial: false` uses 0.999 because it is more sensitive to view direction. (default: `0.99` if `sortRadial` else `0.999`)
|
||||
| **depthBias** | Constant added to Z-depth to bias values into the positive range for `sortRadial: false`, but also used for culling splats "well behind" the viewpoint origin (default: `1.0`)
|
||||
| **sort360** | Set this to true if rendering a 360 to disable "behind the viewpoint" culling during sorting. This is set automatically when rendering 360 envMaps using the `ForgeRenderer.renderEnvMap()` utility function. (default: `false`)
|
||||
|
||||
## `dispose()`
|
||||
@@ -51,17 +51,17 @@ Call this when you are done with the `ForgeViewpoint` and want to free up its re
|
||||
|
||||
## `setAutoUpdate(autoUpdate: boolean)`
|
||||
|
||||
Use this function to change whether this viewpoint will auto-update its sort order whenever the attached `ForgeRenderer` updates the Gsplats. Turn this on or off depending on whether you expect to do renders from this viewpoint most frames.
|
||||
Use this function to change whether this viewpoint will auto-update its sort order whenever the attached `ForgeRenderer` updates the splats. Turn this on or off depending on whether you expect to do renders from this viewpoint for most frames.
|
||||
|
||||
## `async prepareRenderPixels({ scene, camera?, viewToWOrld?, update?, forceOrigin? })`
|
||||
|
||||
Render out a viewpoint as a Uint8Array of RGBA values for the provided scene and any `camera`/`viewToWorld` viewpoint overrides. By default `update` is `true`, which triggers its `ForgeRenderer` to check and potentially update the Gsplats. Setting `update` to `false` disables this and sorts the Gsplats as they are. Setting `forceOrigin` (default: `false`) to `true` forces the view update to recalculate the splats with this view origin, potentially altering any view-dependent effects. If you expect view-dependent effects to play a role in the rendering quality, enable this.
|
||||
Render out a viewpoint as a Uint8Array of RGBA values for the provided scene and any `camera`/`viewToWorld` viewpoint overrides. By default `update` is `true`, which triggers its `ForgeRenderer` to check and potentially update the splats. Setting `update` to `false` disables this and sorts the splats as they are. Setting `forceOrigin` (default: `false`) to `true` forces the view update to recalculate the splats with this view origin, potentially altering any view-dependent effects. If you expect view-dependent effects to play a role in the rendering quality, enable this.
|
||||
|
||||
Underneath, `prepareRenderPixels()` simply calls `await this.prepare(...)`, `this.renderTarget(...)`, and finally returns the result `this.readTarget()`, a Promise to a Uint8Array with RGBA values for all the pixels (potentially downsampled if the `superXY` parameter was used). These steps can also be called manually, for example if you need to alter the scene before and after `this.renderTarget(...)` to hide UI elements from being rendered.
|
||||
Underneath, `prepareRenderPixels()` simply calls `await this.prepare(...)`, `this.renderTarget(...)`, and finally returns the result of `this.readTarget()`, a Promise to a Uint8Array with RGBA values for all the pixels (potentially downsampled if the `superXY` parameter was used). These steps can also be called manually, for example if you need to alter the scene before and after `this.renderTarget(...)` to hide UI elements from being rendered.
|
||||
|
||||
## `async prepare({ scene, camera?, viewToWorld?, update?, forceOrigin? })`
|
||||
|
||||
See above `async prepareRenderPixels()` for explanation of parameters. Awaiting this method updates the Gsplats in the scene and performs a sort of the Gsplats from this viewpoint, preparing it for a subsequent `this.renderTarget()` call in the same tick.
|
||||
See above `async prepareRenderPixels()` for explanation of parameters. Awaiting this method updates the splats in the scene and performs a sort of the splats from this viewpoint, preparing it for a subsequent `this.renderTarget()` call in the same tick.
|
||||
|
||||
## `renderTarget({ scene, camera? })`
|
||||
|
||||
@@ -78,4 +78,3 @@ This is called automatically by `ForgeRenderer`, there is no need to call it! Th
|
||||
## `ForgeViewpoint.EMPTY_TEXTURE`
|
||||
|
||||
If you need an empty `THREE.Texture` to use to initialize a uniform that is updated via `onTextureUpdated(texture)`, this static texture can be handy.
|
||||
|
||||
|
||||
@@ -51,3 +51,11 @@ npm run dev
|
||||
|
||||
This will run a Web server at [http://localhost:8080/](http://localhost:8080/) with the examples.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Read more about Forge from the left navigation panel (expand browser window to see it), or follow one of the links below:
|
||||
|
||||
- [Learn why Forge exists and its feature set](overview.md)
|
||||
- [Forge system design overview](system-design.md)
|
||||
- [Add a SplatMesh to your scene](splat-mesh.md)
|
||||
- [Community resources](community-resources.md)
|
||||
@@ -1,18 +1,19 @@
|
||||
# Loading Gsplats
|
||||
# Loading Splats
|
||||
|
||||
Forge provides loaders for most popular Gsplat file formats, including `.ply` (original "gsplat" format, some compressed variants, and plain x/y/z/r/g/b point clouds) and `.spz` (Niantic open source compressed format), both auto-detected from the file contents.
|
||||
Forge provides loaders for most popular splat file formats, including `.ply` (original "gsplat" format, compressed SuperSplat variant, and plain x/y/z/r/g/b point clouds) and `.spz` (Niantic open source compressed format), both auto-detected from the file contents.
|
||||
|
||||
Forge can also load popular formats `.splat` (from `antimatter15/splat`) and `.ksplat` (from `mkkellogg/GaussianSplats3D`) if the file type can be inferred from the URL/path extension, or set explicitly using the `fileType` property when creating a `SplatMesh` or `PackedSplats`.
|
||||
Forge can also load formats `.splat` (from [`antimatter15/splat`](https://github.com/antimatter15/splat)) and `.ksplat` (from [`mkkellogg/GaussianSplats3D`](https://github.com/mkkellogg/GaussianSplats3D)) if the file type can be inferred from the URL/path extension, or set explicitly using the `fileType` property when creating a `SplatMesh` or `PackedSplats`.
|
||||
|
||||
## Loading auto-detectable formats `.ply` and `.spz`
|
||||
|
||||
Adding an individual `SplatMesh` from an auto-detectable format is easy and can be done as simply as below:
|
||||
Adding an individual `SplatMesh` from an auto-detectable format is easy and can be done as simply as:
|
||||
|
||||
```javascript
|
||||
// Load and create SplatMesh in one go
|
||||
const splats = new SplatMesh({ url: "./butterfly.ply" });
|
||||
scene.add(splats);
|
||||
|
||||
// No file extension but we can auto-detect format from the contents
|
||||
scene.add(new SplatMesh({ url: "plyBin/0123456789abcdef" }));
|
||||
scene.add(new SplatMesh({ url: "spzBin/fedcba9876543210" }));
|
||||
```
|
||||
@@ -60,7 +61,7 @@ loader.loadAsync(url, (event) => {
|
||||
|
||||
## Loading additional formats `.splat` and `.ksplat`
|
||||
|
||||
These formats are reliably auto-detected from the file contents, so we use two fall-back mechanism to enable support for these popular formats.
|
||||
These formats cannot be reliably auto-detected from the file contents, so we use two fall-back mechanism to enable support for these formats.
|
||||
|
||||
First, the auto-detection fails on these files, which triggers file type inference via URL/path file extension. If the URL contains the `.splat` or `.ksplat` extensions (stripping out query parameters etc.), we set the corresponding file type.
|
||||
|
||||
@@ -71,9 +72,11 @@ scene.add(splats);
|
||||
const ksplats = new SplatMesh({ url: "./butterfly.ksplats" });
|
||||
scene.add(ksplats);
|
||||
```
|
||||
|
||||
If the URL contains a path with no obvious file extension, you can set the field `fileType` when constructing a `SplatMesh` or `PackedSplats`.
|
||||
|
||||
```javascript
|
||||
// File type can't be auto-detected, so we set it explicitly
|
||||
scene.add(new SplatMesh({
|
||||
url: "splatBin/0123456789abcdef",
|
||||
fileType: SplatFileType.SPLAT,
|
||||
|
||||
+16
-9
@@ -1,17 +1,24 @@
|
||||
|
||||
*Forge aims to expand what's possible with Gaussian Splatting and help 3D/4D creators bring their visions to life and share it with others.*
|
||||
|
||||
# Overview
|
||||
|
||||
Forge is a dynamic Gaussian splat (Gsplat) renderer built on Three.js. It renders Gsplat-based worlds and objects into Three.js scenes, giving you the ability to fuse AI-generated with photogrammetry and regular triangle meshes. Forge is also programmable and fully dynamic, giving you unprecedented control over how Gsplat elements are generated and rendered into the scene.
|
||||
3D Gaussian Splatting has emerged as a frontrunner in generative AI and 3D reconstruction. By representing 3D scenes and objects as collections of tiny oriented Gaussian-shaped blobs (aka "splats"), machine learning techniques can be used to create detailed, photorealistic 3D content that can be rendered in real-time. However, 3DGS is a relatively new technique that that can't be used in many traditional triangle-based 3D mesh rendering engines. Tools for creating, editing, and rendering 3DGS are in their infancy, mostly able to only work with a single, static 3DGS object at time.
|
||||
|
||||
The web's most popular 3D graphics library Three.js, can't render 3DGS directly. Although libraries exist to render 3DGS on the web, they each come with different limitations and don't treat 3DGS objects as first-class citizens in the scene hierarchy. Limitations include: rendering only one 3DGS object at time, incorrect occlusion between 3DGS objects, inability to dynamically modify the objects, requiring WebGPU, or slow/laggy rendering. We believe 3DGS will play an important role in future 3D/4D content creation, and we built Forge to make it easy to incorporate 3DGS into online experiences.
|
||||
|
||||
## Forge
|
||||
|
||||
Forge is a dynamic 3DGS renderer built for Three.js and WebGL2 that runs in any web browser (desktop, mobile, and WebXR). With a handful of lines of code, anyone using Three.js can easily add 3DGS to their scenes (even by vibe coding!). By creating one or more `SplatMesh` objects and adding them to your scene, Forge will render these alongside traditional triangle-based meshes during your regular `render(scene, camera)` call. `SplatMesh` derives from `THREE.Object3D` and can be translated and rotated like any other object, placed arbitrarily in the scene hierarchy, and animated by adjusting the values each frame. A `SplatMesh` can be created from most splat file formats via the `url` parameter or by directly creting the splats one by one.
|
||||
|
||||
3DGS is still in its infancy, and we expect new techniques will be developed for recoloring/relighting, animation, transitions, and other creative or interactive effects. We designed Forge to be a programmable splat engine from the ground up, giving you unprecendented control over how individual splats are generated, animated, and rendered into the scene. Similar to shader graph systems in modern 3D graphics engines, Forge allows you to compose blocks of functions (called `Dyno`s) into computation graphs that can generate splats procedurally, modify them arbirarily, or anything other computation you can imagine, and will be converted to GLSL to run on the GPU.
|
||||
|
||||
## Features
|
||||
- Render multiple splat objects together with correct sorting
|
||||
- Integrates with Three.js rendering pipeline to fuse Gsplat + mesh-based objects
|
||||
- Integrates with Three.js rendering pipeline to fuse splat and mesh-based objects
|
||||
- Portable: Works across almost all devices, targeting 98%+ WebGL2 support
|
||||
- Fast! Renders fast even on low-powered mobile devices
|
||||
- File format support: all major formats supported including .PLY (including compressed), .SPZ, .SPLAT, .KSPLAT
|
||||
- Renders fast even on low-powered mobile devices
|
||||
- Render multiple splat objects together with correct sorting
|
||||
- Most major splat file formats supported including .PLY (also compressed), .SPZ, .SPLAT, .KSPLAT
|
||||
- Render multiple viewpoints simultaneously
|
||||
- Fully dynamic: each Gsplat can be transformed and edited for animation
|
||||
- Real-time Gsplat color editing and skeletal animation
|
||||
- Shader graph system to dynamically create/edit Gsplats on the GPU
|
||||
- Fully dynamic: each splat can be transformed and edited for animation
|
||||
- Real-time splat color editing, displacement, and skeletal animation
|
||||
- Shader graph system to dynamically create/edit splats on the GPU
|
||||
|
||||
+35
-33
@@ -1,6 +1,6 @@
|
||||
# PackedSplats
|
||||
|
||||
A `PackedSplats` is a collection of Gaussian splats, packed into a format that takes exactly 16 bytes per Gsplat to maximize memory and cache efficiency. The `center` xyz coordinates are encoded as float16 (3 x 2 bytes), `scale` xyz as 3 x uint8 that encode a log scale from e^-9 to e^9, `rgba` as 4 x uint8, and quaternion encoded via axis+angle using 2 x uint8 for octahedral encoding of the axis direction and a uint8 to encode rotation amount from 0..Pi.
|
||||
A `PackedSplats` is a collection of Gaussian splats, packed into a format that takes exactly 16 bytes per splat to maximize memory and cache efficiency. The `center` xyz coordinates are encoded as float16 (3 x 2 bytes), `scale` xyz as 3 x uint8 that encode a log scale from e^-9 to e^9, `rgba` as 4 x uint8, and quaternion encoded via axis+angle using 2 x uint8 for octahedral encoding of the axis direction and a uint8 to encode rotation amount from 0..Pi.
|
||||
|
||||
## Creating a `PackedSplats`
|
||||
|
||||
@@ -27,7 +27,7 @@ const packedSplats = new PackedSplats({
|
||||
|
||||
### Optional parameters
|
||||
|
||||
Like for `SplatMesh` you can create a `new PackedSplats()` with no options, will create a new empty instance with 0 Gsplats. Similarly, you can provide an input `url`, `fileBytes`, `fileType` to decode from a file source. You can also create a `PackedSplats` from a raw `Uint32Array` where each successive 4 Uint32 values encodes one "packed" Gsplat. Finally, a `construct(splats)` callback provides an ergonomic way to create Gsplats procedurally with an in-line callback closure.
|
||||
Like for `SplatMesh` you can create a `new PackedSplats()` with no options, which will create a new empty instance with 0 splats. Similarly, you can provide an input `url` or `fileBytes` to decode from a file source. You can also create a `PackedSplats` from a raw `Uint32Array` where each successive 4 Uint32 values encodes one "packed" splat. Finally, a `construct(splats)` callback provides an ergonomic way to create splats procedurally with an in-line initialization closure.
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
@@ -35,7 +35,7 @@ Like for `SplatMesh` you can create a `new PackedSplats()` with no options, will
|
||||
| **fileBytes** | Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: `undefined`)
|
||||
| **fileType** | Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: `undefined` auto-detects other formats from file contents)
|
||||
| **maxSplats** | Reserve space for at least this many splats when constructing the collection initially. The array will automatically resize past maxSplats so setting it is an optional optimization. (default: `0`)
|
||||
| **packedArray** | Use provided packed data array, where each 4 consecutive uint32 values encode one "packed" Gsplat. (default: `undefined`)
|
||||
| **packedArray** | Use provided packed data array, where each 4 consecutive uint32 values encode one "packed" splat. (default: `undefined`)
|
||||
| **numSplats** | Override number of splats in packed array to use only a subset. (default: length of packed array / 4)
|
||||
| **construct** | Callback function to programmatically create splats at initialization. (default: `undefined`)
|
||||
| **extra** | Additional splat data, such as spherical harmonics components (sh1, sh2, sh3). (default: `{}`)
|
||||
@@ -55,12 +55,12 @@ utils.setPackedSplat(packedSplats.packedArray, index, x, y, z, scaleX, scaleY, .
|
||||
// Set rotation components of underlying Uint32 array directly
|
||||
utils.setPackedSplatQuat(packedSplats.packedArray, index, quatX, quatY, quatZ, quatW);
|
||||
|
||||
// Unpack all Gsplat components from the Uint32 array
|
||||
// Unpack all splat components from the Uint32 array
|
||||
const { center, scales, quaternion, color, opacity } = utils.unpackSplat(packedSplats.packedArray, index);
|
||||
|
||||
// Unpack all Gsplats with callback
|
||||
// Unpack all splats with callback
|
||||
packedSplats.forEachSplat((index, center, scales, quaternion, opacity, color) => {
|
||||
// Use unpacked Gsplat data. Changing the inputs directly has no effect.
|
||||
// Use unpacked splat data. Changing the inputs directly has no effect.
|
||||
// Update just the scales component
|
||||
utils.setPackedSplatScales(packedSplat.packedArray, index, 0.005, 0.01, 0.015);
|
||||
// Update the entire splat
|
||||
@@ -68,15 +68,17 @@ packedSplats.forEachSplat((index, center, scales, quaternion, opacity, color) =>
|
||||
});
|
||||
```
|
||||
|
||||
In GLSL / `dyno` shader contexts you can use the following utility functions that are available via `splatDefines.glsl`:
|
||||
|
||||
In GLSL code you can use the following utility functions that are available via `splatDefines.glsl`, which is included in all GLSL shader programs:
|
||||
```glsl
|
||||
// Pack a Gsplat into a uvec4
|
||||
// Pack a splat into a uvec4
|
||||
uvec4 packSplat(vec3 center, vec3 scales, vec4 quaternion, vec4 rgba);
|
||||
|
||||
// Unpack a Gsplat from a uvec4
|
||||
// Unpack a splat from a uvec4
|
||||
void unpackSplat(uvec4 packed, out vec3 center, out vec3 scales, out vec4 quaternion, out vec4 rgba);
|
||||
```
|
||||
|
||||
In `dyno` shader contexts you can read and unpack a splat from a `PackedSplats`:
|
||||
```javascript
|
||||
// Fetch and unpack a particular index from a PackedSplats.
|
||||
const gsplat = dyno.readPackedSplat(packedSplats.dyno, index);
|
||||
```
|
||||
@@ -101,61 +103,61 @@ Each `PackedSplat` occupies 16 bytes (4 × `uint32`), with the following layout
|
||||
| 14 | scale.z | 1 | Z scale, log-encoded to uint8 |
|
||||
| 15 | quat angle (θ) | 1 | Encoded quaternion rotation angle (uint8, θ/π·255) |
|
||||
|
||||
### Gsplat RGBA encoding
|
||||
### Splat RGBA encoding
|
||||
|
||||
RGB values are encoded are uint8 sRGB values with 0..255 mapping to 0..1. When loading from a PLY file these values are derived by calculating `ply[f_dc_0] * SH_C0 + 0.5`.
|
||||
|
||||
Opacity is encoded on a linear scale where 0..255 maps to 0..1.
|
||||
|
||||
### Gsplat center encoding
|
||||
### Splat center encoding
|
||||
|
||||
The center x/y/z components are encoded as float16, which provides 10 bits of mantissa, or approximately 1K steps (0.1%) of resolution each power of 2 exponent, with a range of up to 32K in distance. If most of the Gsplats are positioned relative to the origin this provides enough positional resolution. Gsplats that are transformed far from the origin, however (for example when bringing multiple `SplatMesh`es together in a scene that are far apart) may lose precision when mapped to the space of `ForgeRenderer`. For scenes where the user camera may move far from the origin, you may want to tie the `ForgeRenderer` origin to your camera by adding it as a child of the camera.
|
||||
The center x/y/z components are encoded as float16, which provides 10 bits of mantissa, or approximately 1K steps (0.1%) of resolution between each successive power of 0 from the origin, with a range of up to 32K in distance. If most of the splats are positioned relative to the origin this provides enough positional resolution. Splats that are transformed far from the origin, however (for example when bringing multiple `SplatMesh`es together in a scene that are far apart) may lose precision when mapped to the space of `ForgeRenderer`. For scenes where the user camera may move far from the origin, you may want to tie the `ForgeRenderer` origin to your camera by adding it as a child of the camera.
|
||||
|
||||
### Gsplat scales encoding
|
||||
### Splat scales encoding
|
||||
|
||||
The XYZ scales are encoded independently using the following mapping: Any scale values below e^-20 are interpreted as "true zero" scale, and encoded as `uint8(0)`. Any other values quantized by computing `ln(scale_xyz)`, mapping the range e^-9..e^9 to uint8 values 1..254, rounding, and clamping. This logarithmic scale range provides values from 0.0001 up to 8K in scale, with approximately 7% steps between discrete values. Seems to have minimal impact on perceptible visual quality.
|
||||
The XYZ scales are encoded independently using the following mapping: Any scale values below e^-20 are interpreted as "true zero" scale, and encoded as `uint8(0)`. Any other values quantized by computing `ln(scale_xyz)`, mapping the range e^-9..e^9 to uint8 values 1..255, rounding, and clamping. This logarithmic scale range can encode values from 0.0001 up to 8K in scale, with approximately 7% steps between discrete sizes, and has minimal impact on perceptible visual quality.
|
||||
|
||||
### Gsplat orientation encoding
|
||||
### Splat orientation encoding
|
||||
|
||||
We encode a Gsplat's quaternion/orientation by encoding it explicitly in an axis + angle representation: 8 bits for each U/V coordinate for octahedral encoding of the axis direction, and 8 bits to encode the rotation angle range 0..Pi.
|
||||
We encode a splat's quaternion/orientation by encoding it explicitly in an axis + angle representation: 8 bits for each U/V coordinate for octahedral encoding of the axis direction, and 8 bits to encode the rotation angle range 0..Pi.
|
||||
|
||||
This representation was chosen oven other internal rotation representations because it provided a good mix of speed/simplicity, uniformity of representable orientations, and especially its ability to handle rotations "near identity". Other encodings such as the more common "3 quaternion components" tend to have poor rotational resolution around I(1), which is a particularly important area of that parameter.
|
||||
This representation was chosen over other internal rotation representations because it provides a good mix of speed/simplicity, uniformity of representable orientations, and especially its ability to handle rotations "near identity". Other encodings such as the more common "3 quaternion components" tend to have poor rotational resolution around I(1), which is a particularly important regime of this parameter.
|
||||
|
||||
### `extra` Gsplat data
|
||||
### `extra` splat data
|
||||
|
||||
Each instance of `PackedSplats` also has a property `extra: Record<string, unknown>` that is used to attach additional Gsplat-related data to the `PackedSplats` container. For example, spherical harmonics degrees 1..3 are stored in `sh1: Uint32Array(numSplats * 2)`, `sh2: Uint32Array(numSplats * 4)`, `sh3: Uint32Array(numSplats * 4)`, and intermediate textures are generated by `SplatMesh` and stored as `sh1Texture` etc.
|
||||
Each instance of `PackedSplats` also has a property `extra: Record<string, unknown>` that is used to attach additional splat-related data to the `PackedSplats` container. For example, spherical harmonics degrees 1..3 are stored in `sh1: Uint32Array(numSplats * 2)`, `sh2: Uint32Array(numSplats * 4)`, `sh3: Uint32Array(numSplats * 4)`, and intermediate textures are generated by `SplatMesh` and stored as `sh1Texture` etc.
|
||||
|
||||
This structure can be used to extend and store additional data in a `PackedSplats`, but there is no specific convention yet to prevent collisions.
|
||||
|
||||
`sh1` stores each of 3 x 3 RGB signed components as Sint7 (mapping -1..1) in 63 bits (8 bytes) per Gsplat. `sh2` stores each of 5 x 3 RGB signed components as Sint8 in 120 bits (16 bytes). `sh3` stores each of 7 x 3 RGB signed components as Sint6 in 126 bits (16 bytes) to improve memory bandwidth efficiency.
|
||||
`sh1` stores each of 3 x 3 RGB signed components as Sint7 (mapping -1..1) in 63 bits (8 bytes) per splat. `sh2` stores each of 5 x 3 RGB signed components as Sint8 in 120 bits (16 bytes). `sh3` stores each of 7 x 3 RGB signed components as Sint6 in 126 bits (16 bytes) to improve memory bandwidth efficiency. Note that using spherical harmonics dramatically increases the memory footprint from 16 bytes per splat up to 56 bytes per splat with SH0..3, which can impact rendering performance.
|
||||
|
||||
## `PackedSplats` instance methods
|
||||
|
||||
### `dispose()`
|
||||
|
||||
Call this when you are finished with the PackedSplats and want to free any buffers it holds.
|
||||
Call this when you are finished with the PackedSplats and want to free any render targets + textures it holds.
|
||||
|
||||
### `ensureSplats(numSplats)`
|
||||
|
||||
Ensures that `this.packedArray` can fit `numSplats` Gsplats. If it's too small, resize exponentially and copy over the original data.
|
||||
Ensures that `this.packedArray` can fit `numSplats` splats. If it's too small, resize exponentially and copy over the original data.
|
||||
|
||||
Typically you don't need to call this, because calling `this.setSplat(index, ...)` and `this.pushSplat(...)` will automatically call `ensureSplats()` so we have enough splats.
|
||||
|
||||
### `getSplat(index): { center, scales, quaternion, opacity, color }`
|
||||
|
||||
Unpack the 16-byte Gsplat data at `index` into the Three.js components `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number 0..1`, `color: THREE.Color 0..1`.
|
||||
Unpack the 16-byte splat data at `index` into the Three.js components `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number 0..1`, `color: THREE.Color 0..1`.
|
||||
|
||||
### `setSplat(index, center, scales, quaternion, opacity, color)`
|
||||
|
||||
Set all PackedSplat components at `index` with the provided Gsplat attributes (can be the same objects returned by `getSplat`). Ensures there is capacity for at least `index+1` Gsplats.
|
||||
Set all PackedSplat components at `index` with the provided splat attributes (can be the same objects returned by `getSplat`). Ensures there is capacity for at least `index+1` splats.
|
||||
|
||||
### `pushSplat(center, scales, quaternion, opacity, color)`
|
||||
|
||||
Effectively calls `this.setSplat(this.numSplats++, center, ...)`, useful on construction where you just want to iterate and create a collection of Gsplats.
|
||||
Effectively calls `this.setSplat(this.numSplats++, center, ...)`, useful on construction where you just want to iterate and create a collection of splats.
|
||||
|
||||
### `forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)`
|
||||
|
||||
Iterate over Gsplats index `0..=(this.numSplats-1)`, unpack each Gsplat and invoke the callback function with the Gsplat attributes.
|
||||
Iterate over splats index `0..=(this.numSplats-1)`, unpack each splat and invoke the callback function with the splat attributes.
|
||||
|
||||
### `getTexture()`
|
||||
|
||||
@@ -165,11 +167,11 @@ Returns a `THREE.DataArrayTexture` representing the PackedSplats content as a Ui
|
||||
|
||||
Can be used where you need an uninitialized `THREE.DataArrayTexture` like a uniform you will update with the result of `this.getTexture()` later.
|
||||
|
||||
## Generating Gsplats on the GPU
|
||||
## Generating splats on the GPU
|
||||
|
||||
To generate a large number of Gsplats we can use the `dyno` shader graph system, which allows to create a computation graph mapping `{ index: DynoVal<"int"> }` to `{ gsplat: DynoVal<Gsplat> }` via Javascript code, then have that synthesize GLSL code, which is finally compiled and executed in parallel on the GPU.
|
||||
To generate a large number of splats we can use the `dyno` shader graph system, which allows you to create a computation graph mapping `{ index: DynoVal<"int"> }` to `{ gsplat: DynoVal<Gsplat> }` via Javascript code, then have that synthesize GLSL code, which is finally compiled and executed in parallel on the GPU.
|
||||
|
||||
This building block is used by `ForgeRenderer` to traverse each visible `SplatMesh`/`SplatGenerator` and have it "generate" its Gsplats into the global `PackedSplats` array managed by a `SplatAccumulator`. At its core a `PackedSplats` has the ability to run `dyno` computation graphs to produce its contents using the following methods, which are typically managed by `ForgeRenderer`:
|
||||
This building block is used by `ForgeRenderer` to traverse each visible `SplatMesh`/`SplatGenerator` and have it "generate" its splats into the global `PackedSplats` array managed by a `SplatAccumulator`. At its core a `PackedSplats` has the ability to run `dyno` computation graphs to produce its contents using the following methods, which are typically managed by `ForgeRenderer`:
|
||||
|
||||
### `generateMapping(splatCounts: number[]): { maxSplats, mapping[] }`
|
||||
|
||||
@@ -177,12 +179,12 @@ Given an array of splatCounts (`.numSplats` for each `SplatGenerator`/`SplatMesh
|
||||
|
||||
### `ensureGenerate(maxSplats)`
|
||||
|
||||
Ensures our `PackedSplats.target` render target has enough space to generate `maxSplats` total Gsplats, and reallocate if not large enough.
|
||||
Ensures our `PackedSplats.target` render target has enough space to generate `maxSplats` total splats, and reallocate if not large enough.
|
||||
|
||||
### `generate({ generator, base, count, ... })`
|
||||
|
||||
Executes a `dyno` program specified by `generator` which is any `DynoBlock` that maps `{ index: "int" }` to `{ gsplat: Gsplat }`. This is called in `ForgeRenderer.updateInternal()` to re-generate Gsplats in the scene for `SplatGenerator` instances whose version is newer than what was generated for it last time.
|
||||
Executes a `dyno` program specified by `generator` which is any `DynoBlock` that maps `{ index: "int" }` to `{ gsplat: Gsplat }`. This is invoked from `ForgeRenderer.updateInternal()` to re-generate splats in the scene for `SplatGenerator` instances whose version is newer than last generated version.
|
||||
|
||||
## Using dynamic PackedSplats inputs in `dyno`
|
||||
|
||||
You can use a `PackedSplats` as a `dyno` block using the function `dyno.readPackedSplats(packedSplats.dyno, dynoIndex)` where `dynoIndex` is of type `DynoVal<"int">` If you need to be able to change the input `PackedSplats` dynamically, however, you should create a `DynoPackedSplats`, whose property `packedSplats` you can change to any `PackedSplats` and that will be used in the `dyno` shader program.
|
||||
You can use a `PackedSplats` in a `dyno` block using the function `dyno.readPackedSplats(packedSplats.dyno, dynoIndex)` where `dynoIndex` is of type `DynoVal<"int">` If you need to be able to change the input `PackedSplats` dynamically, however, you should create a `DynoPackedSplats`, whose property `packedSplats` you can set to any `PackedSplats`, which will be used in the next `dyno` shader program execution.
|
||||
|
||||
@@ -1,16 +1,24 @@
|
||||
# Performance Tuning
|
||||
|
||||
Rendering millions of Gsplats at 60+ fps can be a demanding task, especially for mobile-class GPUs. Each Gsplat is rendered as two triangles that span the footprint of a Gaussian up to `sqrt(8)` standard deviations (default value) from the center. Each Gsplat is rendered as a transparent object and must be blended back-to-front.
|
||||
Rendering millions of splats at 60+ fps can be a demanding task, especially for mobile-class GPUs. Each splat is rendered as two triangles that span the footprint of a Gaussian up to `sqrt(8)` standard deviations (default value) from the center. Each splat is rendered as a transparent object and must be blended back-to-front.
|
||||
|
||||
As a quick rule-of-thumb, the following "Gsplat budgets" are recommended:
|
||||
As a quick rule-of-thumb, the following "splat budgets" are recommended:
|
||||
|
||||
- Quest 3: 1 million Gsplats or less, not too many Gsplats concentrated in a small area
|
||||
- Android phone: 1-2 million Gsplats
|
||||
- iPhone: 1-3 million Gsplats
|
||||
- Computer: 1-5 million Gsplats (10-20+ million on some desktops)
|
||||
- Quest 3: 1 million splats or less, not too many splats concentrated in a small area
|
||||
- Android phone: 1-2 million splats
|
||||
- iPhone: 1-3 million splats
|
||||
- Computer: 1-5 million splats (10-20+ million on some desktops)
|
||||
|
||||
Each Gsplat incurs overhead in transforming it via SplatAccumulator for sorting and rendering, and at around 1 million Gsplats this becomes a bottleneck on some systems. Unintuitively, when a large number of Gsplats are concentrated in a small area (for example 500K Gsplats from a Trellis object at a small screen scale) they can bottleneck the GPU's rendering and blending ability.
|
||||
Each splat incurs overhead in transforming it via SplatAccumulator for sorting and rendering, and at around 1 million splats this becomes a bottleneck on some systems. Unintuitively, when a large number of splats are concentrated in a small area (for example 500K splats from a Trellis object at a small screen scale) they can bottleneck the GPU's rendering and blending ability.
|
||||
|
||||
## maxStdDev
|
||||
## `maxStdDev`
|
||||
|
||||
Adjust `ForgeRenderer.maxStdDev` (either directly on `ForgeRenderer` or via constructor options) to a value less than the default `Math.sqrt(8)`. This limits the extent of the Gaussian fall-off, which by default is approx 2.8. For VR a good value is `Math.sqrt(5)`, which is perceptually very similar to the default.
|
||||
Adjust `ForgeRenderer.maxStdDev` (either directly on `ForgeRenderer` or via constructor options) to a value less than the default `Math.sqrt(8)`. This limits the extent of the Gaussian fall-off, which by default is approx 2.8. For VR a good value is `Math.sqrt(5)`, which is perceptually very similar to the default.
|
||||
|
||||
## `THREE.WebGLRenderer.antialias`
|
||||
|
||||
When constructing a `THREE.WebGLRenderer` you should set `antialias: false` (default value) to avoid the overhead of multisampling. Rendering splats doesn't benefit from multisampling, and adds a significant amount of overhead when drawing millions of splats.
|
||||
|
||||
## `renderer.setPixelRatio(window.devicePixelRatio)`
|
||||
|
||||
Although `ForgeRenderer` is designed to work with any `devicePixelRatio`, it may impact performance due to the increased number of pixels to render and blend. If your scene consists of mostly splats, consider whether you have enough splats to justify the high DPI rendering.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Procedural Splats
|
||||
|
||||
Forge makes it easy to create Gaussian splat collections procedurally, and includes some splat constructors that may be useful for tasks like creating a grid or text made of splats. The example "Procedural Splats" puts some of these to use in a scene.
|
||||
Forge makes it easy to create Gaussian splat collections procedurally, and includes some splat constructors that are useful for common tasks like creating a grid or text made of splats. The example "Procedural Splats" puts some of these to use in a scene.
|
||||
|
||||
## Adding splats to a collection
|
||||
|
||||
@@ -16,7 +16,9 @@ splats.pushSplat(center, scales, quaternion, opacity, color);
|
||||
...
|
||||
```
|
||||
|
||||
The array in `PackedSplats` will be resized automatically to fit any splats you add. Alternatively, you can use the `construct` initializer callback:
|
||||
Note that calling `.pushSplat` and `.setSplat` will automatically resize the `PackedSplats` array to fit the splat. Additionally, the center, scales, etc parameters are encoded into 16 bytes, and the original objects passed in (such as `center: THREE.Vector3`) are no longer used. Re-using these objects for all the splats is a good idea.
|
||||
|
||||
Alternatively, you can use the `construct` initializer callback:
|
||||
```javascript
|
||||
const splats = new PackedSplats({
|
||||
construct: (splats) => {
|
||||
@@ -79,10 +81,10 @@ scene.add(grid);
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `stepSize` | `1` | step size along each grid axis |
|
||||
| `pointRadius` | `0.01` | spherical radius of each Gsplat |
|
||||
| `pointShadowScale` | `2.0` | relative size of the "shadow copy" of each Gsplat placed behind it |
|
||||
| `opacity` | `1.0` | Gsplat opacity |
|
||||
| `color` | RGB-modulated grid | Gsplat color (THREE.Color) or function to set color for position: ((THREE.Color, THREE.Vector3) => void) |
|
||||
| `pointRadius` | `0.01` | spherical radius of each splat |
|
||||
| `pointShadowScale` | `2.0` | relative size of the "shadow copy" of each splat placed behind it |
|
||||
| `opacity` | `1.0` | splat opacity |
|
||||
| `color` | RGB-modulated grid | splat color (THREE.Color) or function to set color for position: ((THREE.Color, THREE.Vector3) => void) |
|
||||
|
||||
|
||||
## XYZ axis
|
||||
@@ -107,13 +109,13 @@ scene.add(axes);
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `scale` | `0.25` | scale (Gsplat scale along axis) |
|
||||
| `axisRadius` | `0.0075` | radius of the axes (Gsplat scale orthogonal to axis) |
|
||||
| `axisShadowScale` | `2.0` | relative size of the "shadow copy" of each Gsplat placed behind it |
|
||||
| `scale` | `0.25` | scale (splat scale along axis) |
|
||||
| `axisRadius` | `0.0075` | radius of the axes (splat scale orthogonal to axis) |
|
||||
| `axisShadowScale` | `2.0` | relative size of the "shadow copy" of each splat placed behind it |
|
||||
| `origins` | `[new THREE.Vector3()]` | origins of the axes (default single axis at origin) |
|
||||
|
||||
|
||||
## Gsplat sphere
|
||||
## Splat sphere
|
||||
|
||||
```javascript
|
||||
import { constructSpherePoints } from "@worldlabsai/forge";
|
||||
@@ -139,11 +141,11 @@ scene.add(sphere);
|
||||
|-----------|---------|-------------|
|
||||
| `origin` | `new THREE.Vector3()` | center of the sphere (default: origin) |
|
||||
| `radius` | `1.0` | radius of the sphere |
|
||||
| `maxDepth` | `3` | maximum depth of recursion for subdividing the sphere. Warning: Gsplat count grows exponentially with depth |
|
||||
| `maxDepth` | `3` | maximum depth of recursion for subdividing the sphere. Warning: splat count grows exponentially with depth |
|
||||
| `filter` | `null` | filter function to apply to each point, for example to select points in a certain direction or other function ((THREE.Vector3) => boolean) |
|
||||
| `pointRadius` | `0.02` | radius of each oriented Gsplat |
|
||||
| `pointThickness` | `0.001` | flatness of each oriented Gsplat |
|
||||
| `color` | `new THREE.Color(1, 1, 1)` | color of each Gsplat (THREE.Color) or function to set color for point: ((THREE.Color, THREE.Vector3) => void) |
|
||||
| `pointRadius` | `0.02` | radius of each oriented splat |
|
||||
| `pointThickness` | `0.001` | flatness of each oriented splat |
|
||||
| `color` | `new THREE.Color(1, 1, 1)` | color of each splat (THREE.Color) or function to set color for point: ((THREE.Color, THREE.Vector3) => void) |
|
||||
|
||||
|
||||
## Rasterizing Text
|
||||
@@ -173,14 +175,14 @@ scene.add(splats);
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `font` | `"Arial"` | browser font to render text with |
|
||||
| `fontSize` | `32` | font size in pixels/Gsplats |
|
||||
| `color` | `new THREE.Color(1, 1, 1)` | SplatMesh.recolor tint assuming white Gsplats |
|
||||
| `rgb` | `new THREE.Color(1, 1, 1)` | Individual Gsplat color |
|
||||
| `dotRadius` | `0.8` | Gsplat radius (0.8 covers 1-unit spacing well) |
|
||||
| `fontSize` | `32` | font size in pixels/splats |
|
||||
| `color` | `new THREE.Color(1, 1, 1)` | SplatMesh.recolor tint assuming white splats |
|
||||
| `rgb` | `new THREE.Color(1, 1, 1)` | Individual splat color |
|
||||
| `dotRadius` | `0.8` | splat radius (0.8 covers 1-unit spacing well) |
|
||||
| `textAlign` | `"start"` | text alignment: "left", "center", "right", "start", "end" |
|
||||
| `lineHeight` | `1.0` | line spacing multiplier, lines delimited by "\n" |
|
||||
|
||||
## Turning images into Gsplats
|
||||
## Turning images into splats
|
||||
|
||||
```typescript
|
||||
const image = imageSplats({
|
||||
@@ -211,21 +213,21 @@ scene.add(image);
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `dotRadius` | `0.8` | Radius of each Gsplat, default covers 1-unit spacing well |
|
||||
| `dotRadius` | `0.8` | Radius of each splat, default covers 1-unit spacing well |
|
||||
| `subXY` | `1` | Subsampling factor for the image. Higher values reduce resolution, for example 2 will halve the width and height by averaging |
|
||||
| `forEachSplat` | `undefined` | Optional callback function to modify each Gsplat before it's added. Return null to skip adding the Gsplat, or a number to set the opacity and add the Gsplat with parameter values in the objects center, rgba etc. were passed into the forEachSplat callback. Ending the callback in `return opacity;` will retain the original opacity. |
|
||||
| `forEachSplat` | `undefined` | Optional callback function to modify each splat before it's added. Return null to skip adding the splat, or a number to set the opacity and add the splat with parameter values in the objects center, rgba etc. were passed into the forEachSplat callback. Ending the callback in `return opacity;` will retain the original opacity. |
|
||||
|
||||
### Example
|
||||
|
||||
```javascript
|
||||
// Load RGBA image from image.png, subsample it 2x
|
||||
// horizontally and vertically, and create Gsplats for
|
||||
// horizontally and vertically, and create splats for
|
||||
// the resulting pixels that have at least 10% opacity.
|
||||
const image = imageSplats({
|
||||
url: "./image.png",
|
||||
subXY: 2,
|
||||
forEachSplat: (width, height, index, center, scales, quaternion, opacity, color) => {
|
||||
// Only keep Gsplats with opacity 10% or higher
|
||||
// Only keep splats with opacity 10% or higher
|
||||
return (opacity >= 0.1) ? opacity : null;
|
||||
},
|
||||
});
|
||||
@@ -234,15 +236,17 @@ scene.add(image);
|
||||
|
||||
## Particle Effects
|
||||
|
||||
Some of the building blocks are meant to serve as "code inspiration", showing how a particle effect animation can be achieved using a "stateless" `dyno` computation graph that uses only the Gsplat `index` input to produce pseudo-random numbers that drive various randomized particle effects.
|
||||
Some of the building blocks are meant to serve as "code inspiration", showing how a particle effect animation can be achieved using a "stateless" `dyno` computation graph that uses only the splat `index` input to produce pseudo-random numbers that drive various randomized particle effects.
|
||||
|
||||
Note that Gsplat sorting takes a little bit of time and can "lag behind" the Gsplat updates each frame, so it's important that there is a reasonably stable correspondence between each frame's Gsplat output for the same `index`.
|
||||
Note that splat sorting takes a little bit of time and can "lag behind" the splat updates each frame, so it's important that there is a reasonably stable correspondence between each frame's splat output for the same `index`.
|
||||
|
||||
In `staticBox`, the AABB is sliced up into a 3D grid of cells, and random X/Y/Z points within those cells are sampled each frame, retaining consistency in Gsplat position between successive frames. See `examples/gsplat-sweets-garden/main.js` for an example that creates a `staticBox`.
|
||||
### `generators.staticBox(...options)`
|
||||
|
||||
In `staticBox` you provide a 3D box to render random "static" splats within. The AABB is sliced up into a 3D grid of cells, and random X/Y/Z points within those cells are sampled each frame, retaining consistency in splat position between successive frames.
|
||||
|
||||
### `generators.snowBox(...options)`
|
||||
|
||||
Similarly, `snowBox` produces Gsplat trajectories that move in a deterministic fashion over time, with high similarity between adjacent frames. See `examples/atmospheric/main.js` for an example that creates a `snowBox`.
|
||||
Similarly, `snowBox` produces splat trajectories that move in a deterministic fashion over time, with high similarity between adjacent frames. See "VFX - Particle Simulation" for an example that creates a `snowBox`.
|
||||
|
||||
A snowBox instance has a collection of properties that can be tuned to achieve different particle effects. DEFAULT_SNOW and DEFAULT_RAIN are example parameter sets that look a lot like snow and rain, and can be used as a starting point for further tweaking: `const mySnow = { ...DEFAULT_SNOW, density: 500 };`
|
||||
|
||||
@@ -271,16 +275,16 @@ scene.add(snowControls.snow);
|
||||
|-----------|---------------|-------------|
|
||||
| `box` | `new THREE.Box3(new THREE.Vector3(-1, -1, -1), new THREE.Vector3(1, 1, 1))` | min and max box extents of the snowBox |
|
||||
| `minY` | `Number.NEGATIVE_INFINITY` | minimum y-coordinate to clamp particle position, which can be used to fake hitting a ground plane and lingering there for a bit |
|
||||
| `numSplats` | calculated from box and density | number of Gsplats to generate |
|
||||
| `density` | `100` | density of Gsplats per unit volume |
|
||||
| `anisoScale` | `new THREE.Vector3(1, 1, 1)` | The xyz anisotropic scale of the Gsplat, which can be used for example to elongate rain particles |
|
||||
| `minScale` | `0.001` | Minimum Gsplat particle scale |
|
||||
| `maxScale` | `0.005` | Maximum Gsplat particle scale |
|
||||
| `numSplats` | calculated from box and density | number of splats to generate |
|
||||
| `density` | `100` | density of splats per unit volume |
|
||||
| `anisoScale` | `new THREE.Vector3(1, 1, 1)` | The xyz anisotropic scale of the splat, which can be used for example to elongate rain particles |
|
||||
| `minScale` | `0.001` | Minimum splat particle scale |
|
||||
| `maxScale` | `0.005` | Maximum splat particle scale |
|
||||
| `fallDirection` | `new THREE.Vector3(0, -1, 0)` | The average direction of fall |
|
||||
| `fallVelocity` | `0.02` | The average speed of the fall (multiplied with fallDirection) |
|
||||
| `wanderScale` | `0.01` | The world scale of wandering overlay motion |
|
||||
| `wanderVariance` | `2` | Controls how uniformly the particles wander in sync, more variance means more randomness in the motion |
|
||||
| `color1` | `new THREE.Color(1, 1, 1)` | Color 1 of the two colors interpolated between |
|
||||
| `color2` | `new THREE.Color(0.5, 0.5, 1)` | Color 2 of the two colors interpolated between |
|
||||
| `opacity` | `1` | The base opacity of the Gsplats |
|
||||
| `opacity` | `1` | The base opacity of the splats |
|
||||
| `onFrame` | `undefined` | Optional callback function to call each frame |
|
||||
|
||||
+35
-35
@@ -1,37 +1,12 @@
|
||||
# Gsplat Editing
|
||||
# Splat Editing
|
||||
|
||||
Forge provides the ability to apply "edits" to Gsplats as part of the standard `SplatMesh` pipeline. These edits take the form of a sequence of operations, applied one at a time to the set of Gsplats in its `packedSplats`. Each operation evaluates a 7-dimensional field (RGBA and XYZ displacement) at each point in space that derives from N=1 or more Signed Distance Field shapes (such as spheres, boxes, planes, etc.), blended together and across inside-outisde boundaries.
|
||||
Forge provides the ability to apply "edits" to splats as part of the standard `SplatMesh` pipeline. These edits take the form of a sequence of operations, applied one at a time to the set of splats in its `packedSplats`. Each operation evaluates a 7-dimensional field (RGBA and XYZ displacement) at each splat's center point in space that derives from 1 or more Signed Distance Field shapes (such as spheres, boxes, planes, etc.), blended together and across inside-outside boundaries.
|
||||
|
||||
The result is a an RGBA,XYZ value for each point in space, which combined with SplatEditRgbaBlendMode.MULTIPLY/SET_RGB/ADD_RGBA can be used to create special effects, for example simulating simple lighting or applying deformations in space, whose parameters can be updated each frame to create animated effects.
|
||||
The result is a an RGBA,XYZ value for each splat, which combined with SplatEditRgbaBlendMode.MULTIPLY/SET_RGB/ADD_RGBA can be used to create special effects. For example, simulating simple lighting can be done with MULTIPLY or ADD_RGBA to light up or adjust the color in regions of space, using spheres for point light sources or infinite cone for a spotlight. Using MULTIPLY with opacity=0 can be used to delete splats from a region of space. The splats can also be displaced in space using the XYZ values, and adjusted each frame to create smooth deformations in space.
|
||||
|
||||
## RGBA blend modes
|
||||
## Creating a splat edit operation
|
||||
|
||||
When creating a `SplatEdit` you can specify a `rgbaBlendMode?: SplatEditRgbaBlendMode` value from the enum to choose between 3 blend modes:
|
||||
|
||||
| Blend Mode | Description |
|
||||
|-------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
|
||||
| `SplatEditRgbaBlendMode.MULTIPLY` | The RGBA of the splat is multiplied component-wise by the SDF’s RGBA value at that point in space. |
|
||||
| `SplatEditRgbaBlendMode.SET_RGB` | Ignore the Alpha value in the SDF, but set the splat’s RGB to equal the SDF’s RGB value at that point. |
|
||||
| `SplatEditRgbaBlendMode.ADD_RGBA` | Add the SDF’s RGBA value at that point to the RGBA value of the Gsplat. This can produce hyper-saturated results, but is useful to easily “light up” areas. |
|
||||
|
||||
## SDF Shapes
|
||||
|
||||
The following SDF shapes are available in the `SplatEditSdfType` enum:
|
||||
|
||||
| SDF Type | Description | Parameters |
|
||||
|----------|-------------|------------|
|
||||
| `ALL` | Affects all points in space | None |
|
||||
| `PLANE` | Infinite plane | position, rotation |
|
||||
| `SPHERE` | Sphere | position, radius |
|
||||
| `BOX` | Box (with optional corner rounding) | position, rotation, sizes, radius |
|
||||
| `ELLIPSOID` | Ellipsoid | position, rotation, sizes |
|
||||
| `CYLINDER` | Cylinder | position, rotation, size_y |
|
||||
| `CAPSULE` | Capsule | position, rotation, size_y |
|
||||
| `INFINITE_CONE` | Infinite cone | position, rotation, radius=angle |
|
||||
|
||||
## Creating a Gsplat edit operation
|
||||
|
||||
A `SplatEdit` operation can be assigned to a particular `SplatMesh` through its `.edits[]` property or by adding the `SplatEdit` as a child of the `SplatMesh` in the scene hierarchy. If the `SplatEdit` has no `SplatMesh` ancestor, its edits will apply globally to all `SplatMesh`es whose `editable` property is set to default true.
|
||||
A `SplatEdit` operation can be assigned to a particular `SplatMesh` through its `.edits[]` property or by adding the `SplatEdit` as a child of the `SplatMesh` in the scene hierarchy. If the `SplatEdit` has no `SplatMesh` ancestor, its edits will apply globally to all `SplatMesh`es whose `editable` property is set to the default `true`.
|
||||
|
||||
```typescript
|
||||
const edit = new SplatEdit({
|
||||
@@ -56,6 +31,16 @@ scene.add(edit);
|
||||
| `invert` | `false` | Invert the SDF evaluation (inside/outside swap). |
|
||||
| `sdfs` | `null` | Explicit array of `SplatEditSdf` objects to include. If `null`, any child `SplatEditSdf` instances are used. |
|
||||
|
||||
### RGBA blend modes
|
||||
|
||||
When creating a `SplatEdit` you can specify a `rgbaBlendMode?: SplatEditRgbaBlendMode` value from the enum to choose between 3 blend modes:
|
||||
|
||||
| Blend Mode | Description |
|
||||
|-------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
|
||||
| `MULTIPLY` | The RGBA of the splat is multiplied component-wise by the SDF’s RGBA value at that point in space. (default) |
|
||||
| `SET_RGB` | Ignore the Alpha value in the SDF, but set the splat’s RGB to equal the SDF’s RGB value at that point. |
|
||||
| `ADD_RGBA` | Add the SDF’s RGBA value at that point to the RGBA value of the splat. This can produce hyper-saturated results, but is useful to easily “light up” areas. Note that having non-zero opacity is often not what you want because low-opacity splats will be made more opaque.|
|
||||
|
||||
## Adding an SDF RGBA-XYZ shape to the edit operation
|
||||
|
||||
```typescript
|
||||
@@ -74,15 +59,30 @@ edit.add(shape1);
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `type` | `SplatEditSdfType.SPHERE` | The SDF shape type: `ALL`, `PLANE`, `SPHERE`, `BOX`, `ELLIPSOID`, `CYLINDER`, `CAPSULE`, or `INFINITE_CONE`. |
|
||||
| `type` | `SPHERE` | The SDF shape type: `ALL`, `PLANE`, `SPHERE`, `BOX`, `ELLIPSOID`, `CYLINDER`, `CAPSULE`, or `INFINITE_CONE`. |
|
||||
| `invert` | `false` | Invert the SDF evaluation, swapping inside and outside regions. |
|
||||
| `opacity` | `1.0` | Opacity / "alpha" value used differently by blending modes |
|
||||
| `color` | `new THREE.Color(1.0, 1.0, 1.0)` | RGB color applied within the shape. |
|
||||
| `displace` | `new THREE.Vector3(0.0, 0.0, 0.0)` | XYZ displacement applied to splat positions inside the shape. |
|
||||
| `color` | `Color(1, 1, 1)` | RGB color applied within the shape. |
|
||||
| `displace` | `Vector3(0, 0, 0)` | XYZ displacement applied to splat positions inside the shape. |
|
||||
| `radius` | `0.0` | Shape-specific size parameter: sphere radius, box corner rounding, cylinder/capsule radius, or for the infinite cone the angle factor (opening half-angle = π/4 × `radius`). |
|
||||
|
||||
### SDF Shapes
|
||||
|
||||
The following SDF shapes are available in the `SplatEditSdfType` enum:
|
||||
|
||||
| SDF Type | Description | SDF Parameters |
|
||||
|----------|-------------|------------|
|
||||
| `ALL` | Affects all points in space | None |
|
||||
| `PLANE` | Infinite plane | position, rotation |
|
||||
| `SPHERE` | Sphere | position, radius |
|
||||
| `BOX` | Box (with optional corner rounding) | position, rotation, sizes, radius |
|
||||
| `ELLIPSOID` | Ellipsoid | position, rotation, sizes |
|
||||
| `CYLINDER` | Cylinder | position, rotation, size_y |
|
||||
| `CAPSULE` | Capsule | position, rotation, size_y |
|
||||
| `INFINITE_CONE` | Infinite cone | position, rotation, radius=angle |
|
||||
|
||||
## Adding multiple SDF RGBA-XYZ shapes to the edit operation
|
||||
|
||||
RGBA-XYZ values are computed by blending together values from all SDF shapes using the exponential "softmax" function, which is commutative (so blending order within a `SplatEdit` operation doesn't matter). The parameter `SplatEdit.sdfSmooth` controls the blending scale *between* SDF shapes, while `SplatEdit.softEdge` controls the scale of soft inside-outside shape edit blending. Their default values start at `0.0` and should be increased to soften the effect.
|
||||
RGBA-XYZ values are computed by blending together values from all SDF shapes using the exponential "softmax" function, which is commutative (so blending order within a `SplatEdit` operation doesn't matter). The parameter `SplatEdit.sdfSmooth` controls the blending scale *between* SDF shapes, while `SplatEdit.softEdge` controls the scale of soft inside-outside shape boundaries. Their default values start at `0.0` and should be increased to soften the edges.
|
||||
|
||||
Note that XYZ displacement values are blended in the same way as RGBA, with a resulting displacement field that can be quite complex but "softly" blending between shapes. These RGBA-XYZ edits, along with time-based and overlapping fields can create many interesting animations and special effects, such as rippling leaves in the wind, an angry fire, or a looping water effects. Simply update the `SplatEdit` and `SplatEditSdf` objects and the operations will be applied immediately to the Gsplats in the scene.
|
||||
Note that XYZ displacement values are blended in the same way as RGBA, with a resulting displacement field that can be quite complex but "softly" blends between shapes. These RGBA-XYZ edits, along with time-based and overlapping fields can create many interesting animations and special effects, such as rippling leaves in the wind, an angry fire, or a looping water effect. Simply update the `SplatEdit` and `SplatEditSdf` objects and the operations will be applied immediately to the splats in the scene.
|
||||
|
||||
+20
-20
@@ -1,10 +1,10 @@
|
||||
# SplatMesh
|
||||
|
||||
A `SplatMesh` is a high-level interface for displaying and manipulating a "Splat mesh", a collection of Gaussian splats that serves as an "object" of sorts. It is analagous to a traditional polygon `THREE.Mesh`, which consists of geometry (points and triangles) and materials (color and lighting). Similarly, a `SplatMesh` contains geometry (Gsplat centers, quaternion, and xyz scales) and materials (RGB color, opacity, spherical harmonics for directional lighting), and can be added anywhere in the scene hierarchy.
|
||||
A `SplatMesh` is a high-level interface for displaying and manipulating a "Splat mesh", a collection of Gaussian splats that serves as an "object" of sorts. It is analagous to a traditional triangle-based `THREE.Mesh`, which consists of geometry (points and triangles) and materials (color and lighting). Similarly, a `SplatMesh` contains geometry (splat centers, quaternion, and xyz scales) and materials (RGB color, opacity, spherical harmonics up to degree 3 for directional lighting), and can be added anywhere in the scene hierarchy.
|
||||
|
||||
The usual Three.js properties `position`, `quaternion`, `rotation` behave as you would expect, however `scale` only allows uniform scaling and averages the x/y/z scales. Additional properties `recolor` and `opacity` are multiplied in with the final Gsplat color and opacity.
|
||||
The usual Three.js properties `position`, `quaternion`, `rotation` behave as you would expect, however `scale` only allows uniform scaling and averages the x/y/z scales. Additional properties `recolor` and `opacity` are multiplied in with the final splat color and opacity.
|
||||
|
||||
`SplatMesh` is a subclass of the more fundamental `SplatGenerator`, which itself is a subclass of `THREE.Object3D`. Any methods and properties on `Object3D` are also available in `SplatMesh`. `SplatGenerator` gives you more control over Gsplat generation and modification, but `SplatMesh` has an easier and higher-level API.
|
||||
`SplatMesh` is a subclass of the more fundamental `SplatGenerator`, which itself is a subclass of `THREE.Object3D`. Any methods and properties on `Object3D` are also available in `SplatMesh`. `SplatGenerator` gives you more control over splat generation and modification, but `SplatMesh` has an simpler higher-level API.
|
||||
|
||||
## Creating a `SplatMesh`
|
||||
|
||||
@@ -32,43 +32,43 @@ const splats = new SplatMesh({
|
||||
time,
|
||||
deltaTime,
|
||||
}: { mesh: SplatMesh; time: number; deltaTime: number }) => void;
|
||||
// Object-space and world-space Gsplat modifiers to apply
|
||||
// Object-space and world-space splat modifiers to apply
|
||||
objectModifier?: GsplatModifier;
|
||||
worldModifier?: GsplatModifier;
|
||||
});
|
||||
// Add to scene to show Gsplats (requires ForgeRenderer as well)
|
||||
// Add to scene to show splats (requires ForgeRenderer as well)
|
||||
scene.add(splats);
|
||||
```
|
||||
|
||||
### Optional parameters
|
||||
|
||||
You can create a `new SplatMesh()` with no options, which will create a new default instance with `.numSplats=0`. Alternatively, you can provide an input `url` to fetch and decode, `fileBytes`, `packedSplats` (an existing collection of tightly "packed" Gsplats). Forge supports most Gsplat file types, including .ply, .splat, .ksplat, .spz. To load filetypes .splat and .ksplat (which can't be reliably auto-detected), use the optional `fileType` argument.
|
||||
You can create a `new SplatMesh()` with no options, which will create a new default instance with `.numSplats=0`. Alternatively, you can provide an input `url` to fetch and decode, `fileBytes`, or `packedSplats` (an existing collection of "packed" splats). Forge supports most splat file types, including .ply (including SuperSplat compressed), .splat, .ksplat, .spz. To load filetypes .splat and .ksplat (which can't be reliably auto-detected), use the optional `fileType` argument.
|
||||
|
||||
Constructor argument callbacks can be used like `constructSplats` to create a collection of Gsplat programmatically at initialization, `onLoad` when loading and initialization completes, `onFrame` to update state on every frame. Gsplat effects can be injected into the standard Gsplat processinng pipeline that operate in object-space and world-space via `objectModifier` and `worldModifier` respectively.
|
||||
Constructor argument callbacks can be used like `constructSplats` to create a collection of splats procedurally at initialization, `onLoad` when loading and initialization completes, `onFrame` to update state every frame. Splat effects can be injected into the standard splat processinng pipeline that operate in object-space and world-space via `objectModifier` and `worldModifier` respectively.
|
||||
|
||||
| **Parameter** | Description |
|
||||
| ----------------- | ----------- |
|
||||
| **url** | URL to fetch a Gaussian splat file from(supports .ply, .splat, .ksplat, .spz formats). (default: `undefined`)
|
||||
| **url** | URL to fetch a Gaussian splat file from (supports .ply, .splat, .ksplat, .spz formats). (default: `undefined`)
|
||||
| **fileBytes** | Raw bytes of a Gaussian splat file to decode directly instead of fetching from URL. (default: `undefined`)
|
||||
| **fileType** | Override the file type detection for formats that can't be reliably auto-detected (.splat, .ksplat). (default: `undefined` auto-detects other formats from file contents)
|
||||
| **packedSplats** | Use an existing PackedSplats object as the source instead of loading from a file. Can be used to share a collection of Gsplats among multiple `SplatMesh`es (default: `undefined` creates a new empty `PackedSplats` or decoded from a data source above)
|
||||
| **packedSplats** | Use an existing PackedSplats object as the source instead of loading from a file. Can be used to share a collection of splats among multiple `SplatMesh`es (default: `undefined` creates a new empty `PackedSplats` or decoded from a data source above)
|
||||
| **maxSplats** | Reserve space for at least this many splats when constructing the mesh initially. (default: determined by file)
|
||||
| **constructSplats** | Callback function to programmatically create splats at initialization in provided `PackedSplats`. (default: `undefined`)
|
||||
| **constructSplats** | Callback function to programmatically create splats at initialization in a newly created `PackedSplats`. (default: `undefined`)
|
||||
| **onLoad** | Callback function that is called when mesh initialization is complete. (default: `undefined`)
|
||||
| **editable** | Controls whether SplatEdits have any effect on this mesh. (default: `true`)
|
||||
| **onFrame** | Callback function that is called every frame to update the mesh. Call `mesh.updateVersion()` if splats need to be regenerated due to some change. Calling `updateVersion()` is not necessary for object transformations, recoloring, or opacity adjustments as these are auto-detected. (default: `undefined`)
|
||||
| **objectModifier** | Gsplat modifier to apply in object-space before any transformations. A `GsplatModifier` is a `dyno` shader-graph block that transforms an input `gsplat: DynoVal<Gsplat>` to an output `gsplat: DynoVal<Gsplat>` with `gsplat.center` coordinate in object-space. (default: `undefined`)
|
||||
| **worldModifier** | Gsplat modifier to apply in world-space after transformations. (default: `undefined`)
|
||||
| **onFrame** | Callback function that is called every frame to update the mesh. Call `mesh.updateVersion()` if splats need to be regenerated due to some change. Calling `updateVersion()` is not necessary for object transformations, recoloring, or opacity adjustments as these changes are auto-detected. (default: `undefined`)
|
||||
| **objectModifier** | Splat modifier to apply in object-space before any transformations. A `GsplatModifier` is a `dyno` shader-graph block that transforms an input `gsplat: DynoVal<Gsplat>` to an output `gsplat: DynoVal<Gsplat>` with `gsplat.center` and other parameters in object-space. (default: `undefined`)
|
||||
| **worldModifier** | Splat modifier to apply in world-space after transformations. (default: `undefined`)
|
||||
|
||||
## Instance properties
|
||||
|
||||
The constructor argument options `packedSplats`, `editable`, `onFrame`, `objectModifier`, and `worldModifier` can be modified directly on the `SplatMesh`.
|
||||
|
||||
If you modify `packedSplats` you should set `splatMesh.packedSplats.needsUpdate = true` to signal to Three.js that it should re-upload the data to the underlying texture. Use this sparingly with objects with smaller Gsplat counts as it requires a CPU-GPU data transfer for each frame. Thousands to tens of thousands of Gsplats ir fine. (See `hands.ts` for an example of rendering "Gsplat hands" in WebXR using this technique.)
|
||||
If you modify `packedSplats` you should set `splatMesh.packedSplats.needsUpdate = true` to signal to Three.js that it should re-upload the data to the underlying texture. Use this sparingly with objects with lower splat counts as it requires a CPU-GPU data transfer for each frame. Thousands to tens of thousands of splats is reasonable. (See `hands.ts` for an example of rendering "splat hands" in WebXR using this technique.)
|
||||
|
||||
If you modify `objectModifier` or `worldModifier` you should call `splatMesh.updateGenerator()` to update the pipeline and have it compile to run efficiently on the GPU.
|
||||
|
||||
Additional properties you can modify on a `SplatMesh` instance:
|
||||
Additional properties on a `SplatMesh` instance:
|
||||
|
||||
| **Property** | Description |
|
||||
| ----------------- | ----------- |
|
||||
@@ -76,7 +76,7 @@ Additional properties you can modify on a `SplatMesh` instance:
|
||||
| **isInitialized** | A `boolean` indicating whether initialization is complete
|
||||
| **recolor** | A `THREE.Color` that can be used to tint all splats in the mesh. (default: `new THREE.Color(1, 1, 1)`)
|
||||
| **opacity** | Global opacity multiplier for all splats in the mesh. (default: `1`)
|
||||
| **context** | A `SplatMeshContext` consisting of useful scene and object `dyno` uniforms that can be used to in the Gsplat processing pipeline, for example via `objectModifier` and `worldModifier`. (created on construction)
|
||||
| **context** | A `SplatMeshContext` consisting of useful scene and object `dyno` uniforms that can be used to in the splat processing pipeline, for example via `objectModifier` and `worldModifier`. (created on construction)
|
||||
| **enableViewToObject** | Set to `true` to have the `viewToObject` property in `context` be updated each frame. If the mesh has `extra.sh1` (first order spherical harmonics directional lighting) this property will always be updated. (default: `false` )
|
||||
| **enableViewToWorld** | Set to `true` to have `context.viewToWorld` updated each frame. (default: `false`)
|
||||
| **enableWorldToView** | Set to `true` to have `context.worldToView` updated each frame. (default: `false`)
|
||||
@@ -91,15 +91,15 @@ Call this when you are finished with the `SplatMesh` and want to free any buffer
|
||||
|
||||
## `pushSplat(center, scales, quaternion, opacity, color)`
|
||||
|
||||
Creates a new Gsplat with the provided parameters (all values in "float" space, i.e. 0-1 for opacity and color) and adds it to the end of the `packedSplats`, increasing `numSplats` by 1. If necessary, reallocates the buffer with an exponential doubling strategy to fit the new data, so it's fairly efficient to just `pushSplat(...)` each Gsplat you want to create in a loop.
|
||||
Creates a new splat with the provided parameters (all values in "float" space, i.e. 0-1 for opacity and color) and adds it to the end of the `packedSplats`, increasing `numSplats` by 1. If necessary, reallocates the buffer with an exponential doubling strategy to fit the new data, so it's fairly efficient to `pushSplat(...)` each splat you want to create in a loop.
|
||||
|
||||
## `forEachSplat(callback: (index, center, scales, quaternion, opacity, color) => void)`
|
||||
|
||||
This method iterates over all Gsplats in this instance's `packedSplats`, invoking the provided callback with `index: number` in `0..=(this.numSplats-1)` and `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number` (0..1), and `color: THREE.Color` (rgb values in 0..1). Note that the objects passed in as `center` etc. are the same for every callback invocation: these objects are reused for efficiency. *Changing these values has no effect* as they are decoded/unpacked copies of the underlying data. To update the `packedSplats`, call `.packedSplats.setSplat(index, center, scales, quaternion, opacity, color)`.
|
||||
This method iterates over all splats in this instance's `packedSplats`, invoking the provided callback with `index: number` in `0..=(this.numSplats-1)`, `center: THREE.Vector3`, `scales: THREE.Vector3`, `quaternion: THREE.Quaternion`, `opacity: number` (0..1), and `color: THREE.Color` (rgb values in 0..1). Note that the objects passed in as `center` etc. are the same for every callback invocation: they are reused for efficiency. *Changing these values has no effect* as they are decoded/unpacked copies of the underlying data. To update the `packedSplats`, call `.packedSplats.setSplat(index, center, scales, quaternion, opacity, color)`.
|
||||
|
||||
## `updateGenerator()`
|
||||
|
||||
Call this whenever something changes in the Gsplat processing pipeline, for example changing `maxSh` or updating `objectModifier` or `worldModifier`. Compiled generators are cached for efficiency and re-use when the same pipeline structure emerges after successive changes.
|
||||
Call this whenever something changes in the splat processing pipeline, for example changing `maxSh` or updating `objectModifier` or `worldModifier`. Compiled generators are cached for efficiency and re-used when the same graph structure emerges after successive changes.
|
||||
|
||||
## `update(...)`
|
||||
|
||||
@@ -107,7 +107,7 @@ This is called automatically by `ForgeRenderer` and you should not have to call
|
||||
|
||||
## `raycast(raycaster, intersects: { distance, point, object}[])`
|
||||
|
||||
This method conforms to the standard `THREE.Raycaster` API, performing object-ray intersections using this method to populate the provided `intersects[]` array with each intersection point.
|
||||
This method conforms to the standard `THREE.Raycaster` API, performing object-ray intersections using this method to populate the provided `intersects[]` array with each intersection point's `distance: number`, `point: THREE.Vector3`, and `object: SplatMesh`. Note that this method is synchronous and uses a WebAssembly-based ray-splat intersection algorithm that iterates over all points. Raycasting against millions of splats have a noticeable delay, and should not be called every frame.
|
||||
|
||||
Usage example:
|
||||
```javascript
|
||||
|
||||
+12
-11
@@ -1,29 +1,30 @@
|
||||
# System Design
|
||||
|
||||
One of the biggest challenges in real-time Gaussian splat (Gsplat) rendering is sorting the splats so that they can be drawn and blended in back-to-front order, known as Painter's algorithm.
|
||||
One of the biggest challenges in real-time Gaussian splat rendering is sorting the splats so that they can be drawn and blended in back-to-front order, known as the Painter's algorithm.
|
||||
|
||||
## Rendering data flow cycle
|
||||
|
||||
`ForgeRenderer` is a key component in Forge that manages this process. It traverses the visible Three.js scene graph and compiles a complete list of all Gsplats across the scene, generated by instances of `SplatMesh` in the scene hierarchy.
|
||||
`ForgeRenderer` is a key component in Forge that manages this process. It traverses the visible Three.js scene graph and compiles a complete list of all splats across the scene, generated by instances of `SplatMesh` in the scene hierarchy.
|
||||
|
||||
Each `ForgeRenderer` has a default `ForgeViewpoint`, which reads back a list of all Gsplat distances from the viewpoint, the computes the Gsplat draw order using an efficient bucket sort algorithm, run in a background worker thread via `SplatWorker`. You can spawn additional `ForgeViewpoint`s to create multiple simultaneous render viewpoints.
|
||||
Each `ForgeRenderer` has a default `ForgeViewpoint` that reads back a list of all splat viewpoint distances from the GPU, then determines the splat draw order using an efficient bucket sort algorithm, run in a background worker thread via `SplatWorker`. You can spawn additional `ForgeViewpoint`s to create multiple simultaneous render viewpoints.
|
||||
|
||||
Finally, on the next Three.js render() call, `ForgeRenderer` invokes a single instanced geometry draw call to draw all the scene's Gsplats in the correct back-to-front order, merging with other opaque Three.js geometry using the Z buffer.
|
||||
Finally, on the next Three.js render() call, `ForgeRenderer` invokes a single instanced geometry draw call to draw all the scene's splats in the correct back-to-front order, merging with other opaque Three.js geometry using the Z buffer. The sort order lags the render by at least one frame, but possibly more on older devices, but is not usually perceptible.
|
||||
|
||||
```typescript
|
||||
// Optionally add a ForgeRenderer to the scene to manage SplatMesh rendering.
|
||||
// If none is created, Forge will create one for you automatically.
|
||||
const forge = new ForgeRenderer({ renderer: webGlRenderer });
|
||||
// Add it to the scene and it will manage rendering of SplatMeshes
|
||||
scene.add(forge);
|
||||
```
|
||||
|
||||
This design allows Gsplats from distinct scenes / object splat files to coexist in space and sort correctly w.r.t. each other's Gsplats. Gsplats from independent `SplatMesh`es are aggregated using a `SplatAccumulator`, which produces a `PackedSplats`, a collection Gsplats stored in a cache-efficient 16-byte/Gsplat format.
|
||||
This design allows splats from distinct objects/scenes to coexist in space and sort correctly w.r.t. each other's splats. Splats from independent `SplatMesh`es are aggregated using a `SplatAccumulator`, which produces a `PackedSplats`, a collection splats stored in a cache-efficient 16-byte/splat format.
|
||||
|
||||
## "Programmable Gsplats"
|
||||
## "Programmable Splats"
|
||||
|
||||
Forge also uses this opportunity to run a user-programmable data pipeline on each Gsplat on the GPU. The standard pipeline provide high-level controls, such as rigid transforms, adjusting RGB / opacity, and spherical harmonics, but also special effects (via `SplatEdit`) and a skeletal animation system (`SplatSkinning`). The standard pipeline also allows injecting arbitrary code to modify each Gsplat via `dyno` shader graph system.
|
||||
Forge also uses this opportunity to run a user-programmable data pipeline on each splat on the GPU. The standard pipeline provides high-level functionality, such as applying rigid transforms, adjusting RGB / opacity, and spherical harmonics, but also color editing and perturbations (via `SplatEdit`) and a dual-quaternion skeletal animation system (`SplatSkinning`). The standard pipeline also allows injecting arbitrary code to modify each splat via `dyno` shader graph system.
|
||||
|
||||
`SplatMesh` derives from a more general base class `SplatGenerator`, which itself derives from `THREE.Object3D`. As such, it can be placed anywhere in the scene hierarchy and obeys expected local and global coordinate transforms. A `SplatGenerator` is the most general form of a "Gsplat object", whose Gsplats are produced programmatically via a `dyno` shader graph that maps `{ index: "int" }` to `{ gsplat: "Gsplat" }`. A `SplatMesh` is a higher-level object that implements such a mapping, reading source Gsplats from a template (loaded via a `url` constructor parameter or otherwise) at the given `index`, then applying functions such as transforming to world space.
|
||||
`SplatMesh` derives from a more general base class `SplatGenerator`, which itself derives from `THREE.Object3D`. As such, it can be placed anywhere in the scene hierarchy and obeys expected local and global coordinate transforms. A `SplatGenerator` is the most general form of a "splat object", whose splats are produced programmatically via a `dyno` shader graph function that maps `{ index: "int" }` to `{ gsplat: "Gsplat" }`. A `SplatMesh` is a higher-level object that implements such a mapping, reading source splats from a template (loaded via a `url` constructor parameter or otherwise) at the given `index`, then applying functions such as transforming to world space.
|
||||
|
||||
In contrast, implementing a `SplatGenerator` gives you full control to write any function that programmatically computes a Gsplat's attributes (center, scales, quaternion, rgba). These could be stateless (relying only on `index`, random-number generators, etc), or could rely on a complex combination of textures and global parameters for real-time procedural generation, and can vary with time to produce real-time animations.
|
||||
In contrast, implementing a `SplatGenerator` gives you full control to write any function that programmatically computes a splat's attributes (center, scales, quaternion, rgba). These could be stateless (relying only on `index`, random-number generators, etc), or could rely on a complex combination of splat files, textures, and other global parameters for real-time procedural generation, and can vary with time to produce real-time animations.
|
||||
|
||||
The `dyno` shader graph system allows you to create these programmatic pipelines with Javascript code, which is synthesized into GLSL code and compiled and run on the GPU. This `dyno` system powers other components of Forge as well, such as `Readback` (which can perform any computation and read back the resulting value), used to compute the sort distance metric for pairs of Gsplats and read them back for CPU sorting.
|
||||
The `dyno` shader graph system allows you to create these programmatic pipelines with Javascript code, which is synthesized into GLSL code and compiled and run on the GPU. This `dyno` system powers other components of Forge as well, such as `Readback` (which can perform any computation and read back the resulting value), used to compute the sort distance metric for pairs of splats and read them back for CPU sorting.
|
||||
+7
-1
@@ -7,7 +7,13 @@ hide:
|
||||
|
||||
<div class="hero">
|
||||
<h1><img src="/assets/images/logo-hero.png"/></h1>
|
||||
<h2>An advanced 3D Gaussian Splatting renderer for THREE.js</h2>
|
||||
<h2>3D Gaussian Splatting for Developers</h2>
|
||||
<div class="feature-grid">
|
||||
<li class="feature-item">Integrates with Three.js scenes</li>
|
||||
<li class="feature-item">Fast rendering on all devices via WebGL2</li>
|
||||
<li class="feature-item">Combine multiple splat objects</li>
|
||||
<li class="feature-item">Programmable GPU-driven dynamic splat effects</li>
|
||||
</div>
|
||||
<a href="/docs/" class="md-button md-button--primary">Get started →</a>
|
||||
<iframe class="hero-image" src="/examples/hello-world/carousel.html"></iframe>
|
||||
</div>
|
||||
@@ -124,3 +124,15 @@
|
||||
.md-custom-header-right .md-nav__link:hover {
|
||||
color: white;
|
||||
}
|
||||
|
||||
.feature-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 0.5rem;
|
||||
max-width: 800px;
|
||||
margin: 1.5rem auto;
|
||||
}
|
||||
|
||||
.feature-item {
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user