This PR adds the Scaling Gizmo to the StoryTeller Engine.
Features
- Objects can now be scaled by Axis or Plane
Limitations
- Currently there is no implementation for combining the scale gizmo
with any of the other gizmo. This should be amended with further PRs.
Let me know if there is anything that I missed!
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR cherry-picks some improvements from #63 that are general-purpose
and not specifically related to IK.
* Some logic has been added to `AppElement` to track the currently
selected Scene Element
* `disabled` property has been added to `ButtonElement`, with custom
styling and accessibility features
* Tagging entities with their `SceneElement` type has been made more
robust and reliable (previously, it was possible for all entities to
appear in the hierarchy as "Generic" elements depending on the order of
system execution)
Additionally:
* Forces the Netlify CI pipeline to always used the latest stable Rust
version
This PR exposes skeletal joint entities to the frontend hierarchy,
allowing the user to pose rigged models by selecting the joints and
transforming them.
Additionally, entity-type information (e.g., Mesh, Skeleton, Bone,
Light, etc.) has been added to the data structure passed to the
frontend, which is rendered with icons in the tree view.
This PR updates the UI/UX of the Image Generator Demo.
* Hierarchy moved to the left of the editor view instead of overlaying
it
* Hierarchy can now be resized
* Bevy canvas resizes trigger a screen-capture update
* Transform Toolbar now centered over the editor view instead of the
entire viewport
* Added Prompt and Seed text fields, with a refresh button for the Seed
This PR Adds the ability to set the color of the skybox using the URL
parameters.
- Now you can pass a color's hexadecimal number (without the #) in the
skybox parameter, and that will set the skybox to that color.
This PR heavily refactors the entry-point functions in
`studio/src/lib.rs` to simplify and consolidate. Additionally, the
frontend application should now be configurable as expected with any
valid combination of the following URL parameters:
* `objectId` : A pre-bundled library object to load (e.g., "couch.gltf")
* `bvh` : Any resolvable path to a BVH animation with a MocapNET
skeleton (e.g., "mocap/shuffle.bvh")
* `mixamo` : Any resolvable path to a glTF animation with the Mixamo
skeleton (e.g., "mocap/hip-hop-dancing.gltf")
* `sceneImport` : Any resolvable path to an arbitrary glTF scene
(untested, but should theoretically work)
* `scene` : Any resolvable path to a previously saved Storyteller Studio
scene in `*.scn.ron` format
* `skybox` : The ID of a pre-bundled skybox asset. Valid options are:
- "gum_trees_4k"
- "kloofenfal_28d_misty_4k"
- "meadow_4k"
- "promenade_de_vidy_4k"
- "scythian_tombs_4k"
- "test_scene" (this is the default if not specified)
* `mode` : One of "studio" or "viewer" (the default if not specified)
"Validity" rules are as follows:
* `objectId`, `bvh`, `mixamo`, `sceneImport` and `scene` are mutually
exclusive
* `skybox` is not supported for `scene` (the skybox should be saved and
loaded as part of the scene description, though this is not currently
working)
* Both `mode` options are valid for all other parameters
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds Storyteller.ai API integration for saving and loading
Storyteller Studio scenes to/from the backend.
### Known issues
* The skybox is not currently stored in the serialized scene format
This PR adds a WIP timeline UI for animating elements in the scene.
### Implemented Features
* The timeline can be panned with MMB and zoomed with Ctrl + Mouse Wheel
* The timeline can be "scrubbed" by dragging the red "play-head" with
LMB
* The timeline sidebar (displaying the list of tracks) can be resized by
dragging on the edge of the sidebar
* Keyframes can be added by pressing the "Add Keyframe" CTA in the UI
* Keyframes can be selected for editing by clicking the diamond-shaped
markers in the timeline
* Selected entities (including but not limited to keyframes) can be
deleted by pressing the "Delete" key
* Keyframes can be moved temporally by dragging the diamond-shaped
markers in the timeline
### Known issues
* Zooming the timeline in too far results in a soft-lock
* When zooming the timeline, the "transform origin" is currently
centered on the `00:00` timestamp. For ease of use, it should center on
the mouse cursor position instead.
* The timeline track-list is currently hard-coded to list the Camera and
Camera Target. Clicking on these entries currently serves no useful
purpose and may result in a crash.
* Keyframe markers in the timeline are not currently associated with
entries in the track-list and simply appear vertically centered in the
timeline view.
* Added keyframes currently record the position and orientation of the
scene camera when added, and are rendered in the world with debug
visualizations, but serve no useful purpose. Some design work is needed
(both user-experience and architectural) to decide on a workflow for
actually animating scene elements.
* Entity deletion cannot currently be undone/redone
This PR adds support for Mixamo FBX imports (via backend conversion to
glTF/glB) and some miscellaneous improvements to the animation systems.
### User-facing improvements
* Retargeting math has been fixed to eliminate jittering issues
* Retargeter can now handle animated bone translations when present
(generally this is only used for root motion)
### Housekeeping / Code improvements
* Animation-related components and system logic have been moved to a
dedicated `anim` module and `AnimPlugin`
* Retargeting stuff has been moved from the `bvh` module to
`anim::retargeting`
* Component-based solution for marking an entity as an animation
retargeting source makes it easy to extend the retargeting functionality
to new types of source animations
### Limitations
Currently the code assumes that there will be at most one animation
source at a time. That single animation (if present) will be retargeted
to all other skeletons in the scene. A component-based solution for
directing a specific target skeleton to use a specific source animation
is possible, but will require some additional consideration around the
implementation design for the read-only "viewer" modes.
### Test animations:
* Hip-Hop Dancing (Mixamo): ?mode=viewer&mixamo=mocap/hip-hop-dancing.gltf
* Shuffle (MocapNET): ?mode=viewer&bvh=mocap/shuffle.bvh
* Scott (MocapNET): ?mode=viewer&bvh=mocap/scott.bvh
* Scott 1 (MocapNET): ?mode=viewer&bvh=mocap/scott1.bvh
This PR replaces the hard-coded BVH animation with a URL parameter, and
adds a specialized BVH viewer web component and entry-point function for
previewing the specified animation on our generic mannequin.
This PR adds a `StudioMode` resource to toggle editing capabilities, and
configures the frontend application to read the desired mode from a URL
parameter.
For now, setting the app to `StudioMode::Viewer` with
`<app-url>?mode=viewer` hides the Transform Toolbar and disables most of
the interactive features of the app, but otherwise exhibits the same
behavior as `StudioMode::Editor` / `<app-url>?mode=studio`.
In the future, the "Viewer" and "Editor" modes will receive different
initialization inputs to better serve the needs of their use cases. In
particular:
* The "Viewer" mode will accept a BVH animation handle, which should be
played in a loop on a simple mannequin asset, allowing a user to preview
a given animation before using it within the full studio workflow.
* The "Editor" mode will accept a full Bevy scene handle which can be
edited and saved to the user's account. These scenes will serve as the
basis for the generative-AI workflows Storyteller Studio is ultimately
seeking to facilitate.
* It will likely be useful for "Editor" mode to alternatively accept
only a BVH animation handle, which would be used to initialize a new
scene for editing similar to "Viewer" mode, but with editing
capabilities enabled.
This PR does some rudimentary cleanup / reorganizing to the
`studio-frontend` project.
### Changes
* Refactored `studio-frontend` to use the `<sts-canvas-provider/>` and
`<sts-viewer/>` components from `studio-web`
* Added some CSS layout utilities for positioning overlayed UI elements
* Extracted `<sts-toolbar/>` and `<sts-toolbar-group/>` to their own
components
* Extracted `<sts-transform-toolbar/>` to its own component
* Add plugins for `wasm` and `top-level-await` to Vite config
* Avoid bundling dependencies
* Add external dependencies to project-level `package.json` manifests
This PR adds a `CanvasProviderElement` to `@storyteller/studio-web` to manage
the complexity of maintaining a single concrete `HTMLCanvasElement` for
`@storyteller/studio`, while still being able to treat components like the
`ViewerElement` as self-contained "views" that behave as expected with regards
to CSS styling, DOM placement, etc.
Consuming applications should wrap the new `<sts-canvas-provider />` element
around some common ancestor of all views into the `@storyteller/studio` app,
similar to a React context provider.
The `react-frontend` application has been updated to take advantage of the new
`CanvasProviderElement` by demonstrating a more realistic use case for the
`ViewerElement`. The new demo presents a grid of "thumbnails," one for each
object in our hard-coded "object library." Clicking on one of these thumbnails
will render a `ViewerElement` for that object in a modal dialog.
This commit implements a Sketchfab-style single-object viewer as a minimal
proof-of-concept.
### `@storyteller/studio-web`
This is a new package to house web components for interfacing with the
`@storyteller/studio` WASM library. It currently includes a single component,
`<sts-viewer />`, which is exported from `@storyteller/studio-web/viewer`.
### `react-frontend`
This is a new application intended as a proof-of-concept and testbed for React
interop with `@storyteller/studio-web`.
Web components are currently a little bit fiddly to use in a React application.
`react-frontend/src/app/studio-viewer.tsx` demonstrates how a React-idiomatic
component can be written as a thin wrapper around a web component, and
`react-frontend/src/app/app.tsx` demonstrates that wrapper component in use.
Brandon Thomas