This PR updates the Media Library feature to implement asset imports
from the local file system, and spawning assets from the library into
the scene. Additionally, some placeholder functionality has been
scaffolded for adding animations onto existing characters in the scene.
### Additional Changes
* Dialogs can be set to `nonBlocking` to allow interaction with
background elements (note: this is fraught for accessibility and likely
temporary, to facilitate testing while the UI/UX redesign is in
progress)
* Dialogs can be moved around the screen by dragging their title-bars
* The `scene` module has been moved into a folder with a separate
`wasm.rs` file for its submodule. (`scene/mod.rs` remains one of our
largest single Rust files and could probably stand to be partitioned
further.)
### To-Do
* Support for adding animations is not yet fully implemented, though
some prerequisite functionality has been scaffolded. Namely, the "Add to
Scene" button will be disabled if:
- There is a mix of animations and non-animations selected in the Media
Library
- There is more than one animation selected in the Media Library
- There is one animation selected in the Media Library, but no
animatable character selected in the scene
* Adding full Storyteller-format scenes to the existing scene is not a
use case that has been tested, and may not be a use case we wish to
support in the long-term, as some of the data stored in Storyteller
scenes are intended to be "singleton" resources that may conflict with
existing data in the scene.
**Attempting to spawn a Storyteller scene through the Media Library has
not been explicitly blocked for now, but may cause unexpected
behavior.** This needs additional UX consideration and grooming.
This PR fixes two issues that can cause a degraded user experience under
certain conditions:
### Focus-stealing
When focus lands on the `document.body`, it's typically bad for
accessibility and a sign of improper input handling. This generally
occurs when the currently-focused element is removed from the DOM. In
our particular case, it also prevents the Bevy application from
correctly handling keyboard events.
Previously, the problem of `document.body` focus had been fixed by
checking the `document.activeElement` every frame, and redirecting focus
to the Bevy canvas whenever it landed on the `document.body`. However,
when running in an iframe embedded on https://fakeyou.com, our
application will report `document.body` as the `document.activeElement`
whenever the parent application has focus, effectively stealing focus
from the parent application and making it impossible for the user to,
e.g., interact with text fields in the parent app.
This PR removes that focus-stealing behavior. Instead, we should
properly implement the ARIA specifications for any elements that may
disappear from the DOM so that user focus is managed in a precise and
predictable way. That will be addressed in follow-up PRs.
### "Stuck" input states
In certain circumstances, the browser window hosting our application may
lose focus while certain keys/buttons are held down which are bound to
long-running actions in the Bevy application, like camera manipulation.
When that occurs, Bevy is unable to detect when the bound keys/buttons
are released, causing the input state to become "stuck" reading those
keys/buttons as being pressed.
As a workaround, this PR introduces a listener for "blur" events on the
`window`, which dispatches synthetic "keyup"/"pointerup" events to the
Bevy canvas for all affected keyboard keys and mouse buttons.
This PR updates the Transform > Rotation section of the Entity Inspector
to display Euler angles in degrees instead of the vector component of
the input quaternion.
Additionally, this updates the `studio/ui/inspector` module to add a
special-case function for updating the selected entity's rotation from
the inspector in a way that's consistent with the behavior of the
rotation gizmo.
### Known Issues
The values displayed for the angles in the inspector are not always
intuitive, especially while actively manipulating them. To some extent
this is just the nature of Euler angles (e.g., Unreal Engine exhibits
similar issues), but I suspect there's room for improvement of our
implementation (e.g., Unreal's rotation display is typically _more_ sane
than ours).
Regardless, fixing this requires a firmer grasp of the math than I
possess, so a fix for this issue is not currently planned.
This PR adds Material component inspection to the frontend Entity
Inspector, and adds functional editing capabilities for the Transform
and Material inspectors.
### Changes
* Added a `ScalarFieldElement` for numeric form fields, with a hybrid
slider/text-input UX inspired by Blender, Unreal Engine, etc.
* Added `MaterialInspectorElement` for inspecting the Bevy
`StandardMaterial` component
* Wired up the `TransformInspectorElement` and
`MaterialInspectorElement` to apply changes made in the UI to the Bevy
scene
* Refactored the `studio/ui/inspector` module to output `Result::Err`s
to the dev console
* Refactored the `studio/interaction/selection` module to use only the
`Selection` resource, removing the redundant `Selected` component which
added some unnecessary complexity. This fixes#39
* Fixed primitive shapes not receiving a selection outline when selected
### To-Do
These tasks will be addressed in a forthcoming PR:
* The **Transform** > **Rotation** section in the inspector is still not
displaying sensible values. Currently it's just displaying the raw
quaternion `xyz` vector rather than displaying the orientation in
human-readable terms (e.g., as Euler angles).
* We're currently ignoring the Primitive Shape component, which would
allow the user to configure some additional geometric properties of the
shape or replace it with a different one.
---------
Co-authored-by: rewin <rewin1996@gmail.com>
This PR overhauls the Dialog abstractions in `studio-ui` to be more
flexible for a wider variety of use cases, and adds a placeholder
"Import Asset" sub-dialog to the Media Library dialog.
This PR adds a new "Add to Scene" dropdown menu for adding elements to
the current scene, and a new Media Library dialog for importing and/or
adding Storyteller Engine assets (characters, scenes, animations, etc.)
associated with the user's Storyteller.ai account.
The scope of this PR is limited to the UI — actual asset importing
behavior, and adding elements other than primitive shapes to the current
scene, will be implemented in follow-up PRs.
### Technical Details
In order to facilitate these changes, a new "overlay" abstraction has
been added to `@storyteller/studio-ui` (loosely modeled after
[`@angular/cdk/overlay`](https://material.angular.io/cdk/overlay/overview))
for managing DOM elements which should render in a self-contained
stacking context "above" the rest of the application.
Two new base UI components — `<sts-menu>` and `<sts-dialog>`
— have also been added to make use of this overlay system, but
it's worth noting that these new components have not yet been
exhaustively tested or thoroughly fleshed out, so they may require
significant changes in order to accommodate future use cases or address
unforeseen bugs.
### Testing
The Media Library makes some API calls which will run into CORS errors
in a normal browser. This can be bypassed by running Chrome with the
following command-line flags:
```
--disable-web-security --ignore-certificate-errors --user-data-dir=path/to/some/temporary/directory
```
Verify all of the following:
* The "Add to Scene" menu opens and closes as expected
* The primitive-shape menu items (Cube, Sphere, etc.) add the
corresponding elements to the scene
* The "Media Library..." menu item correctly opens the Media Library
dialog
* The Media Library's search bar and category filters work as expected
* A single asset can be selected by clicking on it
* Multiple assets can be selected by holding the "Ctrl" or "Cmd" key
* The selection can be cleared by clicking in an empty space in the
results grid
* An "Add to Scene" button appears if (and only if) one or more assets
are selected
* The Media Library dialog can be closed by:
- Clicking the "X" in the upper right-hand corner
- Pressing the "Esc" key
- Clicking anywhere outside of the dialog boundaries
Additionally, ensure that the Media Library dialog handles all of the
following situations gracefully:
* User is not logged in to fakeyou.com
* User is logged in but does not have any engine-compatible assets
uploaded (this can be tested by selecting any of the categories from the
sidebar, since the database has not yet been migrated to use the new
`media_file_class` field, so all of those categories should currently
yield an empty list)
* User is logged in and there are engine-compatible assets to list (this
can be tested by pressing the "Save" button in the main application
toolbar to save the current scene to your account, and selecting the
"Home" entry from the Media Library sidebar)
This PR adds the Scaling Gizmo to the StoryTeller Engine.
Features
- Objects can now be scaled by Axis or Plane
Limitations
- Currently there is no implementation for combining the scale gizmo
with any of the other gizmo. This should be amended with further PRs.
Let me know if there is anything that I missed!
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR cherry-picks some improvements from #63 that are general-purpose
and not specifically related to IK.
* Some logic has been added to `AppElement` to track the currently
selected Scene Element
* `disabled` property has been added to `ButtonElement`, with custom
styling and accessibility features
* Tagging entities with their `SceneElement` type has been made more
robust and reliable (previously, it was possible for all entities to
appear in the hierarchy as "Generic" elements depending on the order of
system execution)
Additionally:
* Forces the Netlify CI pipeline to always used the latest stable Rust
version
This PR exposes skeletal joint entities to the frontend hierarchy,
allowing the user to pose rigged models by selecting the joints and
transforming them.
Additionally, entity-type information (e.g., Mesh, Skeleton, Bone,
Light, etc.) has been added to the data structure passed to the
frontend, which is rendered with icons in the tree view.
This PR updates the UI/UX of the Image Generator Demo.
* Hierarchy moved to the left of the editor view instead of overlaying
it
* Hierarchy can now be resized
* Bevy canvas resizes trigger a screen-capture update
* Transform Toolbar now centered over the editor view instead of the
entire viewport
* Added Prompt and Seed text fields, with a refresh button for the Seed
This PR Adds the ability to set the color of the skybox using the URL
parameters.
- Now you can pass a color's hexadecimal number (without the #) in the
skybox parameter, and that will set the skybox to that color.
This PR heavily refactors the entry-point functions in
`studio/src/lib.rs` to simplify and consolidate. Additionally, the
frontend application should now be configurable as expected with any
valid combination of the following URL parameters:
* `objectId` : A pre-bundled library object to load (e.g., "couch.gltf")
* `bvh` : Any resolvable path to a BVH animation with a MocapNET
skeleton (e.g., "mocap/shuffle.bvh")
* `mixamo` : Any resolvable path to a glTF animation with the Mixamo
skeleton (e.g., "mocap/hip-hop-dancing.gltf")
* `sceneImport` : Any resolvable path to an arbitrary glTF scene
(untested, but should theoretically work)
* `scene` : Any resolvable path to a previously saved Storyteller Studio
scene in `*.scn.ron` format
* `skybox` : The ID of a pre-bundled skybox asset. Valid options are:
- "gum_trees_4k"
- "kloofenfal_28d_misty_4k"
- "meadow_4k"
- "promenade_de_vidy_4k"
- "scythian_tombs_4k"
- "test_scene" (this is the default if not specified)
* `mode` : One of "studio" or "viewer" (the default if not specified)
"Validity" rules are as follows:
* `objectId`, `bvh`, `mixamo`, `sceneImport` and `scene` are mutually
exclusive
* `skybox` is not supported for `scene` (the skybox should be saved and
loaded as part of the scene description, though this is not currently
working)
* Both `mode` options are valid for all other parameters
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds Storyteller.ai API integration for saving and loading
Storyteller Studio scenes to/from the backend.
### Known issues
* The skybox is not currently stored in the serialized scene format
This PR adds a WIP timeline UI for animating elements in the scene.
### Implemented Features
* The timeline can be panned with MMB and zoomed with Ctrl + Mouse Wheel
* The timeline can be "scrubbed" by dragging the red "play-head" with
LMB
* The timeline sidebar (displaying the list of tracks) can be resized by
dragging on the edge of the sidebar
* Keyframes can be added by pressing the "Add Keyframe" CTA in the UI
* Keyframes can be selected for editing by clicking the diamond-shaped
markers in the timeline
* Selected entities (including but not limited to keyframes) can be
deleted by pressing the "Delete" key
* Keyframes can be moved temporally by dragging the diamond-shaped
markers in the timeline
### Known issues
* Zooming the timeline in too far results in a soft-lock
* When zooming the timeline, the "transform origin" is currently
centered on the `00:00` timestamp. For ease of use, it should center on
the mouse cursor position instead.
* The timeline track-list is currently hard-coded to list the Camera and
Camera Target. Clicking on these entries currently serves no useful
purpose and may result in a crash.
* Keyframe markers in the timeline are not currently associated with
entries in the track-list and simply appear vertically centered in the
timeline view.
* Added keyframes currently record the position and orientation of the
scene camera when added, and are rendered in the world with debug
visualizations, but serve no useful purpose. Some design work is needed
(both user-experience and architectural) to decide on a workflow for
actually animating scene elements.
* Entity deletion cannot currently be undone/redone
This PR adds support for Mixamo FBX imports (via backend conversion to
glTF/glB) and some miscellaneous improvements to the animation systems.
### User-facing improvements
* Retargeting math has been fixed to eliminate jittering issues
* Retargeter can now handle animated bone translations when present
(generally this is only used for root motion)
### Housekeeping / Code improvements
* Animation-related components and system logic have been moved to a
dedicated `anim` module and `AnimPlugin`
* Retargeting stuff has been moved from the `bvh` module to
`anim::retargeting`
* Component-based solution for marking an entity as an animation
retargeting source makes it easy to extend the retargeting functionality
to new types of source animations
### Limitations
Currently the code assumes that there will be at most one animation
source at a time. That single animation (if present) will be retargeted
to all other skeletons in the scene. A component-based solution for
directing a specific target skeleton to use a specific source animation
is possible, but will require some additional consideration around the
implementation design for the read-only "viewer" modes.
### Test animations:
* Hip-Hop Dancing (Mixamo): ?mode=viewer&mixamo=mocap/hip-hop-dancing.gltf
* Shuffle (MocapNET): ?mode=viewer&bvh=mocap/shuffle.bvh
* Scott (MocapNET): ?mode=viewer&bvh=mocap/scott.bvh
* Scott 1 (MocapNET): ?mode=viewer&bvh=mocap/scott1.bvh
This PR replaces the hard-coded BVH animation with a URL parameter, and
adds a specialized BVH viewer web component and entry-point function for
previewing the specified animation on our generic mannequin.
This PR adds a `StudioMode` resource to toggle editing capabilities, and
configures the frontend application to read the desired mode from a URL
parameter.
For now, setting the app to `StudioMode::Viewer` with
`<app-url>?mode=viewer` hides the Transform Toolbar and disables most of
the interactive features of the app, but otherwise exhibits the same
behavior as `StudioMode::Editor` / `<app-url>?mode=studio`.
In the future, the "Viewer" and "Editor" modes will receive different
initialization inputs to better serve the needs of their use cases. In
particular:
* The "Viewer" mode will accept a BVH animation handle, which should be
played in a loop on a simple mannequin asset, allowing a user to preview
a given animation before using it within the full studio workflow.
* The "Editor" mode will accept a full Bevy scene handle which can be
edited and saved to the user's account. These scenes will serve as the
basis for the generative-AI workflows Storyteller Studio is ultimately
seeking to facilitate.
* It will likely be useful for "Editor" mode to alternatively accept
only a BVH animation handle, which would be used to initialize a new
scene for editing similar to "Viewer" mode, but with editing
capabilities enabled.
This PR does some rudimentary cleanup / reorganizing to the
`studio-frontend` project.
### Changes
* Refactored `studio-frontend` to use the `<sts-canvas-provider/>` and
`<sts-viewer/>` components from `studio-web`
* Added some CSS layout utilities for positioning overlayed UI elements
* Extracted `<sts-toolbar/>` and `<sts-toolbar-group/>` to their own
components
* Extracted `<sts-transform-toolbar/>` to its own component
* Add plugins for `wasm` and `top-level-await` to Vite config
* Avoid bundling dependencies
* Add external dependencies to project-level `package.json` manifests
This PR adds a `CanvasProviderElement` to `@storyteller/studio-web` to manage
the complexity of maintaining a single concrete `HTMLCanvasElement` for
`@storyteller/studio`, while still being able to treat components like the
`ViewerElement` as self-contained "views" that behave as expected with regards
to CSS styling, DOM placement, etc.
Consuming applications should wrap the new `<sts-canvas-provider />` element
around some common ancestor of all views into the `@storyteller/studio` app,
similar to a React context provider.
The `react-frontend` application has been updated to take advantage of the new
`CanvasProviderElement` by demonstrating a more realistic use case for the
`ViewerElement`. The new demo presents a grid of "thumbnails," one for each
object in our hard-coded "object library." Clicking on one of these thumbnails
will render a `ViewerElement` for that object in a modal dialog.
This commit implements a Sketchfab-style single-object viewer as a minimal
proof-of-concept.
### `@storyteller/studio-web`
This is a new package to house web components for interfacing with the
`@storyteller/studio` WASM library. It currently includes a single component,
`<sts-viewer />`, which is exported from `@storyteller/studio-web/viewer`.
### `react-frontend`
This is a new application intended as a proof-of-concept and testbed for React
interop with `@storyteller/studio-web`.
Web components are currently a little bit fiddly to use in a React application.
`react-frontend/src/app/studio-viewer.tsx` demonstrates how a React-idiomatic
component can be written as a thin wrapper around a web component, and
`react-frontend/src/app/app.tsx` demonstrates that wrapper component in use.
Danny McGee