This PR updates the Media Library feature to implement asset imports
from the local file system, and spawning assets from the library into
the scene. Additionally, some placeholder functionality has been
scaffolded for adding animations onto existing characters in the scene.
### Additional Changes
* Dialogs can be set to `nonBlocking` to allow interaction with
background elements (note: this is fraught for accessibility and likely
temporary, to facilitate testing while the UI/UX redesign is in
progress)
* Dialogs can be moved around the screen by dragging their title-bars
* The `scene` module has been moved into a folder with a separate
`wasm.rs` file for its submodule. (`scene/mod.rs` remains one of our
largest single Rust files and could probably stand to be partitioned
further.)
### To-Do
* Support for adding animations is not yet fully implemented, though
some prerequisite functionality has been scaffolded. Namely, the "Add to
Scene" button will be disabled if:
- There is a mix of animations and non-animations selected in the Media
Library
- There is more than one animation selected in the Media Library
- There is one animation selected in the Media Library, but no
animatable character selected in the scene
* Adding full Storyteller-format scenes to the existing scene is not a
use case that has been tested, and may not be a use case we wish to
support in the long-term, as some of the data stored in Storyteller
scenes are intended to be "singleton" resources that may conflict with
existing data in the scene.
**Attempting to spawn a Storyteller scene through the Media Library has
not been explicitly blocked for now, but may cause unexpected
behavior.** This needs additional UX consideration and grooming.
This PR updates the Transform > Rotation section of the Entity Inspector
to display Euler angles in degrees instead of the vector component of
the input quaternion.
Additionally, this updates the `studio/ui/inspector` module to add a
special-case function for updating the selected entity's rotation from
the inspector in a way that's consistent with the behavior of the
rotation gizmo.
### Known Issues
The values displayed for the angles in the inspector are not always
intuitive, especially while actively manipulating them. To some extent
this is just the nature of Euler angles (e.g., Unreal Engine exhibits
similar issues), but I suspect there's room for improvement of our
implementation (e.g., Unreal's rotation display is typically _more_ sane
than ours).
Regardless, fixing this requires a firmer grasp of the math than I
possess, so a fix for this issue is not currently planned.
This PR adds Material component inspection to the frontend Entity
Inspector, and adds functional editing capabilities for the Transform
and Material inspectors.
### Changes
* Added a `ScalarFieldElement` for numeric form fields, with a hybrid
slider/text-input UX inspired by Blender, Unreal Engine, etc.
* Added `MaterialInspectorElement` for inspecting the Bevy
`StandardMaterial` component
* Wired up the `TransformInspectorElement` and
`MaterialInspectorElement` to apply changes made in the UI to the Bevy
scene
* Refactored the `studio/ui/inspector` module to output `Result::Err`s
to the dev console
* Refactored the `studio/interaction/selection` module to use only the
`Selection` resource, removing the redundant `Selected` component which
added some unnecessary complexity. This fixes#39
* Fixed primitive shapes not receiving a selection outline when selected
### To-Do
These tasks will be addressed in a forthcoming PR:
* The **Transform** > **Rotation** section in the inspector is still not
displaying sensible values. Currently it's just displaying the raw
quaternion `xyz` vector rather than displaying the orientation in
human-readable terms (e.g., as Euler angles).
* We're currently ignoring the Primitive Shape component, which would
allow the user to configure some additional geometric properties of the
shape or replace it with a different one.
---------
Co-authored-by: rewin <rewin1996@gmail.com>
This PR overhauls the Dialog abstractions in `studio-ui` to be more
flexible for a wider variety of use cases, and adds a placeholder
"Import Asset" sub-dialog to the Media Library dialog.
This PR adds an initial animation timeline.
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds a new "Add to Scene" dropdown menu for adding elements to
the current scene, and a new Media Library dialog for importing and/or
adding Storyteller Engine assets (characters, scenes, animations, etc.)
associated with the user's Storyteller.ai account.
The scope of this PR is limited to the UI — actual asset importing
behavior, and adding elements other than primitive shapes to the current
scene, will be implemented in follow-up PRs.
### Technical Details
In order to facilitate these changes, a new "overlay" abstraction has
been added to `@storyteller/studio-ui` (loosely modeled after
[`@angular/cdk/overlay`](https://material.angular.io/cdk/overlay/overview))
for managing DOM elements which should render in a self-contained
stacking context "above" the rest of the application.
Two new base UI components — `<sts-menu>` and `<sts-dialog>`
— have also been added to make use of this overlay system, but
it's worth noting that these new components have not yet been
exhaustively tested or thoroughly fleshed out, so they may require
significant changes in order to accommodate future use cases or address
unforeseen bugs.
### Testing
The Media Library makes some API calls which will run into CORS errors
in a normal browser. This can be bypassed by running Chrome with the
following command-line flags:
```
--disable-web-security --ignore-certificate-errors --user-data-dir=path/to/some/temporary/directory
```
Verify all of the following:
* The "Add to Scene" menu opens and closes as expected
* The primitive-shape menu items (Cube, Sphere, etc.) add the
corresponding elements to the scene
* The "Media Library..." menu item correctly opens the Media Library
dialog
* The Media Library's search bar and category filters work as expected
* A single asset can be selected by clicking on it
* Multiple assets can be selected by holding the "Ctrl" or "Cmd" key
* The selection can be cleared by clicking in an empty space in the
results grid
* An "Add to Scene" button appears if (and only if) one or more assets
are selected
* The Media Library dialog can be closed by:
- Clicking the "X" in the upper right-hand corner
- Pressing the "Esc" key
- Clicking anywhere outside of the dialog boundaries
Additionally, ensure that the Media Library dialog handles all of the
following situations gracefully:
* User is not logged in to fakeyou.com
* User is logged in but does not have any engine-compatible assets
uploaded (this can be tested by selecting any of the categories from the
sidebar, since the database has not yet been migrated to use the new
`media_file_class` field, so all of those categories should currently
yield an empty list)
* User is logged in and there are engine-compatible assets to list (this
can be tested by pressing the "Save" button in the main application
toolbar to save the current scene to your account, and selecting the
"Home" entry from the Media Library sidebar)
This PR cherry-picks some improvements from #63 that are general-purpose
and not specifically related to IK.
* Some logic has been added to `AppElement` to track the currently
selected Scene Element
* `disabled` property has been added to `ButtonElement`, with custom
styling and accessibility features
* Tagging entities with their `SceneElement` type has been made more
robust and reliable (previously, it was possible for all entities to
appear in the hierarchy as "Generic" elements depending on the order of
system execution)
Additionally:
* Forces the Netlify CI pipeline to always used the latest stable Rust
version
This PR exposes skeletal joint entities to the frontend hierarchy,
allowing the user to pose rigged models by selecting the joints and
transforming them.
Additionally, entity-type information (e.g., Mesh, Skeleton, Bone,
Light, etc.) has been added to the data structure passed to the
frontend, which is rendered with icons in the tree view.
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
Danny McGee