This PR heavily refactors the entry-point functions in
`studio/src/lib.rs` to simplify and consolidate. Additionally, the
frontend application should now be configurable as expected with any
valid combination of the following URL parameters:
* `objectId` : A pre-bundled library object to load (e.g., "couch.gltf")
* `bvh` : Any resolvable path to a BVH animation with a MocapNET
skeleton (e.g., "mocap/shuffle.bvh")
* `mixamo` : Any resolvable path to a glTF animation with the Mixamo
skeleton (e.g., "mocap/hip-hop-dancing.gltf")
* `sceneImport` : Any resolvable path to an arbitrary glTF scene
(untested, but should theoretically work)
* `scene` : Any resolvable path to a previously saved Storyteller Studio
scene in `*.scn.ron` format
* `skybox` : The ID of a pre-bundled skybox asset. Valid options are:
- "gum_trees_4k"
- "kloofenfal_28d_misty_4k"
- "meadow_4k"
- "promenade_de_vidy_4k"
- "scythian_tombs_4k"
- "test_scene" (this is the default if not specified)
* `mode` : One of "studio" or "viewer" (the default if not specified)
"Validity" rules are as follows:
* `objectId`, `bvh`, `mixamo`, `sceneImport` and `scene` are mutually
exclusive
* `skybox` is not supported for `scene` (the skybox should be saved and
loaded as part of the scene description, though this is not currently
working)
* Both `mode` options are valid for all other parameters
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds Storyteller.ai API integration for saving and loading
Storyteller Studio scenes to/from the backend.
### Known issues
* The skybox is not currently stored in the serialized scene format
This PR adds support for Mixamo FBX imports (via backend conversion to
glTF/glB) and some miscellaneous improvements to the animation systems.
### User-facing improvements
* Retargeting math has been fixed to eliminate jittering issues
* Retargeter can now handle animated bone translations when present
(generally this is only used for root motion)
### Housekeeping / Code improvements
* Animation-related components and system logic have been moved to a
dedicated `anim` module and `AnimPlugin`
* Retargeting stuff has been moved from the `bvh` module to
`anim::retargeting`
* Component-based solution for marking an entity as an animation
retargeting source makes it easy to extend the retargeting functionality
to new types of source animations
### Limitations
Currently the code assumes that there will be at most one animation
source at a time. That single animation (if present) will be retargeted
to all other skeletons in the scene. A component-based solution for
directing a specific target skeleton to use a specific source animation
is possible, but will require some additional consideration around the
implementation design for the read-only "viewer" modes.
### Test animations:
* Hip-Hop Dancing (Mixamo): ?mode=viewer&mixamo=mocap/hip-hop-dancing.gltf
* Shuffle (MocapNET): ?mode=viewer&bvh=mocap/shuffle.bvh
* Scott (MocapNET): ?mode=viewer&bvh=mocap/scott.bvh
* Scott 1 (MocapNET): ?mode=viewer&bvh=mocap/scott1.bvh
This PR replaces the hard-coded BVH animation with a URL parameter, and
adds a specialized BVH viewer web component and entry-point function for
previewing the specified animation on our generic mannequin.
This PR adds a `CanvasProviderElement` to `@storyteller/studio-web` to manage
the complexity of maintaining a single concrete `HTMLCanvasElement` for
`@storyteller/studio`, while still being able to treat components like the
`ViewerElement` as self-contained "views" that behave as expected with regards
to CSS styling, DOM placement, etc.
Consuming applications should wrap the new `<sts-canvas-provider />` element
around some common ancestor of all views into the `@storyteller/studio` app,
similar to a React context provider.
The `react-frontend` application has been updated to take advantage of the new
`CanvasProviderElement` by demonstrating a more realistic use case for the
`ViewerElement`. The new demo presents a grid of "thumbnails," one for each
object in our hard-coded "object library." Clicking on one of these thumbnails
will render a `ViewerElement` for that object in a modal dialog.
This commit implements a Sketchfab-style single-object viewer as a minimal
proof-of-concept.
### `@storyteller/studio-web`
This is a new package to house web components for interfacing with the
`@storyteller/studio` WASM library. It currently includes a single component,
`<sts-viewer />`, which is exported from `@storyteller/studio-web/viewer`.
### `react-frontend`
This is a new application intended as a proof-of-concept and testbed for React
interop with `@storyteller/studio-web`.
Web components are currently a little bit fiddly to use in a React application.
`react-frontend/src/app/studio-viewer.tsx` demonstrates how a React-idiomatic
component can be written as a thin wrapper around a web component, and
`react-frontend/src/app/app.tsx` demonstrates that wrapper component in use.
Danny McGee