Problems with the previous code:
- `async-compat` is necessary for headless mode (cli server rendering)
to load remote scenes from the API without blowing up. Without it, the
code complains about the lack of a Tokio runtime.
- Unfortunately, `async-compat` doesn't work with WASM and the entire
web build is broken by its presence.
This change introduces conditional compilation so `async-compat` is only
leveraged in headless mode.
This PR adds an initial animation timeline.
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds a new "Add to Scene" dropdown menu for adding elements to
the current scene, and a new Media Library dialog for importing and/or
adding Storyteller Engine assets (characters, scenes, animations, etc.)
associated with the user's Storyteller.ai account.
The scope of this PR is limited to the UI — actual asset importing
behavior, and adding elements other than primitive shapes to the current
scene, will be implemented in follow-up PRs.
### Technical Details
In order to facilitate these changes, a new "overlay" abstraction has
been added to `@storyteller/studio-ui` (loosely modeled after
[`@angular/cdk/overlay`](https://material.angular.io/cdk/overlay/overview))
for managing DOM elements which should render in a self-contained
stacking context "above" the rest of the application.
Two new base UI components — `<sts-menu>` and `<sts-dialog>`
— have also been added to make use of this overlay system, but
it's worth noting that these new components have not yet been
exhaustively tested or thoroughly fleshed out, so they may require
significant changes in order to accommodate future use cases or address
unforeseen bugs.
### Testing
The Media Library makes some API calls which will run into CORS errors
in a normal browser. This can be bypassed by running Chrome with the
following command-line flags:
```
--disable-web-security --ignore-certificate-errors --user-data-dir=path/to/some/temporary/directory
```
Verify all of the following:
* The "Add to Scene" menu opens and closes as expected
* The primitive-shape menu items (Cube, Sphere, etc.) add the
corresponding elements to the scene
* The "Media Library..." menu item correctly opens the Media Library
dialog
* The Media Library's search bar and category filters work as expected
* A single asset can be selected by clicking on it
* Multiple assets can be selected by holding the "Ctrl" or "Cmd" key
* The selection can be cleared by clicking in an empty space in the
results grid
* An "Add to Scene" button appears if (and only if) one or more assets
are selected
* The Media Library dialog can be closed by:
- Clicking the "X" in the upper right-hand corner
- Pressing the "Esc" key
- Clicking anywhere outside of the dialog boundaries
Additionally, ensure that the Media Library dialog handles all of the
following situations gracefully:
* User is not logged in to fakeyou.com
* User is logged in but does not have any engine-compatible assets
uploaded (this can be tested by selecting any of the categories from the
sidebar, since the database has not yet been migrated to use the new
`media_file_class` field, so all of those categories should currently
yield an empty list)
* User is logged in and there are engine-compatible assets to list (this
can be tested by pressing the "Save" button in the main application
toolbar to save the current scene to your account, and selecting the
"Home" entry from the Media Library sidebar)
This PR adds the camera motions to the composite mode. There are
currently 4 camera motion modes: `Orbit`, `Static`, `Pan`, `Zoom`. They
are accessed by using the `--camera-motion [motion]` argument. For
example, if I want to render a scene with an orbiting motion, I would do
the following:
`studio-headless scene --camera-motion orbit scene.ron `
Some things to note:
- The argument `--orbit` no longer is there. It has been deprecated!
- Currently the speed is hard coded. This can be set as a command arg
later.
- This only works with scenes, as this is intended for the `sample-room`
scene. May not work great for other scenes
Let me know if there is anything that I can fix!
This PR attempts to avoid bankrupting the company by running 15 minutes
of unit tests on unchanged code or re-downloading
`@fortawesome/fontawesome-pro` 5 times for every PR commit.
This PR adds the Scaling Gizmo to the StoryTeller Engine.
Features
- Objects can now be scaled by Axis or Plane
Limitations
- Currently there is no implementation for combining the scale gizmo
with any of the other gizmo. This should be amended with further PRs.
Let me know if there is anything that I missed!
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR cherry-picks some improvements from #63 that are general-purpose
and not specifically related to IK.
* Some logic has been added to `AppElement` to track the currently
selected Scene Element
* `disabled` property has been added to `ButtonElement`, with custom
styling and accessibility features
* Tagging entities with their `SceneElement` type has been made more
robust and reliable (previously, it was possible for all entities to
appear in the hierarchy as "Generic" elements depending on the order of
system execution)
Additionally:
* Forces the Netlify CI pipeline to always used the latest stable Rust
version
This PR exposes skeletal joint entities to the frontend hierarchy,
allowing the user to pose rigged models by selecting the joints and
transforming them.
Additionally, entity-type information (e.g., Mesh, Skeleton, Bone,
Light, etc.) has been added to the data structure passed to the
frontend, which is rendered with icons in the tree view.
This PR updates the UI/UX of the Image Generator Demo.
* Hierarchy moved to the left of the editor view instead of overlaying
it
* Hierarchy can now be resized
* Bevy canvas resizes trigger a screen-capture update
* Transform Toolbar now centered over the editor view instead of the
entire viewport
* Added Prompt and Seed text fields, with a refresh button for the Seed
This PR Adds the ability to set the color of the skybox using the URL
parameters.
- Now you can pass a color's hexadecimal number (without the #) in the
skybox parameter, and that will set the skybox to that color.
This PR heavily refactors the entry-point functions in
`studio/src/lib.rs` to simplify and consolidate. Additionally, the
frontend application should now be configurable as expected with any
valid combination of the following URL parameters:
* `objectId` : A pre-bundled library object to load (e.g., "couch.gltf")
* `bvh` : Any resolvable path to a BVH animation with a MocapNET
skeleton (e.g., "mocap/shuffle.bvh")
* `mixamo` : Any resolvable path to a glTF animation with the Mixamo
skeleton (e.g., "mocap/hip-hop-dancing.gltf")
* `sceneImport` : Any resolvable path to an arbitrary glTF scene
(untested, but should theoretically work)
* `scene` : Any resolvable path to a previously saved Storyteller Studio
scene in `*.scn.ron` format
* `skybox` : The ID of a pre-bundled skybox asset. Valid options are:
- "gum_trees_4k"
- "kloofenfal_28d_misty_4k"
- "meadow_4k"
- "promenade_de_vidy_4k"
- "scythian_tombs_4k"
- "test_scene" (this is the default if not specified)
* `mode` : One of "studio" or "viewer" (the default if not specified)
"Validity" rules are as follows:
* `objectId`, `bvh`, `mixamo`, `sceneImport` and `scene` are mutually
exclusive
* `skybox` is not supported for `scene` (the skybox should be saved and
loaded as part of the scene description, though this is not currently
working)
* Both `mode` options are valid for all other parameters
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds Storyteller.ai API integration for saving and loading
Storyteller Studio scenes to/from the backend.
### Known issues
* The skybox is not currently stored in the serialized scene format
This PR integrates asset retrieval from the Storyteller.ai back-end API,
and begins implementing long-term scene saving and storage.
### Media retrieval
Instead of fetching raw asset URLs, going forward, media files from
remote sources (e.g., in URL parameters) should be specified with a path
like the following:
```
remote://<token>.<filetype>
```
where
* `token` is a media token (e.g., "m_70b77neq9jsgmy5v31mekzchcbd223")
* `filetype` is a file extension appropriate for the type of file being
fetched (e.g., "gltf")
The `remote` protocol will be recognized by a [custom asset
reader](https://github.com/storytold/storyteller-bevy/blob/70726db111f6ccafd47feb055c5546540fef1786/studio/src/remote/reader.rs)
which will retrieve the file by calling a Storyteller API endpoint.
### Scene storage
Additionally, this PR begins laying the groundwork for
storing/retrieving Bevy scenes to/from the backend instead of solely
in-memory. This work is still incomplete -- as of this PR, a button has
been added to the web UI allowing the user to download the scene
file locally for testing/debugging purposes.
### Test URL
Media retrieval from the backend can be tested by [visiting this
URL](https://deploy-preview-49--storyteller-studio.netlify.app/?mode=viewer&mixamo=remote://m_70b77neq9jsgmy5v31mekzchcbd223.gltf)
in Chrome with the `--disable-web-security` flag (to bypass CORS errors)
This PR adds the ability to render out mixamo animations from the
headless mode. In addition to that, there were some bugs that have been
managed and some readability changes. Let me know if you have any
questions!
This PR adds a WIP timeline UI for animating elements in the scene.
### Implemented Features
* The timeline can be panned with MMB and zoomed with Ctrl + Mouse Wheel
* The timeline can be "scrubbed" by dragging the red "play-head" with
LMB
* The timeline sidebar (displaying the list of tracks) can be resized by
dragging on the edge of the sidebar
* Keyframes can be added by pressing the "Add Keyframe" CTA in the UI
* Keyframes can be selected for editing by clicking the diamond-shaped
markers in the timeline
* Selected entities (including but not limited to keyframes) can be
deleted by pressing the "Delete" key
* Keyframes can be moved temporally by dragging the diamond-shaped
markers in the timeline
### Known issues
* Zooming the timeline in too far results in a soft-lock
* When zooming the timeline, the "transform origin" is currently
centered on the `00:00` timestamp. For ease of use, it should center on
the mouse cursor position instead.
* The timeline track-list is currently hard-coded to list the Camera and
Camera Target. Clicking on these entries currently serves no useful
purpose and may result in a crash.
* Keyframe markers in the timeline are not currently associated with
entries in the track-list and simply appear vertically centered in the
timeline view.
* Added keyframes currently record the position and orientation of the
scene camera when added, and are rendered in the world with debug
visualizations, but serve no useful purpose. Some design work is needed
(both user-experience and architectural) to decide on a workflow for
actually animating scene elements.
* Entity deletion cannot currently be undone/redone
Lee-Orr