This PR fixes two issues that can cause a degraded user experience under
certain conditions:
### Focus-stealing
When focus lands on the `document.body`, it's typically bad for
accessibility and a sign of improper input handling. This generally
occurs when the currently-focused element is removed from the DOM. In
our particular case, it also prevents the Bevy application from
correctly handling keyboard events.
Previously, the problem of `document.body` focus had been fixed by
checking the `document.activeElement` every frame, and redirecting focus
to the Bevy canvas whenever it landed on the `document.body`. However,
when running in an iframe embedded on https://fakeyou.com, our
application will report `document.body` as the `document.activeElement`
whenever the parent application has focus, effectively stealing focus
from the parent application and making it impossible for the user to,
e.g., interact with text fields in the parent app.
This PR removes that focus-stealing behavior. Instead, we should
properly implement the ARIA specifications for any elements that may
disappear from the DOM so that user focus is managed in a precise and
predictable way. That will be addressed in follow-up PRs.
### "Stuck" input states
In certain circumstances, the browser window hosting our application may
lose focus while certain keys/buttons are held down which are bound to
long-running actions in the Bevy application, like camera manipulation.
When that occurs, Bevy is unable to detect when the bound keys/buttons
are released, causing the input state to become "stuck" reading those
keys/buttons as being pressed.
As a workaround, this PR introduces a listener for "blur" events on the
`window`, which dispatches synthetic "keyup"/"pointerup" events to the
Bevy canvas for all affected keyboard keys and mouse buttons.
This PR updates the Transform > Rotation section of the Entity Inspector
to display Euler angles in degrees instead of the vector component of
the input quaternion.
Additionally, this updates the `studio/ui/inspector` module to add a
special-case function for updating the selected entity's rotation from
the inspector in a way that's consistent with the behavior of the
rotation gizmo.
### Known Issues
The values displayed for the angles in the inspector are not always
intuitive, especially while actively manipulating them. To some extent
this is just the nature of Euler angles (e.g., Unreal Engine exhibits
similar issues), but I suspect there's room for improvement of our
implementation (e.g., Unreal's rotation display is typically _more_ sane
than ours).
Regardless, fixing this requires a firmer grasp of the math than I
possess, so a fix for this issue is not currently planned.
Issue turned out to be that in some circumstances, the `group_by` iterator will create multiple groups for a single key - and sometimes those groups can be empty. This lead to situations where an animation track will end up empty until enough changed to re-shuffle things in the query iterator.
The fix is switching to `into_group_map_by` - which will put all grouped items into a hashmap instead, ensuring no duplicate keys.
This PR adds Material component inspection to the frontend Entity
Inspector, and adds functional editing capabilities for the Transform
and Material inspectors.
### Changes
* Added a `ScalarFieldElement` for numeric form fields, with a hybrid
slider/text-input UX inspired by Blender, Unreal Engine, etc.
* Added `MaterialInspectorElement` for inspecting the Bevy
`StandardMaterial` component
* Wired up the `TransformInspectorElement` and
`MaterialInspectorElement` to apply changes made in the UI to the Bevy
scene
* Refactored the `studio/ui/inspector` module to output `Result::Err`s
to the dev console
* Refactored the `studio/interaction/selection` module to use only the
`Selection` resource, removing the redundant `Selected` component which
added some unnecessary complexity. This fixes#39
* Fixed primitive shapes not receiving a selection outline when selected
### To-Do
These tasks will be addressed in a forthcoming PR:
* The **Transform** > **Rotation** section in the inspector is still not
displaying sensible values. Currently it's just displaying the raw
quaternion `xyz` vector rather than displaying the orientation in
human-readable terms (e.g., as Euler angles).
* We're currently ignoring the Primitive Shape component, which would
allow the user to configure some additional geometric properties of the
shape or replace it with a different one.
---------
Co-authored-by: rewin <rewin1996@gmail.com>
…to avoid re-processing timeline info
- move generating animation to first thing in a frame, and notifying
about timeline changes to last, to ensure that it happens before the
time updates
- re-generate the animation tweens whenever you start playing or
manually seek a new time in the timeline to prevent inconsistencies
This PR adds a system to find animation clips in imported glTF scenes
and auto-play them on their associated entities.
The new system runs only in "Viewer" mode, for the purposes of
previewing a scene import on its Storyteller.ai Media File page. In
"Editor" mode, we will eventually expose animation clips to the user
through some other mechanism (TDB -- needs design work, grooming,
etc.).
Problems with the previous code:
- `async-compat` is necessary for headless mode (cli server rendering)
to load remote scenes from the API without blowing up. Without it, the
code complains about the lack of a Tokio runtime.
- Unfortunately, `async-compat` doesn't work with WASM and the entire
web build is broken by its presence.
This change introduces conditional compilation so `async-compat` is only
leveraged in headless mode.
This PR adds an initial animation timeline.
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds the camera motions to the composite mode. There are
currently 4 camera motion modes: `Orbit`, `Static`, `Pan`, `Zoom`. They
are accessed by using the `--camera-motion [motion]` argument. For
example, if I want to render a scene with an orbiting motion, I would do
the following:
`studio-headless scene --camera-motion orbit scene.ron `
Some things to note:
- The argument `--orbit` no longer is there. It has been deprecated!
- Currently the speed is hard coded. This can be set as a command arg
later.
- This only works with scenes, as this is intended for the `sample-room`
scene. May not work great for other scenes
Let me know if there is anything that I can fix!
This PR adds the Scaling Gizmo to the StoryTeller Engine.
Features
- Objects can now be scaled by Axis or Plane
Limitations
- Currently there is no implementation for combining the scale gizmo
with any of the other gizmo. This should be amended with further PRs.
Let me know if there is anything that I missed!
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR cherry-picks some improvements from #63 that are general-purpose
and not specifically related to IK.
* Some logic has been added to `AppElement` to track the currently
selected Scene Element
* `disabled` property has been added to `ButtonElement`, with custom
styling and accessibility features
* Tagging entities with their `SceneElement` type has been made more
robust and reliable (previously, it was possible for all entities to
appear in the hierarchy as "Generic" elements depending on the order of
system execution)
Additionally:
* Forces the Netlify CI pipeline to always used the latest stable Rust
version
This PR exposes skeletal joint entities to the frontend hierarchy,
allowing the user to pose rigged models by selecting the joints and
transforming them.
Additionally, entity-type information (e.g., Mesh, Skeleton, Bone,
Light, etc.) has been added to the data structure passed to the
frontend, which is rendered with icons in the tree view.
This PR Adds the ability to set the color of the skybox using the URL
parameters.
- Now you can pass a color's hexadecimal number (without the #) in the
skybox parameter, and that will set the skybox to that color.
This PR heavily refactors the entry-point functions in
`studio/src/lib.rs` to simplify and consolidate. Additionally, the
frontend application should now be configurable as expected with any
valid combination of the following URL parameters:
* `objectId` : A pre-bundled library object to load (e.g., "couch.gltf")
* `bvh` : Any resolvable path to a BVH animation with a MocapNET
skeleton (e.g., "mocap/shuffle.bvh")
* `mixamo` : Any resolvable path to a glTF animation with the Mixamo
skeleton (e.g., "mocap/hip-hop-dancing.gltf")
* `sceneImport` : Any resolvable path to an arbitrary glTF scene
(untested, but should theoretically work)
* `scene` : Any resolvable path to a previously saved Storyteller Studio
scene in `*.scn.ron` format
* `skybox` : The ID of a pre-bundled skybox asset. Valid options are:
- "gum_trees_4k"
- "kloofenfal_28d_misty_4k"
- "meadow_4k"
- "promenade_de_vidy_4k"
- "scythian_tombs_4k"
- "test_scene" (this is the default if not specified)
* `mode` : One of "studio" or "viewer" (the default if not specified)
"Validity" rules are as follows:
* `objectId`, `bvh`, `mixamo`, `sceneImport` and `scene` are mutually
exclusive
* `skybox` is not supported for `scene` (the skybox should be saved and
loaded as part of the scene description, though this is not currently
working)
* Both `mode` options are valid for all other parameters
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds Storyteller.ai API integration for saving and loading
Storyteller Studio scenes to/from the backend.
### Known issues
* The skybox is not currently stored in the serialized scene format
This PR integrates asset retrieval from the Storyteller.ai back-end API,
and begins implementing long-term scene saving and storage.
### Media retrieval
Instead of fetching raw asset URLs, going forward, media files from
remote sources (e.g., in URL parameters) should be specified with a path
like the following:
```
remote://<token>.<filetype>
```
where
* `token` is a media token (e.g., "m_70b77neq9jsgmy5v31mekzchcbd223")
* `filetype` is a file extension appropriate for the type of file being
fetched (e.g., "gltf")
The `remote` protocol will be recognized by a [custom asset
reader](https://github.com/storytold/storyteller-bevy/blob/70726db111f6ccafd47feb055c5546540fef1786/studio/src/remote/reader.rs)
which will retrieve the file by calling a Storyteller API endpoint.
### Scene storage
Additionally, this PR begins laying the groundwork for
storing/retrieving Bevy scenes to/from the backend instead of solely
in-memory. This work is still incomplete -- as of this PR, a button has
been added to the web UI allowing the user to download the scene
file locally for testing/debugging purposes.
### Test URL
Media retrieval from the backend can be tested by [visiting this
URL](https://deploy-preview-49--storyteller-studio.netlify.app/?mode=viewer&mixamo=remote://m_70b77neq9jsgmy5v31mekzchcbd223.gltf)
in Chrome with the `--disable-web-security` flag (to bypass CORS errors)
This PR adds the ability to render out mixamo animations from the
headless mode. In addition to that, there were some bugs that have been
managed and some readability changes. Let me know if you have any
questions!
This PR adds a WIP timeline UI for animating elements in the scene.
### Implemented Features
* The timeline can be panned with MMB and zoomed with Ctrl + Mouse Wheel
* The timeline can be "scrubbed" by dragging the red "play-head" with
LMB
* The timeline sidebar (displaying the list of tracks) can be resized by
dragging on the edge of the sidebar
* Keyframes can be added by pressing the "Add Keyframe" CTA in the UI
* Keyframes can be selected for editing by clicking the diamond-shaped
markers in the timeline
* Selected entities (including but not limited to keyframes) can be
deleted by pressing the "Delete" key
* Keyframes can be moved temporally by dragging the diamond-shaped
markers in the timeline
### Known issues
* Zooming the timeline in too far results in a soft-lock
* When zooming the timeline, the "transform origin" is currently
centered on the `00:00` timestamp. For ease of use, it should center on
the mouse cursor position instead.
* The timeline track-list is currently hard-coded to list the Camera and
Camera Target. Clicking on these entries currently serves no useful
purpose and may result in a crash.
* Keyframe markers in the timeline are not currently associated with
entries in the track-list and simply appear vertically centered in the
timeline view.
* Added keyframes currently record the position and orientation of the
scene camera when added, and are rendered in the world with debug
visualizations, but serve no useful purpose. Some design work is needed
(both user-experience and architectural) to decide on a workflow for
actually animating scene elements.
* Entity deletion cannot currently be undone/redone
This PR Removes the gridlines from headless renders and Removes useless
frames from the end of headless renders. Really small PR. Let me know
what you think!
rewin