This PR exposes skeletal joint entities to the frontend hierarchy,
allowing the user to pose rigged models by selecting the joints and
transforming them.
Additionally, entity-type information (e.g., Mesh, Skeleton, Bone,
Light, etc.) has been added to the data structure passed to the
frontend, which is rendered with icons in the tree view.
This PR Adds the ability to set the color of the skybox using the URL
parameters.
- Now you can pass a color's hexadecimal number (without the #) in the
skybox parameter, and that will set the skybox to that color.
This PR heavily refactors the entry-point functions in
`studio/src/lib.rs` to simplify and consolidate. Additionally, the
frontend application should now be configurable as expected with any
valid combination of the following URL parameters:
* `objectId` : A pre-bundled library object to load (e.g., "couch.gltf")
* `bvh` : Any resolvable path to a BVH animation with a MocapNET
skeleton (e.g., "mocap/shuffle.bvh")
* `mixamo` : Any resolvable path to a glTF animation with the Mixamo
skeleton (e.g., "mocap/hip-hop-dancing.gltf")
* `sceneImport` : Any resolvable path to an arbitrary glTF scene
(untested, but should theoretically work)
* `scene` : Any resolvable path to a previously saved Storyteller Studio
scene in `*.scn.ron` format
* `skybox` : The ID of a pre-bundled skybox asset. Valid options are:
- "gum_trees_4k"
- "kloofenfal_28d_misty_4k"
- "meadow_4k"
- "promenade_de_vidy_4k"
- "scythian_tombs_4k"
- "test_scene" (this is the default if not specified)
* `mode` : One of "studio" or "viewer" (the default if not specified)
"Validity" rules are as follows:
* `objectId`, `bvh`, `mixamo`, `sceneImport` and `scene` are mutually
exclusive
* `skybox` is not supported for `scene` (the skybox should be saved and
loaded as part of the scene description, though this is not currently
working)
* Both `mode` options are valid for all other parameters
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.
### Design Overview
* The application renders a "split-screen" view, with Storyteller Studio
on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
- A screenshot is taken from the Bevy application
- Screenshot is piped into Stable Diffusion Turbo model
- Model output is rendered to the screen
* This all happens in as close to real-time as possible
### Work completed in this PR
* The `studio-frontend` project has been (somewhat crudely) refactored
to split most of its components into reusable libraries
- `studio-ui` is a use-case-agnostic design system library for core UI
components. These are "dumb" components which are controlled
via `@property` inputs and send `CustomEvent`s in response to user
input.
- `studio-frontend` is a collection of reusable Storyteller Studio
widgets. These are "smart" components which depend on
`@storyteller/studio` directly, manage their own state, and require very
little configuration or styling. They are designed to be drop-in
building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
- Currently, this simply grabs screenshots from the Bevy application and
renders those screenshots on the right side of the split-screen view. A
future PR will wire up to the Storyteller.ai backend to pipe these
screenshots through Stable Diffusion Turbo, and render that model's
output instead.
### Testing
* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
- Use the camera controller in the left side of the view to manipulate
the Bevy scene
- Verify the right side of the view updates with a fresh screenshot at a
rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
- Verify the right side view no longer updates while the user is
actively manipulating the editor
- This should allow the app to run smoother on lower-spec hardware
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds Storyteller.ai API integration for saving and loading
Storyteller Studio scenes to/from the backend.
### Known issues
* The skybox is not currently stored in the serialized scene format
This PR integrates asset retrieval from the Storyteller.ai back-end API,
and begins implementing long-term scene saving and storage.
### Media retrieval
Instead of fetching raw asset URLs, going forward, media files from
remote sources (e.g., in URL parameters) should be specified with a path
like the following:
```
remote://<token>.<filetype>
```
where
* `token` is a media token (e.g., "m_70b77neq9jsgmy5v31mekzchcbd223")
* `filetype` is a file extension appropriate for the type of file being
fetched (e.g., "gltf")
The `remote` protocol will be recognized by a [custom asset
reader](https://github.com/storytold/storyteller-bevy/blob/70726db111f6ccafd47feb055c5546540fef1786/studio/src/remote/reader.rs)
which will retrieve the file by calling a Storyteller API endpoint.
### Scene storage
Additionally, this PR begins laying the groundwork for
storing/retrieving Bevy scenes to/from the backend instead of solely
in-memory. This work is still incomplete -- as of this PR, a button has
been added to the web UI allowing the user to download the scene
file locally for testing/debugging purposes.
### Test URL
Media retrieval from the backend can be tested by [visiting this
URL](https://deploy-preview-49--storyteller-studio.netlify.app/?mode=viewer&mixamo=remote://m_70b77neq9jsgmy5v31mekzchcbd223.gltf)
in Chrome with the `--disable-web-security` flag (to bypass CORS errors)
This PR adds the ability to render out mixamo animations from the
headless mode. In addition to that, there were some bugs that have been
managed and some readability changes. Let me know if you have any
questions!
This PR adds a WIP timeline UI for animating elements in the scene.
### Implemented Features
* The timeline can be panned with MMB and zoomed with Ctrl + Mouse Wheel
* The timeline can be "scrubbed" by dragging the red "play-head" with
LMB
* The timeline sidebar (displaying the list of tracks) can be resized by
dragging on the edge of the sidebar
* Keyframes can be added by pressing the "Add Keyframe" CTA in the UI
* Keyframes can be selected for editing by clicking the diamond-shaped
markers in the timeline
* Selected entities (including but not limited to keyframes) can be
deleted by pressing the "Delete" key
* Keyframes can be moved temporally by dragging the diamond-shaped
markers in the timeline
### Known issues
* Zooming the timeline in too far results in a soft-lock
* When zooming the timeline, the "transform origin" is currently
centered on the `00:00` timestamp. For ease of use, it should center on
the mouse cursor position instead.
* The timeline track-list is currently hard-coded to list the Camera and
Camera Target. Clicking on these entries currently serves no useful
purpose and may result in a crash.
* Keyframe markers in the timeline are not currently associated with
entries in the track-list and simply appear vertically centered in the
timeline view.
* Added keyframes currently record the position and orientation of the
scene camera when added, and are rendered in the world with debug
visualizations, but serve no useful purpose. Some design work is needed
(both user-experience and architectural) to decide on a workflow for
actually animating scene elements.
* Entity deletion cannot currently be undone/redone
This PR Removes the gridlines from headless renders and Removes useless
frames from the end of headless renders. Really small PR. Let me know
what you think!
This is a small feature push that adds a simple Shader for the Gizmo.
This shader only takes one value: the color of the object. Very simple
and fast to compile.
This PR is missing:
- Shader Pre Compiling. This is something that I am looking into now,
however I thought it out of scope for this feature.
Just thought I would drop this real fast. Let me know what y'all think!
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds the capaibility to run Stroyteller studio from the command
line, rendering the result to images. This is the basic functionality,
and there are somethings that I am saving for later when we make this a
more fully fledged feature.
Limitations:
- Right now, you cannot do image output resolutions that aren't 2^n.
This is a bug I plan on tackling later.
If you have questions on how the command line works, use the --help
argument.
Let me know what you think!
---------
Co-authored-by: Danny McGee <dannymcgee@gmail.com>
This PR adds `cargo fmt -- --check`, `nx lint ...` and `nx test ...` as
GitHub Actions. This will cause future PRs to automatically fail when
formatting issues, Clippy warnings, or test failures are present in the
changed code.
This PR adds support for Mixamo FBX imports (via backend conversion to
glTF/glB) and some miscellaneous improvements to the animation systems.
### User-facing improvements
* Retargeting math has been fixed to eliminate jittering issues
* Retargeter can now handle animated bone translations when present
(generally this is only used for root motion)
### Housekeeping / Code improvements
* Animation-related components and system logic have been moved to a
dedicated `anim` module and `AnimPlugin`
* Retargeting stuff has been moved from the `bvh` module to
`anim::retargeting`
* Component-based solution for marking an entity as an animation
retargeting source makes it easy to extend the retargeting functionality
to new types of source animations
### Limitations
Currently the code assumes that there will be at most one animation
source at a time. That single animation (if present) will be retargeted
to all other skeletons in the scene. A component-based solution for
directing a specific target skeleton to use a specific source animation
is possible, but will require some additional consideration around the
implementation design for the read-only "viewer" modes.
### Test animations:
* Hip-Hop Dancing (Mixamo): ?mode=viewer&mixamo=mocap/hip-hop-dancing.gltf
* Shuffle (MocapNET): ?mode=viewer&bvh=mocap/shuffle.bvh
* Scott (MocapNET): ?mode=viewer&bvh=mocap/scott.bvh
* Scott 1 (MocapNET): ?mode=viewer&bvh=mocap/scott1.bvh
This PR replaces the hard-coded BVH animation with a URL parameter, and
adds a specialized BVH viewer web component and entry-point function for
previewing the specified animation on our generic mannequin.
The initial BVH retargeting solution produced very jittery results. Much
of that jitter was eliminated by using linear interpolation between
frames instead of immediate stepped transitions, but some jittering
still remained, particularly noticeable in the finger motions.
This PR largely fixes remaining jitter issues by making a few changes:
#### Frame-time quantization
Previously, we had been using the `BvhAnim::timer` field incorrectly, by
manually resetting it after `Timer::just_finished` returned `true`, and
returning `0.0` as the interpolation parameter for that frame. This
resulted in a herky-jerky animation due to misalignment between the BVH
animation's frame time and the actual application refresh rate, meaning
we would lose fractions of a frame every time the timer elapsed and the
frame was incremented. This PR addresses that by never calling
`Timer::reset` and always using `Timer::percent` for the interpolation
parameter.
#### Matrix instability
This PR refactors the retargeting system to operate on `Affine3A`
matrices instead of `Mat4`s. `Affine3A` represents a SIMD-optimized 3x3
matrix + a translation vector, so using it provides a small performance
boost in addition to being slightly more numerically stable.
Additionally, we now periodically re-orthonormalize the 3x3 matrices
in-between operations to mitigate drifting due to the accumulation of
floating-point errors.
Just adding the mannequin from Scott into the main branch so that it can
be used.
- added the mannequin.gltf
- also now requiring the file extention when selecting file to load.
This is to add flexability with file selection. Let me know if this is a
bad idea.
This PR adds support for BVH animation files, with a custom animation
player and retargeting solution for our Blender Rigify-based skeletal
meshes.
---------
Co-authored-by: Brooks Palin <brookspalin710@gmail.com>
This PR refactors the `studio` project's entry-point functions for
better maintainability and ease of use for local development.
### Overview
* `start_internal` refactored to return a `bevy::app::App` instead of
calling `App::run`, allowing for finer-grained configuration of unique
use cases
* Temporary `start` function added for WASM
* `init` function removed
* `main` standalone entry-point refactored to call `start_internal`
### Fixes
* Crash on non-WASM builds due to missing `StudioMode` resource
* Warning on non-WASM builds due to an unused variable
This PR adds a `StudioMode` resource to toggle editing capabilities, and
configures the frontend application to read the desired mode from a URL
parameter.
For now, setting the app to `StudioMode::Viewer` with
`<app-url>?mode=viewer` hides the Transform Toolbar and disables most of
the interactive features of the app, but otherwise exhibits the same
behavior as `StudioMode::Editor` / `<app-url>?mode=studio`.
In the future, the "Viewer" and "Editor" modes will receive different
initialization inputs to better serve the needs of their use cases. In
particular:
* The "Viewer" mode will accept a BVH animation handle, which should be
played in a loop on a simple mannequin asset, allowing a user to preview
a given animation before using it within the full studio workflow.
* The "Editor" mode will accept a full Bevy scene handle which can be
edited and saved to the user's account. These scenes will serve as the
basis for the generative-AI workflows Storyteller Studio is ultimately
seeking to facilitate.
* It will likely be useful for "Editor" mode to alternatively accept
only a BVH animation handle, which would be used to initialize a new
scene for editing similar to "Viewer" mode, but with editing
capabilities enabled.
* Add plugins for `wasm` and `top-level-await` to Vite config
* Avoid bundling dependencies
* Add external dependencies to project-level `package.json` manifests
This PR adds a `CanvasProviderElement` to `@storyteller/studio-web` to manage
the complexity of maintaining a single concrete `HTMLCanvasElement` for
`@storyteller/studio`, while still being able to treat components like the
`ViewerElement` as self-contained "views" that behave as expected with regards
to CSS styling, DOM placement, etc.
Consuming applications should wrap the new `<sts-canvas-provider />` element
around some common ancestor of all views into the `@storyteller/studio` app,
similar to a React context provider.
The `react-frontend` application has been updated to take advantage of the new
`CanvasProviderElement` by demonstrating a more realistic use case for the
`ViewerElement`. The new demo presents a grid of "thumbnails," one for each
object in our hard-coded "object library." Clicking on one of these thumbnails
will render a `ViewerElement` for that object in a modal dialog.
This commit implements a Sketchfab-style single-object viewer as a minimal
proof-of-concept.
### `@storyteller/studio-web`
This is a new package to house web components for interfacing with the
`@storyteller/studio` WASM library. It currently includes a single component,
`<sts-viewer />`, which is exported from `@storyteller/studio-web/viewer`.
### `react-frontend`
This is a new application intended as a proof-of-concept and testbed for React
interop with `@storyteller/studio-web`.
Web components are currently a little bit fiddly to use in a React application.
`react-frontend/src/app/studio-viewer.tsx` demonstrates how a React-idiomatic
component can be written as a thin wrapper around a web component, and
`react-frontend/src/app/app.tsx` demonstrates that wrapper component in use.
* Migrate `System::pipe(...)` calls to `System::map(...)`
* Migrate `EventReader::iter()` and `EventReader::into_iter()` calls to
`EventReader::read()`
* Migrate asset handling
* Generate `*.meta` files for assets
* Migrate a shader import
* Fix emissive materials double-applying their emissive strength
* Fix non-normalized bone weights resulting in incorrect vertex positions
* Set shadow filtering to `Hardware2x2` to mitigate increase in shader
compilation times
This PR adds the bones of a user-facing scene inspector.
### Work Done
To facilitate this work, the Bevy/frontend interface has been expanded to
accommodate two-way communications:
- The Bevy app can now dispatch JavaScript `CustomEvent`s via
`wasm::dispatch_to_js(...)`
- The frontend application can receive those events by listening for the given
event type on the `window` object
In its current state, the inspection system works by attaching the `Inspectable`
marker component to any entities which should be exposed to the user. A Bevy
system runs on `Update` to check for newly spawned or despawned entities with
that component, and emits events to notify the frontend. The frontend listens
for those events and updates its model of the scene accordingly, which is then
displayed with a simple tree view.
### Future Enhancements
There are many improvements that must be made to make this system actually
useful, but this PR is being expedited to focus on migrating to Bevy 0.12. A
shortlist of planned updates:
- Identify and expose the "type" of entity (e.g. lights, cameras, meshes,
skeletons, etc.) via a field on the `Inspectable` component
- Highlight the currently selected entity in the frontend hierarchy
- Ask the Bevy app to update the selection when the user selects an entry in the
hierarchy
- Add a separate UI panel to inspect and modify the selected entity's components
via the frontend
This PR adds functional transform gizmos, along with some new frontend UI to
configure their state.
## Work Done
- [x] Global / Local space toggling
- [x] Gizmo type toggling
- [x] Translation gizmo
- [x] Axis-constrained translation
- [x] Plane-constrained translation
- [x] Screen-space translation
- [x] Rotation gizmo
- [x] Axis-constrained rotation
- [x] Screen-space rotation
- [x] "Trackball" rotation
- [ ] Scale gizmo
- [ ] Single-axis non-uniform scaling
- [ ] Two-axis non-uniform scaling
- [ ] Uniform scaling
- [ ] Combination gizmos
- [x] Translation + Rotation
- [ ] Translation + Scale
- [ ] Rotation + Scale
- [ ] Translation + Rotation + Scale
## Future Enhancements
### Better visual feedback during transformations
Most DCCs update the visual representation of the transform gizmo while it's in
the process of being actively manipulated. (For example, Blender draws lines in
world-space to indicate that the transformation is constrained to a particular
axis or plane, hides widgets that aren't participating in the current
transformation, and illustrates the angle of difference between the original and
updated rotation.) At a minimum, we should indicate the current widget being
manipulated with a color change (like we already do when the widget is being
hovered). This requires a bit of additional state management effort but
shouldn't be overly difficult to add in a future PR.
### Better visual representation for rotation widgets
Currently, each axis of rotation is represented on the gizmo with a full-circle,
unlit "ring" shape. This can make it difficult to tell which half of the ring is
closer to the viewer, and thus to get an accurate understanding of how the
transform is currently oriented, without moving the camera around. A [previous
version of this work](#4) bisected the rings along a view-oriented plane and
rendered the "back-side" ring halves with reduced opacity. This PR uses static
meshes instead of an immediate-mode drawing API to render the gizmos, so a
custom shader would be the ideal way to reproduce that effect. That effort
should be deferred until after upgrading to Bevy 0.12, as it brings some changes
which would likely require some updates to that code.
### Better visual representation for translation arrows
A [previous version of this work](#4) faded the opacity of the translation
arrows when they were very close to parallel with the view direction — an idea
that was borrowed from Blender. It's a nice UX enhancement because translating
along an axis that's very close to parallel with the view direction can be janky
and error-prone, so it makes sense to disallow it and instead provide an
unobstructed view of the screen-space translation widget. This would also
require a custom shader, in addition to some extra state management logic (to
disable the widgets when their opacity is zeroed out). Given that it's only a
"nice-to-have," I think it makes sense to backlog this for the time being.
## Known Issues
### Picking
There's currently a bug in `bevy_mod_picking` that affects the hit-testing of
the gizmo widgets. The gizmos are currently _rendering_ on top of everything
else (via a second camera and render-layer configurations), but the picking
behavior doesn't take the rendering order into account. This means that portions
of the gizmo widgets which are occluded by geometry in the world cannot be
interacted with (because the occluding geometry interrupts the picking raycast).
> Update: This is fixed in the new release of `bevy_mod_picking` for Bevy 0.12.
### Delay on first-render
The first time a new widget shape is spawned, there's a noticeable delay (of 1
to several seconds) before the widget is rendered, during which the UI becomes
unresponsive. This is likely the same issue that triggers a much longer delay
when the glTF scene is being spawned. I'd like to see if Bevy 0.12 resolves or
mitigates this issue before addressing it on our end, but if it doesn't, I think
we can simply spawn an instance of each of the gizmo widgets off-screen on
application startup to prevent lag spikes from happening while a user is
interacting with the scene.
### Non-deterministic hover states
The widget colors are currently being updated by, e.g., multiplying on
mouser-over and dividing on mouse-out. Occasionally those events happen in quick
enough succession that our mouse-out listener is not invoked, so a widget gets
"stuck" in its hovered state. The fix for this is trivial but fairly annoying,
so I haven't gotten around to it yet. 🙂
> Update: This issue mostly occurs when in world-space mode. The gizmo is torn
> down and re-built from scratch once a drag event completes, which can despawn
> the gizmo while an element is still hovered. The solution will likely involve
> completely separate materials for default and hover states, similar to how
> `bevy_mod_picking`'s highlight plugin works.
### Object Selection
There's still some jank happening with object selection and camera orbiting, as
mentioned in #8. I don't anticipate this being overly difficult to resolve, but
I'm deferring that work until more of the core functionality is built out since
it's not a blocker for now.
### Skinned Mesh Selection
There's also still the issue of the skinned mesh being unselectable — which, it
turns out, is [technically not actually the
case](https://github.com/bevyengine/bevy/issues/4971). Rather, the picking
raycast "sees" the mesh as if it were still in its original rest pose, because
the mesh deformations are processed on the GPU and not reflected back to the
CPU-side representation.
I think the ideal workaround for this will be to attach capsule-shaped Rapier
colliders to the skeleton (similar to Unreal Engine's auto-generated "physics
assets" for skeletal meshes) and use those for picking instead. This would
automatically enable skeletal pose manipulation (by transforming the individual
joints), which is already a long term goal. It also makes more sense from a
practical standpoint, as the mesh itself cannot be independently transformed
while it's bound to the skeleton, so there's little point in directly selecting
it.
#8 introduced some additional debug information to the view, which was already
rendering some information and visualizations which were not necessary relevant
to the end-user experience. This PR adds an input action (bound to Ctrl + `) to
toggle the display of those visualizations, and disables them by default.
This PR implements basic object selection, with help from `bevy_mod_picking` and
`bevy_mod_outline`.
### Work Done
* Click to select objects in the world
* Selected objects become visually highlighted
* The selected entity is available globally on the `Res<Selection>` resource
* Selected entities can also be queried via the `Selected` marker component
* Pressing `F` will "focus" the camera on the selected entity
* Pressing `Esc` will clear the active selection
### Known Issues
* We're currently using `bevy_mod_picking`'s "Selection" feature, which isn't
aware of our own input events. As a consequence, it will select the object
under the cursor when releasing `LMB` after orbiting the camera, which is not
what we want. This should be pretty easily fixable by dropping
`bevy_mod_picking`'s selection feature and reading the object under the
cursor in our own click listener that only runs if the LMB event is
unhandled.
* The skinned mesh is not currently selectable. This is likely fixable by
adjusting the logic in `scene::mark_scene_objects`... somehow. The `Gltf`
data structure is kind of a nightmare to work with, so this will take some
investigation.
### Future Enhancements
* For parity with other DCC apps, the "Focus" action should also adjust the
camera zoom level to "frame" the selected object in the view. Punting on that
for now because I'm not super confident about the math involved.
* Group selection with Shift/Ctrl and/or marquee selection would be really nice
to have, but probably not critical for now.
### New `math` module
* Traits for error-tolerant floating-point comparisons
* Traits for interpolation and easing
* Bevy plugin to manage 3D interpolations with ECS components
### `skybox` module refactor
* ECS resource to enable external skybox selection/updates
* System run conditions for more idiomatic/ECS-friendly control flow
### Camera controller refactor
* Renamed from `orbit_cam` to `camera_controller`
* Now delegates to the new interpolation plugin for WASD motion smoothing
* Simple API for attaching the `CameraController` component to the camera from
another module -- this resolves an awkward separation-of-concerns situation
where we had to configure a bunch of render-quality settings in the camera
controller module
### Scene module refactor
* Renamed from `test_scene` to `scene`
* This module is now responsible for configuring the skybox
* This module is now responsible for spawning the camera and configuring the
render settings
Adds `Shift` as an alternate modifier for FPS-style camera rotation, and binds
wheel events to camera zoom/dolly.
An attempt was made to translate 2D scroll events on a touchpad to camera
panning (this is what Blender does, and it's smooth as butter), but Bevy and/or
`winit` and/or the browser is doing some weird preprocessing of those events
before we're able to read them, which makes that unfeasible for now. That issue
has been documented with a `FIXME` comment.
This PR adds FPS-style controls to the camera in addition to the existing DCC-style pan/orbit/dolly controls. Additionally, some architectural changes have been made to facilitate this and pave the way toward making all of the app controls configurable in the future. Namely, a new `input` module has been added, loosely inspired by Unreal Engine's "Enhanced Input" system (and as I understand it, conceptually similar to Unity's "new" input system as well).
The basic idea is that raw device inputs (coming from `bevy::input`) are mapped to named, app-specific "input actions" before being consumed by other parts of the application. For example, in the camera motion system, instead of checking for mouse motion while the middle mouse button is pressed, we now check for a `CameraMotion::Pan` input action event.
Input actions carry information about their lifecycle phase (`Start`, `Ongoing`, and `Stop` — a stand-in for Bevy's `just_pressed`, `pressed`, and `just_released` methods) and some type of magnitude (`f32`, `Vec2`, or `Vec3` depending on the action).
At present, we've simply moved the hard-coded device input checks from the `orbit_cam` module to the new `input` module, but theoretically, this could eventually be refactored to read the mappings between device inputs and input actions from some kind of user configuration.
Danny McGee