Files
storyteller-bevy/studio/src/change_tracking.rs
T
Brooks Palin d51eec52c3 Bootstrap Image Generator application (#53)
This PR begins scaffolding the Stable Diffusion Turbo application (title
TBD), which is a separate frontend for Storyteller Engine which focuses
on image generation.

### Design Overview

* The application renders a "split-screen" view, with Storyteller Studio
  on the left, and the Stable Diffusion Turbo output on the right
* When changes are made in the Studio editor:
  - A screenshot is taken from the Bevy application
  - Screenshot is piped into Stable Diffusion Turbo model
  - Model output is rendered to the screen
* This all happens in as close to real-time as possible

### Work completed in this PR

* The `studio-frontend` project has been (somewhat crudely) refactored
  to split most of its components into reusable libraries
  - `studio-ui` is a use-case-agnostic design system library for core UI
    components. These are "dumb" components which are controlled
    via `@property` inputs and send `CustomEvent`s in response to user
    input.
  - `studio-frontend` is a collection of reusable Storyteller Studio
    widgets. These are "smart" components which depend on
    `@storyteller/studio` directly, manage their own state, and require very
    little configuration or styling. They are designed to be drop-in
    building blocks that can be reused between various frontends.
* The new application has been bootstrapped as `image-generator-demo`
  - Currently, this simply grabs screenshots from the Bevy application and
    renders those screenshots on the right side of the split-screen view. A
    future PR will wire up to the Storyteller.ai backend to pipe these
    screenshots through Stable Diffusion Turbo, and render that model's
    output instead.

### Testing

* Install dependencies as described in the repo's README
* Run `npx nx serve image-generator-demo`
* Open [localhost:4200](http://localhost:4200) in your browser
  - Use the camera controller in the left side of the view to manipulate
    the Bevy scene
  - Verify the right side of the view updates with a fresh screenshot at a
    rate of ~5 fps
* Toggle the "Realtime Capture" checkbox
  - Verify the right side view no longer updates while the user is
    actively manipulating the editor
  - This should allow the app to run smoother on lower-spec hardware

---------

Co-authored-by: Danny McGee <dannymcgee@gmail.com>
2024-02-26 17:13:07 -05:00

78 lines
2.0 KiB
Rust

use bevy::prelude::*;
pub struct ChangeTrackingPlugin;
impl Plugin for ChangeTrackingPlugin {
fn build(&self, app: &mut App) {
app.add_event::<EntityChangedEvent>();
// Add more of these systems for each component that we want to track
app.add_systems(Update, send_change_events::<Transform>);
#[cfg(feature = "wasm")]
app.add_systems(
Update,
wasm::send_change_events::<Transform>.map(bevy::utils::error),
);
}
}
#[derive(Component, Debug)]
pub struct ChangeTracker;
fn send_change_events<T: Component>(
query: Query<Entity, (Changed<T>, With<ChangeTracker>)>,
mut entity_change_writer: EventWriter<EntityChangedEvent>,
) {
for entity in query.iter() {
entity_change_writer.send(EntityChangedEvent(entity));
}
}
#[derive(Event, Deref)]
pub struct EntityChangedEvent(pub Entity);
#[cfg(feature = "wasm")]
mod wasm {
use bevy::{prelude::*, window::PrimaryWindow};
use wasm_bindgen::prelude::wasm_bindgen;
use crate::wasm;
use super::ChangeTracker;
#[wasm_bindgen(typescript_custom_section)]
const ENTITY_CHANGES_EVENT: &str = r#"
export interface EntityChangesEvent extends CustomEvent {
type: "entity-changes";
detail: string[];
}
declare global {
export interface GlobalEventHandlersEventMap {
"entity-changes": EntityChangesEvent;
}
}
"#;
pub(super) fn send_change_events<T: Component>(
q_changed_ents: Query<Entity, (Changed<T>, With<ChangeTracker>)>,
q_window: Query<&Window, With<PrimaryWindow>>,
) -> Result<(), String> {
if q_changed_ents.is_empty() {
return Ok(());
}
let Ok(window) = q_window.get_single() else {
return Ok(());
};
let entities = q_changed_ents
.iter()
.map(|ent| format!("{ent:?}"))
.collect::<Vec<_>>();
wasm::dispatch_to_js(window, "entity-changes", &entities)
}
}