mirror of
https://github.com/storytold/craft-ai-integrations.git
synced 2026-11-02 18:06:49 +00:00
Jeremy
1a02afd6be
* refactor(engine-rten): let another engine reuse the SAM 2.1 request handling Loaded, Backend, Prediction, read_prompt, prompt_points and best_proposal become public. Visibility only: the click and box mapping, the picture kept between requests, the empty-mask answer and the size checks then live in one place for every SAM 2.1 engine. * feat(engine-burn): crate shell and the script that generates the SAM 2.1 architecture tools/burn/generate_sam21.py checks the two published ONNX files against the pinned size and SHA-256, runs the pinned burn-onnx 0.22.0 (an example of craft-ai-burn-setup), applies the five asserted fixes, drops the generated loaders that panic on a bad file, splits by submodule so every file is under 700 lines, formats and writes. --check proves the tree is what the script produces. REVISION must equal the saved-weights cache revision. Cargo.lock additions are Burn with wgpu and ndarray (generated). * feat(burn-setup): prepare SAM 2.1 weights for Burn once, on this computer Brings in craft-ai-burn-setup (feature `conversion`): checks the two published ONNX files by size and SHA-256, converts them to Burn weights with the pinned burn-onnx 0.22.0, and keeps one checked, locked, shared copy. Nothing is hosted or downloaded. Mutation group bp (10 cases). Cargo.lock additions are the optional burn-onnx tree (generated). * feat(engine-burn): GENERATED SAM 2.1 tiny architecture (30 files) Output of tools/burn/generate_sam21.py, revision sam21-tiny-burn-0.22.1-onnx-0.22.0-v1, from the published encoder.onnx and decoder.onnx with burn-onnx 0.22.0 and its five asserted fixes. Do not review line by line; rerun the script with --check instead. Longest file 699 lines. Marked linguist-generated. * feat(engine-burn): SAM 2.1 runner for image.segment on Burn Sam21Runner (engine burn, model sam21) over the generated architecture. The request handling, picture kept between requests, empty-mask answer and click/box mapping are the shared Loaded from craft-ai-engine-rten, so only the model run lives here. Weights load through BurnpackStore and come back as errors; every call into the generated code is guarded against panics. Graphics card when one answers, else the processor. An .onnx file is answered with 'prepare it first', not a guess. 16 unit tests on a fake backend: click and box mapping, empty mask, short mask, a failing model, shared conformance run with Cancel, damaged and truncated files. * test(engine-burn): live comparison with rten, and the shared revision check The ignored live test runs the real SAM 2.1 files through rten (published ONNX) and Burn (weights prepared on this computer) on the astronaut sample: one click and a box, best-mask IoU, timings, and a truncated prepared file refused. The revision test fails when the generator, the compiled architecture and the saved-weights cache key disagree. * test+docs(engine-burn): mutation group be, weight-loading tests and the Burn engine guide Group be: 15 cases, one target each, all caught (saved-weights marker, load completeness, panic guard, answer length, best proposal, file names, declared card makers, the revision shared by script, architecture and cache key, the catalogue route and the .bpk binding rule). fully_applied is split out so the load checks can be tested without the store's own refusals. Group nf's onnx case follows the rewritten extension check. docs/BURN-ENGINE.md: how to regenerate, the five fixes, the revision bump rule, limits. * feat(engine-burn): CRAFT_AI_BURN_DEVICE=cpu forces the processor, and the processor path is measured * feat(engine-burn): prepare the saved copy when loading from the downloaded ONNX files The host step: given the downloaded encoder.onnx with decoder.onnx beside it, the runner calls craft-ai-burn-setup (exact source bytes checked, reused when already saved) and loads from the burn-prepared folder beside the model folder. Behind the default 'prepare' feature; mutation group be gains two cases (17/17); docs say what is measured. * feat(client): optional burn feature and a catalogue route for SAM 2.1 on Burn craft-ai-client gains the cargo feature burn (ProviderFactory::standard() then knows the engine). SAM 2.1's recipe lists a burn NativeRoute after rten, with the same two published ONNX files (one download for both, nothing converted is hosted). rten stays first and the default. The binding accepts the prepared .bpk file. Engines now list rten, burn, candle; the tests that assumed the Burn slot had no files, or that no built-in engine exists whatever cargo features are on, say what they mean. burn_engine.rs runs two apps through the client with the real prepared files (ignored). * fix(burn-setup): sweep the scratch folders a killed run left, and keep craft-ai-features a dev dependency CodeRabbit on #89: tempdir_in(cache) folders hold the two source copies (about 130 MB) and were removed only by Drop, so a killed app leaked them for good. They now carry the cache key and a role in their names, and after the key's lock is taken the same key's leftovers are removed (other keys' folders, the lock file and links are left alone). Red first: leftovers_of_a_killed_run_are_swept_for_this_key_only. Mutation cases burn-killed-run-leftovers-kept and burn-sweep-takes-other-keys, both caught. craft-ai-features is only used by a unit test, so it moves to dev-dependencies. * fix(engine-burn): review findings on #90 CodeRabbit: the generator now refuses an unknown option (a mistyped --chek used to select the run that rewrites src/model; red first: tools/burn/test_generate_sam21.py failed to import parse_args); the preparation live test needs the prepare feature and is ignored by default; the refusal test no longer leaves burn-prepared/<key>.lock in the shared system temp folder (the model folder is nested one level inside the test's own folder; confirmed the old layout leaked it on this PC); docs, STATUS and the mutation README no longer describe the client feature and catalogue route, which arrive in the next PR, as available. * docs(client): say the Burn route and client feature exist, and what the status line really covers CodeRabbit on #91: the STATUS row said the host step was not built. Given the downloaded encoder.onnx the runner prepares the saved copy itself on first load; the row now says that, names the limits (no progress or Cancel during that load, no app run through the shared binding, no live run on macOS or Linux) and LOCAL-BURN-PREPARATION.md no longer calls engine registration separate work. Also restores the client feature and catalogue route in the docs and rustdoc that the #90 review fix had marked as planned, and the three mutation-README claims for them. * fix(burn-setup): sweep only the scratch folders, not a renamed copy of the key's folder CodeRabbit on #89 (second round): sweep_stale removed every folder starting with <key>., so a damaged copy a person renamed to <key>.backup would be deleted on the next preparation. It now matches only <key>.work., <key>.conv. and <key>.out., the names the scratch folders are made with (one constant for both). Red first: the sweep test gained a <key>.backup folder that must survive. Mutation group bp 13/13 caught, including the new case burn-sweep-takes-a-renamed-copy. * test(burn): the generator's usage errors must exit with code 1 CodeRabbit on #90 (second round): the parse_args tests asserted only SystemExit, which a SystemExit(0) would also satisfy. They now assert the code. Checked against a mutant that exits 0: 6 failures. * fix(engine-burn): a damaged saved copy says it is damaged and where it lives CodeRabbit on #90 (third round) pointed out that a saved copy with a bad manifest or weights fails the same way on every retry, behind the message 'could not be read or written. Try again.' It proposed deleting the copy. The rule that a derived copy is refused, never repaired or deleted for the person, stays (#89 and its tests pin it); the dead end was the wording. craft-ai-burn-setup now answers every unusable saved copy with DAMAGED_COPY (remove it, prepare again, no 'try again'), and the Burn engine adds where it lives: the burn-prepared folder beside the model folder. Tests written first (they did not compile without DAMAGED_COPY and explain): a damaged copy of three kinds gets that answer and is left untouched; the engine explains it and passes other errors on. Mutation: bp 2 new cases and be 1 new case, all caught. * fix(burn-setup): a missing or unreadable saved weight is a damaged copy too CodeRabbit on #90 (fourth round): hash_file(...)? on a saved encoder.bpk or decoder.bpk that is gone returned the generic 'try again' error before the damaged-copy branch could run. It now maps to DAMAGED_COPY. Red first: the damaged-copy test gained 'missing encoder' and 'missing decoder' cases and failed with the old message. New mutation case burn-missing-saved-weight-answered-with-try-again; the saved-hash case was re-aimed at the new line. bp 16/16 caught. * fix(burn-setup): a manifest that cannot be opened or read is a damaged copy too CodeRabbit on #91 (round 5): File::open and read_to_end on an existing manifest returned the generic 'try again' error, so the engine gave no deletion guidance. Both now map to DAMAGED_COPY. Red first: a_manifest_that_cannot_be_opened_is_a_damaged_copy (Windows: another handle holds it exclusively; Unix: mode 0, ending quietly where permissions are not enforced) failed with the old message. The Unix variant was not run here. New mutation case burn-unopenable-manifest-answered-with-try-again; bp 17/17 caught (Windows). * test(burn-setup): write the Unix permission as octal (clippy::non_octal_unix_permissions, CI only) --------- Co-authored-by: scratch <s@s>
Craft AI integrations
One AI connection for every Craft app. A person sets up a tool and its model once, in the launcher or natively in any app, and every app that has that tool uses it with no further setup. The shared crates own connections, models, the tool to model binding, the model picker, consent and jobs. Each app keeps its own interface for each tool. Where the suggested models or providers do not suit someone, they can add their own connection, model or tool route and plug it in.
This workspace is the runtime half of the Craft AI layer: provider adapters, secret stores, the app-facing
client, local runtime setup and reusable egui widgets. The pure contract they all speak
(craft-ai-core: connections, capabilities, requests, jobs, consent, the shared store format) has
landed in storytold/craft-libs. This workspace still uses a local copy with newer consent-term and
model-folder additions. It carries the upstream safety fixes, but cannot switch to a pinned upstream
revision until the required additions are reviewed there. See docs/CORE-SYNC.md
for that boundary and the dependency-switch gates. The extension APIs are still evolving.
Layout and rules mirror craft-libs: edition 2024, MIT OR Apache-2.0, no unsafe, and every
lib.rs denies unwrap, expect and panic in shipped code.
Crates
| Crate | What it is |
|---|---|
craft-ai-core |
The data model and provider trait. Serde only, no I/O, no async runtime, compiles for wasm. |
craft-ai-native |
Desktop only: secret stores (OS keychain, passphrase-encrypted file, environment), the shared store file with a locked atomic writer, consent and history files. |
craft-ai-client |
What an app holds: AiClient (status, plan, consent, start, Job), ToolDesk (which model does each tool), ConnectionManager, and the ProviderFactory that adapters register into. |
craft-ai-features |
Wasm-safe catalogue of AI tools, where each would appear, how each can run on a machine (model files with size, SHA-256, source and licence) and what a machine still needs. |
craft-ai-http |
The one TLS setting every hosted call and download uses: the platform's certificate check. |
craft-ai-media |
Durable local copies of hosted media links, with the checks that needs. |
craft-ai-openai |
OpenAI-compatible text (streaming, tools, structured output) and images (generate, edit, inpaint). |
craft-ai-bfl, craft-ai-vertex, craft-ai-seedance, craft-ai-artcraft |
Black Forest Labs FLUX, Vertex express-mode Gemini images, BytePlus ModelArk Seedance video, and the ArtCraft Omni API-key route. |
craft-ai-declarative |
A provider described by a small data file instead of Rust. Design: docs/design/declarative-adapter.md. |
craft-ai-comfyui, craft-ai-sam3 |
Local ComfyUI workflows, including imported subject-selection graphs (image.segment) whose masks are checked before an app sees them; the SAM 3 point and session clients; the older direct SAM 3 detector bridge. |
craft-ai-engine, craft-ai-engine-candle, craft-ai-engine-rten, craft-ai-engine-burn |
Built-in model engines: the runner contract and the one provider that runs any registered model on this computer; SAM 3 on Candle (Apache-2.0 only; see its NOTICE); SAM 2.1 on rten, a pure Rust engine that reads the publisher's .onnx files as they are, on the processor (the default); and SAM 2.1 on Burn, an optional graphics-card engine that reads weights prepared once on this computer (craft-ai-burn-setup). Guides: docs/ADD-AN-ENGINE.md, docs/BURN-ENGINE.md. |
craft-ai-comfy-env |
Finds an existing ComfyUI Desktop install, inventories its model files and node packs, downloads missing model files into it. |
craft-ai-egui |
Theme-neutral egui widgets: status, model picker, consent prompt, job progress, connection panel, history. Needs Rust 1.95. |
Building on this layer (add a service, a tool, or use one from an app): docs/ADOPT-A-TOOL.md.
Which crate owns what, and what stays in each app: docs/LAYOUT.md. The shared-tool contract and its limits: docs/PLATFORM-CONTRACT.md. Custom connections, model sources and node packs: docs/custom-connections-and-sources.md.
Building and testing
Builds are heavy, so keep them targeted and run one cargo process at a time:
cargo test -p craft-ai-client -j 8
cargo clippy -p craft-ai-client --all-targets --locked -j 8 -- -D warnings
cargo check -p craft-ai-core --target wasm32-unknown-unknown --locked
cargo fmt --all --check
CI (.github/workflows/ci.yml) runs fmt, clippy on the whole workspace, the wasm check, the tests on
Linux and Windows, and cargo-deny. The test that touches the real OS keychain is #[ignore]d:
cargo test -p craft-ai-native -- --ignored keychain_round_trip. Deliberate-bug checks live in
tools/mutation.
Examples: cargo run -p craft-ai-client --example custom_tool, and
cargo run -p craft-ai-egui --example picker_demo (sample data only).
What has and has not been verified
Read docs/STATUS.md before relying on a provider or tool. In short:
- Tested against scripted servers on localhost: every hosted adapter, the media copier, the generic ComfyUI adapter, the declarative adapter and the client flow.
- Run for real on one machine: the SAM 3 point and session clients through a headless ComfyUI; SAM 2.1 on rten with the published ONNX files; and an imported selection graph on ComfyUI 0.22.3. All on Windows, with one sample picture.
- Not verified: any real hosted service, TLS behind an intercepting proxy, macOS and Linux keychains, macOS builds of the test suite, and any app running a tool through the shared binding.
Licence
MIT or Apache-2.0, at your option (LICENSE-MIT, LICENSE-APACHE).