Jeremy
611195b50e
* feat(ai-comfyui): import a selection workflow and run it with the picture sent first Adds the `image.segment` capability to imported ComfyUI workflows. A selection workflow maps a text prompt and one source picture; the picture is uploaded, the graph is queued with the name ComfyUI gave it, and every queued graph gets a sink name of its own so a repeat is not answered from cache with no output. Clicks or a box are refused in plain words before anything is sent, and a stored workflow file may declare a source binding and no mask. The answer is the graph's image as it comes. Checking the mask is the next change. * feat(ai-comfyui): check a selection workflow's mask and answer in the shared shape A selection workflow's image is now a mask the shared contract can trust. It must be the size of the picture sent (a JPEG may come back turned a quarter, because ComfyUI's loader applies the EXIF turn), is reduced to one 8-bit channel, and is followed by the shared `Found` result. A mask with no pixel above half is an empty mask and `matched: false`, never a failed job. Picture sizes are read from the PNG or JPEG header and bounded. This is not calibrated absence detection: the example Florence graph finds a box for any phrase. * feat(ai-comfyui): send a selection picture once and resend it only when it is gone A repeat selection on the same picture reuses the file already in ComfyUI's scratch area, keyed by a hash of the bytes. Only the last four pictures are remembered. If ComfyUI has cleared its scratch area (a restart), the first refusal of a reused name drops the entry and sends the picture again once; a refusal of a fresh upload is never retried. A failed selection job clears the cache, since the file may be what went wrong. Adds the ignored live test against a real ComfyUI. * test(ai-comfyui): count refused prompts without the deprecated fetch_update Newer Rust renames AtomicUsize::fetch_update to try_update and warns, which the strict Clippy job turns into an error on CI. The scripted server handles one job at a time here, so a load and a fetch_sub do the same job and build on both toolchains. * fix(ai-comfyui): hand on a binary mask, white where selected and black elsewhere A review finding: the first channel was kept as the graph saved it, so pixels from 128 to 254 reached the app as partly selected, while the shared contract says selected pixels are white. Every pixel above half is now 255 and the rest 0. Written first and seen to fail (the answer kept [0, 100, 128, 200, 255]); mutation cases added for a grey kept as selected and a low grey kept. * docs(ai-comfyui): state the selection mask's byte limit, test it, and rename a mutation case Two review findings. The mask is read inline under the provider's output limit (never more than the 32 MiB inline cap) and never to the output folder, because the checks need its bytes in hand; that is now written in the module header and the provider, and a test shows a mask over a lowered limit fails the job and hands on nothing, with a mutation case for the limit being ignored. The case that only removed the media-type guard is renamed so it no longer implies JPEG answers are covered. The limit test passed on first run, as the behaviour already existed: it is a guard, not a red-first fix. * fix(ai-comfyui): forget the cached pictures when any selection job fails A review finding: only an error status from ComfyUI cleared the upload cache, so a job whose mask was fetched but refused (wrong size, not a PNG) left its picture cached and the next selection could reuse a stale scratch name. The cache is now cleared once, after the job's state is settled, whenever a selection job ends failed. Written first and seen to fail (a wrong-size mask left one upload for two selections); the existing mutation case is re-aimed at the single clear.
Craft AI integrations
One AI connection for every Craft app. A person sets up a tool and its model once, in the launcher or natively in any app, and every app that has that tool uses it with no further setup. The shared crates own connections, models, the tool to model binding, the model picker, consent and jobs. Each app keeps its own interface for each tool. Where the suggested models or providers do not suit someone, they can add their own connection, model or tool route and plug it in.
This workspace is the runtime half of the Craft AI layer: provider adapters, secret stores, the app-facing
client, local runtime setup and reusable egui widgets. The pure contract they all speak
(craft-ai-core: connections, capabilities, requests, jobs, consent, the shared store format) has
landed in storytold/craft-libs. This workspace still uses a local copy with newer consent-term and
model-folder additions. It carries the upstream safety fixes, but cannot switch to a pinned upstream
revision until the required additions are reviewed there. See docs/CORE-SYNC.md
for that boundary and the dependency-switch gates. The extension APIs are still evolving.
Layout and rules mirror craft-libs: edition 2024, MIT OR Apache-2.0, no unsafe, and every
lib.rs denies unwrap, expect and panic in shipped code.
Crates
| Crate | What it is |
|---|---|
craft-ai-core |
The data model and provider trait. Serde only, no I/O, no async runtime, compiles for wasm. |
craft-ai-native |
Desktop only: secret stores (OS keychain, passphrase-encrypted file, environment), the shared store file with a locked atomic writer, consent and history files. |
craft-ai-client |
What an app holds: AiClient (status, plan, consent, start, Job), ToolDesk (which model does each tool), ConnectionManager, and the ProviderFactory that adapters register into. |
craft-ai-features |
Wasm-safe catalogue of AI tools, where each would appear, how each can run on a machine (model files with size, SHA-256, source and licence) and what a machine still needs. |
craft-ai-http |
The one TLS setting every hosted call and download uses: the platform's certificate check. |
craft-ai-media |
Durable local copies of hosted media links, with the checks that needs. |
craft-ai-openai |
OpenAI-compatible text (streaming, tools, structured output) and images (generate, edit, inpaint). |
craft-ai-bfl, craft-ai-vertex, craft-ai-seedance, craft-ai-artcraft |
Black Forest Labs FLUX, Vertex express-mode Gemini images, BytePlus ModelArk Seedance video, and the ArtCraft Omni API-key route. |
craft-ai-declarative |
A provider described by a small data file instead of Rust. Design: docs/design/declarative-adapter.md. |
craft-ai-comfyui, craft-ai-sam3 |
Local ComfyUI workflows with the SAM 3 point and session clients; the older direct SAM 3 detector bridge. |
craft-ai-engine, craft-ai-engine-candle |
Built-in model engines: the runner contract and the one provider that runs any registered model on this computer, and SAM 3 on Candle (Apache-2.0 only; see its NOTICE). Guide: docs/ADD-AN-ENGINE.md. |
craft-ai-comfy-env |
Finds an existing ComfyUI Desktop install, inventories its model files and node packs, downloads missing model files into it. |
craft-ai-egui |
Theme-neutral egui widgets: status, model picker, consent prompt, job progress, connection panel, history. Needs Rust 1.95. |
Building on this layer (add a service, a tool, or use one from an app): docs/ADOPT-A-TOOL.md.
Which crate owns what, and what stays in each app: docs/LAYOUT.md. The shared-tool contract and its limits: docs/PLATFORM-CONTRACT.md. Custom connections, model sources and node packs: docs/custom-connections-and-sources.md.
Building and testing
Builds are heavy, so keep them targeted and run one cargo process at a time:
cargo test -p craft-ai-client -j 8
cargo clippy -p craft-ai-client --all-targets --locked -j 8 -- -D warnings
cargo check -p craft-ai-core --target wasm32-unknown-unknown --locked
cargo fmt --all --check
CI (.github/workflows/ci.yml) runs fmt, clippy on the whole workspace, the wasm check, the tests on
Linux and Windows, and cargo-deny. The test that touches the real OS keychain is #[ignore]d:
cargo test -p craft-ai-native -- --ignored keychain_round_trip. Deliberate-bug checks live in
tools/mutation.
Examples: cargo run -p craft-ai-client --example custom_tool, and
cargo run -p craft-ai-egui --example picker_demo (sample data only).
What has and has not been verified
Read docs/STATUS.md before relying on a provider or tool. In short:
- Tested against scripted servers on localhost: every hosted adapter, the media copier, the generic ComfyUI adapter, the declarative adapter and the client flow.
- Run for real on one machine: the SAM 3 point and session clients through a headless ComfyUI.
- Not verified: any real hosted service, TLS behind an intercepting proxy, macOS and Linux keychains, macOS builds of the test suite, and any app running a tool through the shared binding.
Licence
MIT or Apache-2.0, at your option (LICENSE-MIT, LICENSE-APACHE).