mirror of
https://github.com/storytold/craft-ai-integrations.git
synced 2026-11-02 18:06:49 +00:00
Jeremy
8be4066b1f
* feat(ai-engine): a runner contract for models that run on this computer
Candle and Burn were named in the catalogue but no app could share them: each would have needed
its own job handling, cancel, progress and weights loading. This adds the seam so an engine author
writes only the model.
- craft-ai-engine: ModelRunner (engine, model spec, load) and LoadedModel (run), a RunContext for
progress, cancel and notices, an EngineRegistry, and EngineProvider, the one Provider that runs
any registered model (weights loaded once, jobs for one model take turns, a panicking runner
becomes a failed job).
- testing::{FakeRunner, check_runner}: puts a runner behind the real provider and runs the shared
conformance suite.
- craft-ai-client: ProviderFactory::register_engines is the whole of enabling an engine in an app;
register now forgets what was declared about the adapter it replaces, and unregister removes one.
- docs/ADD-AN-ENGINE.md is the crate documentation, so its code is compiled by the doctests.
* feat(ai-engine-candle): SAM 3 on the Candle engine, and the catalogue names the files it reads
The built-in SAM 3 route was documented as reading the same file as ComfyUI. It does not: the
ComfyUI file is Meta's layout (detector.backbone..., 1465 tensors); the Candle port reads the
Hugging Face layout (detector_model..., 1797 tensors) plus vocab.json and merges.txt. A route
pointed at the ComfyUI file could never load.
- ai-features: NativeRoute gains its own `models`, so a runtime can name the files it reads. The
SAM 3 routes name facebook/sam3 at a pinned revision, with sizes and SHA-256 checked against a
real download. NativeRoute::weights and total_bytes read them.
- craft-ai-engine-candle: the model code from LightCraft's lightcraft-segment crate (see NOTICE),
behind the runner contract. image.segment takes a PNG source and either a text prompt or click
points, and answers with a grayscale PNG mask the size of the source. Apache-2.0 only, because
the port is a derivative of Hugging Face Transformers.
- craft-ai-client: the `candle` cargo feature makes ProviderFactory::standard() know the engine, so
an app adopts it by turning the feature on.
* fix(ai-engine,ai-client): keep one provider per connection, and let a cancelled job leave the queue without loading or running
The client asks the factory for a provider on every job, so each job built a fresh EngineProvider and
loaded the weights again; two jobs at once loaded two copies. The factory now keeps one provider per
connection and builds another only when the connection's models or weight files change.
A job waiting behind another for the same model reported Running, and if it was cancelled it still
loaded the weights and ran. It now reports Queued until it holds the model, and a cancel before then
or during a slow load stops it before any work. At most 16 jobs may wait or run at once; a burst is
refused in plain words rather than piling up threads that each hold a copy of the picture.
* fix(ai-client): write the kept-provider key with escapes, not a literal newline and tab
* fix(ai-features,deny): say what the Candle route really runs on, offer no download for Burn, and record the candle advisories
The Candle route said it ran on Nvidia or Apple graphics, but this build declares the Apple card only and
runs on the processor elsewhere; the label now says so. The Burn route named the 3.4 GB Hugging Face files
although no Burn code exists, so it now names none and offers no download.
cargo-deny failed on candle-core's zip (RUSTSEC-2026-0367, 0368, 0369) and paste (RUSTSEC-2024-0436).
Each is ignored with the reason it cannot be reached here: only safetensors are read and no archive is
extracted.
* feat(ai-engine-candle): read JPEG pictures, and turn them upright by their EXIF orientation
A source can now be a PNG or a JPEG, told apart by its first bytes. JPEGs are decoded with zune-jpeg through
the image crate (baseline, progressive, greyscale), refused from the header when over 100 megapixels, and
given an allocation cap. The EXIF orientation is applied, as browsers, photo editors and OpenCV do, so a
phone photo taken sideways is segmented upright and a click normalised to the picture lands where the person
clicked. The mask comes back the size of the upright picture. A PNG is read as stored.
Verified on Meta's checkpoint in release mode: a 128x192 JPEG marked 'turn 90 degrees clockwise' gives a
192x128 mask centred on the disc. Unit tests cover colour and greyscale, every damaged and cut-off case,
a header patched to 30000x30000, and the turn itself (the test fails without it).
jpeg-encoder makes the test JPEGs; it carries the IJG licence, allowed for that one crate in deny.toml.
* test(ai-engine-candle): write the EXIF header with escapes, not raw NUL bytes, so git shows the file as text
* fix(ai-engine-candle): lift the PNG decoder's buffer limit, and bound tokenizer work for long prompts
A one-row picture inside the 100 megapixel limit but with a row over 64 MiB was refused by the png crate's default allowance; the decoder now gets the same 900 MiB cap as the JPEG route. Tokenizer::encode now clips the text to CONTEXT * 32 characters and each word to CONTEXT * 8 before the merge loop, and stops once the ids fill the context. Both found by CodeRabbit on #41.
* docs(ai-engine-candle): doc comments on the functions touched by the review fixes
* fix(ai-engine-candle): collapse whitespace before the tokenizer's character limit
Spaces at the start of a prompt used up the 1,024 character allowance, so a prompt of 1,024 spaces then a word reached the model as nothing. Whitespace is now collapsed first, and the text is copied only as far as the allowance reaches. Found by CodeRabbit on #41.
---------
Co-authored-by: scratch <s@s>
Co-authored-by: t <t@t>
Craft AI integrations
One AI connection for every Craft app. A person sets up a tool and its model once, in the launcher or natively in any app, and every app that has that tool uses it with no further setup. The shared crates own connections, models, the tool to model binding, the model picker, consent and jobs. Each app keeps its own interface for each tool. Where the suggested models or providers do not suit someone, they can add their own connection, model or tool route and plug it in.
This workspace is the runtime half of the Craft AI layer: provider adapters, secret stores, the app-facing
client, local runtime setup and reusable egui widgets. The pure contract they all speak
(craft-ai-core: connections, capabilities, requests, jobs, consent, the shared store format) has
landed in storytold/craft-libs. This workspace still uses a local copy with newer consent-term and
model-folder additions. It carries the upstream safety fixes, but cannot switch to a pinned upstream
revision until the required additions are reviewed there. See docs/CORE-SYNC.md
for that boundary and the dependency-switch gates. The extension APIs are still evolving.
Layout and rules mirror craft-libs: edition 2024, MIT OR Apache-2.0, no unsafe, and every
lib.rs denies unwrap, expect and panic in shipped code.
Crates
| Crate | What it is |
|---|---|
craft-ai-core |
The data model and provider trait. Serde only, no I/O, no async runtime, compiles for wasm. |
craft-ai-native |
Desktop only: secret stores (OS keychain, passphrase-encrypted file, environment), the shared store file with a locked atomic writer, consent and history files. |
craft-ai-client |
What an app holds: AiClient (status, plan, consent, start, Job), ToolDesk (which model does each tool), ConnectionManager, and the ProviderFactory that adapters register into. |
craft-ai-features |
Wasm-safe catalogue of AI tools, where each would appear, how each can run on a machine (model files with size, SHA-256, source and licence) and what a machine still needs. |
craft-ai-http |
The one TLS setting every hosted call and download uses: the platform's certificate check. |
craft-ai-media |
Durable local copies of hosted media links, with the checks that needs. |
craft-ai-openai |
OpenAI-compatible text (streaming, tools, structured output) and images (generate, edit, inpaint). |
craft-ai-bfl, craft-ai-vertex, craft-ai-seedance, craft-ai-artcraft |
Black Forest Labs FLUX, Vertex express-mode Gemini images, BytePlus ModelArk Seedance video, and the ArtCraft Omni API-key route. |
craft-ai-declarative |
A provider described by a small data file instead of Rust. Design: docs/design/declarative-adapter.md. |
craft-ai-comfyui, craft-ai-sam3 |
Local ComfyUI workflows with the SAM 3 point and session clients; the older direct SAM 3 detector bridge. |
craft-ai-engine, craft-ai-engine-candle |
Built-in model engines: the runner contract and the one provider that runs any registered model on this computer, and SAM 3 on Candle (Apache-2.0 only; see its NOTICE). Guide: docs/ADD-AN-ENGINE.md. |
craft-ai-comfy-env |
Finds an existing ComfyUI Desktop install, inventories its model files and node packs, downloads missing model files into it. |
craft-ai-egui |
Theme-neutral egui widgets: status, model picker, consent prompt, job progress, connection panel, history. Needs Rust 1.95. |
Building on this layer (add a service, a tool, or use one from an app): docs/ADOPT-A-TOOL.md.
Which crate owns what, and what stays in each app: docs/LAYOUT.md. The shared-tool contract and its limits: docs/PLATFORM-CONTRACT.md. Custom connections, model sources and node packs: docs/custom-connections-and-sources.md.
Building and testing
Builds are heavy, so keep them targeted and run one cargo process at a time:
cargo test -p craft-ai-client -j 8
cargo clippy -p craft-ai-client --all-targets --locked -j 8 -- -D warnings
cargo check -p craft-ai-core --target wasm32-unknown-unknown --locked
cargo fmt --all --check
CI (.github/workflows/ci.yml) runs fmt, clippy on the whole workspace, the wasm check, the tests on
Linux and Windows, and cargo-deny. The test that touches the real OS keychain is #[ignore]d:
cargo test -p craft-ai-native -- --ignored keychain_round_trip. Deliberate-bug checks live in
tools/mutation.
Examples: cargo run -p craft-ai-client --example custom_tool, and
cargo run -p craft-ai-egui --example picker_demo (sample data only).
What has and has not been verified
Read docs/STATUS.md before relying on a provider or tool. In short:
- Tested against scripted servers on localhost: every hosted adapter, the media copier, the generic ComfyUI adapter, the declarative adapter and the client flow.
- Run for real on one machine: the SAM 3 point and session clients through a headless ComfyUI.
- Not verified: any real hosted service, TLS behind an intercepting proxy, macOS and Linux keychains, macOS builds of the test suite, and any app running a tool through the shared binding.
Licence
MIT or Apache-2.0, at your option (LICENSE-MIT, LICENSE-APACHE).