mirror of
https://github.com/storytold/craft-ai-integrations.git
synced 2026-11-02 18:06:49 +00:00
imperator
e8f320321a
* feat(ai-engine-burn): InSPyReNet for image.matte (Remove background) A second model on the Burn engine: `inspyrenet` answers `image.matte` from the picture alone, a grayscale PNG at the picture's size (white is the subject, soft along hair and fur). `remove-background` has had no local model; this is one. - Network: InSPyReNet with the Swin-B backbone, written by hand in Burn after the MIT reference code. Module names follow the PyTorch keys, so the published .safetensors loads by name through Burn's PyTorch adapter. A load that leaves a weight unset, fails one, or finds a foreign tensor is refused in plain words. - Any picture size: full resolution up to 1280 px on the long side; above that a reflect-padded square reduced with antialiased Lanczos-3, the matte enlarged back and refined at full resolution with a colour guided filter on the tiles near the edge (the Gyre approach), then a 0.2 fringe shrink. - Weights: 1038lab/inspyrenet inspyrenet.safetensors at a pinned revision, checked tensor for tensor against the author's ckpt_base.pth. Checked on one Windows machine (release): every stage against transparent-background's PyTorch code (worst relative difference 3.2e-5 on the processor, 2.4e-4 on the card, no pixel changing side); the shared conformance suite; on a real PNG the mask is byte-identical to the same port in PhotoCraft. Four new deliberate-bug cases in `be`, all caught. Not done: a catalogue recipe (a recipe names a ComfyUI workflow today), clean timings, macOS and Linux runs, a run through an app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ai-features): a recipe that is built in only, InSPyReNet for Remove background `LocalRecipe::comfyui` (default true) marks a recipe ComfyUI runs. With false it is only its built-in routes: never a ComfyUI setup option, plan, install offer or connection, and engine advice does not list ComfyUI for a tool without a ComfyUI recipe. `Feature::comfyui_recipes()` gives those recipes; `engine_recipe`, `plan_native`, `setup_offer_native` and `enable_native` are unchanged. `remove-background` gets one such recipe: InSPyReNet on Burn, the pinned `inspyrenet.safetensors` (1038lab/inspyrenet @ 0f594663, 367,120,444 bytes) in the folder `inspyrenet`, so Set up AI offers one checked download and no ComfyUI. Tests: the catalogue, setup options, plans and offers (`ai-features/tests/built_in_only.rs`), registration and advice (`ai-client/tests/built_in_only.rs`), and through `AiClient` on Burn (`ai-client/tests/burn_matte.rs`, the real run ignored). Mutation group `bo`: 8 of 8 caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(tools): retarget two `lc` mutation cases at the ComfyUI recipe lookups `unrunnable-recipe-accepted` and `second-recipe-takes-plain-id` searched for the `feature.locals` lookups in `ai-client/src/local.rs`, which now go through `Feature::comfyui_recipes()`. Same mutations, new targets; group `lc`: 21 caught, one known equivalent. Every case in `GROUPS` again has exactly one target (640 checked). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ai-engine-burn): InSPyReNet runner checks around a stand-in, readout without panic CI has no graphics card, only a software renderer (lavapipe, WARP), and the engine takes it. There the tiny network was too slow for the conformance suite's poll limit, and on lavapipe the readout failed. Unoptimised, Burn's processor backend takes minutes for the same network. - The loaded model runs the pipeline around a `Predict`: the network, or in tests a stand-in, as the SAM 2.1 tests here use a fake backend. The conformance suite and the mask test use the stand-in (debug: the crate's unit tests in about 18 s). - The tiny network still runs through load and run on the processor in an ignored release test (about 4 s). The runner's device choice is a field, so tests can name the processor. - The prediction is read with `try_into_data`, so a device that fails while running is an error, not a guarded panic. - docs/BURN-ENGINE.md: the test split and that a software renderer counts as a card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ai-engine-burn): InSPyReNet moves to the processor when the card fails A card can answer the engine's probe and still fail on the network: on CI a software renderer (lavapipe) failed while reading the answer back. The first failure on a card that is not a cancel now builds the network again on the processor from the same file, runs that picture again there with a notice, and stays on the processor (`inspyrenet/fallback.rs`). A network loaded on the processor in the first place has nothing to move to. Tests with stand-ins: a working card is kept; a failing one moves once and stays; a cancel is not a failure; a rebuild that fails is the answer. Mutation group `be`: three new cases, all seven InSPyReNet cases caught. A real photo on the card still runs on the card. docs/BURN-ENGINE.md: the move, and the client run's picture is 2848 x 4272, not 3101 x 1613. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(ai-engine-burn): the move to the processor stays only after a rebuild that worked The rebuild is tried once. If it fails, its error is the answer and later pictures go to the card again, as `a_rebuild_that_fails_is_the_answer` already pins. The module docs and docs/BURN-ENGINE.md said the model stays on the processor without that condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ai-engine-burn): retry a failed InSPyReNet load on the card on the processor The move to the processor covered failures while predicting, not a load on the card that fails. `load_or_processor` now retries such a load on the processor, unless the file itself is not the checkpoint (`Unsupported`), which is refused at once rather than read a second time. A network loaded on the processor in the first place is loaded once. Tests: a card that loads keeps the network; one that fails loads on the processor; a bad file is not read twice; on the processor there is one load. Mutation group `be`: two new cases, all nine InSPyReNet cases caught; every case in the checker has one target (645). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(ai-engine-burn): method references in the guided filter, one name for the fallback path Three SonarCloud findings on #112: two closures in `guided.rs` become method references, and the mutation cases name `inspyrenet/fallback.rs` once. No change in behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: imperator <imperator.maximus@gmail.com>
63 lines
11 KiB
Markdown
63 lines
11 KiB
Markdown
# Status: what runs, what is tested, what is open
|
|
|
|
Three levels, kept apart on purpose. A saved key, a static catalogue, a passing scripted test and a live
|
|
job prove different things.
|
|
|
|
## Built and tested against scripted servers only
|
|
|
|
No real hosted service, download host or repository was contacted by these tests.
|
|
|
|
| Area | Crate | Notes |
|
|
|---|---|---|
|
|
| OpenAI-compatible text and images | `craft-ai-openai` | Streaming, tools, structured output; image generate, edit and inpaint. A streamed reply is bounded three ways: per line (1 MiB), total bytes kept (the same as a whole reply, 128 MiB), and events held for one poll (65,536; a stream that passes it without a poll fails the job and the connection is dropped). |
|
|
| FLUX image and video | `craft-ai-bfl` | |
|
|
| Seedance video through BytePlus ModelArk | `craft-ai-seedance` | The Volcengine China address is refused. |
|
|
| Gemini images on Vertex express mode | `craft-ai-vertex` | An express-mode key is not a full-project Vertex credential and has no free key probe. |
|
|
| ArtCraft Omni API key | `craft-ai-artcraft` | Prompt-only image and video. Account or subscription access has no adapter: no official programmatic route is documented. An API key is not evidence that a subscription covers the requests. |
|
|
| Hosted media copies | `craft-ai-media` | https only (loopback only for a connection marked local), every redirect re-checked, private addresses refused, size bounded, file type taken from the bytes. A failed copy keeps the original link with a notice. A busy host is retried with a bounded, doubling pause. Not built: retries for provider API calls, expiry tracking, resume, and an input-side upload cache. |
|
|
| Generic ComfyUI workflows | `craft-ai-comfyui` | Maps a prompt field; `image.inpaint` needs mapped `LoadImage` and `LoadImageMask` fields. A workflow is offered as a model only after `/object_info` confirms its nodes and checkpoint names; submission repeats the check. Output stays inline (32 MiB) unless the app gives a directory (1 GiB, local `file:` handle, app owns cleanup). Targeted cancel is off by default because an older server may treat it as a global interrupt. Imported image.segment workflows map a description and source to a checked, source-sized grayscale mask plus the shared result. Empty masks mean no match; source uploads are remembered for selection jobs. This is not calibrated absence detection: the example Florence graph can find a box for an absent subject. No input file handles or general parameter mapping yet. |
|
|
| Built-in engines | `craft-ai-engine`, `craft-ai-engine-candle`, `craft-ai-engine-rten`, `craft-ai-engine-burn` | Contract, registry and provider are tested with a fake runner and the shared conformance suite. SAM 3 on Candle (point and text prompts, processor, or Metal on macOS) ran on a real photograph and passes the same suite on Meta's checkpoint; the live tests skip without `CRAFT_SAM3_WEIGHTS`. It reads the Hugging Face checkpoint layout with tokenizer files, not the ComfyUI file. Pictures go in as PNG or JPEG (a JPEG's EXIF turn is applied). No Nvidia (CUDA) build, and the launcher does not download the checkpoint yet. SAM 2.1 on rten (processor only) reads the publisher's two `.onnx` files, pinned by size and SHA-256, with click and box prompts; a description is refused in plain words. Pictures are bounded by side and pixel count. With the real files in release on one Windows machine: a mask at the picture's size, and a hover through the ordinary job path with a median of about 35 ms after a first look of about 0.7 s (one picture, not a benchmark). SAM 2.1 is the first recipe for Select subject, so the plain built-in offer is SAM 2.1 on rten. No macOS or Linux run, and no app has run it through the shared binding yet. SAM 2.1 on Burn (`craft-ai-engine-burn`, cargo feature `burn`, optional, after rten in the recipe) reads weights that `craft-ai-burn-setup` prepares once on the computer; on one Windows machine with a graphics card its mask matched rten's (best-mask IoU 1.0, one click and one box, one picture, not a benchmark). Given the downloaded `encoder.onnx`, the runner prepares the saved copy itself on first load (in `burn-prepared/` beside the model folder; no progress and no Cancel during that load). No app has run it through the shared binding yet, and there has been no live run on macOS or Linux. InSPyReNet on Burn (`inspyrenet`, same crate and feature) answers `image.matte` from the picture alone with a grayscale PNG at the picture's size, at any picture size. Its hand-written network reads the published `.safetensors` checkpoint (checked tensor for tensor against the author's `ckpt_base.pth`); on one Windows machine every stage matched transparent-background's PyTorch code (worst relative difference 3.2e-5 on the processor, 2.4e-4 on the card, no pixel changing side), it passes the shared conformance suite, and on a real PNG its mask was byte-identical to the same port inside PhotoCraft. Time per picture was not measured cleanly (the card was shared during the runs). `remove-background` lists it as its one recipe, built in only (`LocalRecipe::comfyui` false): Set up AI offers the Burn route and its pinned file, never a ComfyUI model, and scripted tests cover the offer, the plan, the registration and the advice, and an ignored test ran the real file from that route through `AiClient` for two app ids; the file has not been downloaded through Set up AI, and no real app has mounted it. See `docs/BURN-ENGINE.md`. |
|
|
| First-run setup flow | `craft-ai-client`, `craft-ai-egui`, `craft-ai-features` | One start button on `SetupSession` (checks complete files on disk, otherwise downloads; `blocker` says why it does nothing), a licence tick only where a file's licence needs one, `FolderDesk::ensure_default` for the suggested folder, `ToolDesk::adopt` for checked files only, a try-it sample that clicks for built-in selection models and fails a selection that chose nothing, status and picker that agree when a model's file has gone, and engine advice that does not call a measured-quick built-in engine (rten, one machine) slow. Scripted tests and a fake provider only: no real download, no model run through it, no app has mounted it, no macOS or Linux run. |
|
|
| Shared egui widgets | `craft-ai-egui` | Source and tests migrated to egui 0.37 with the reviewed storytold egui family at `6d518fc799681571963842f95b55a6b68431b15a` and winit at `a517b3846ee8457d84ee4975110bfda6797c44dc`. The current headless candidate passed 126 tests with 0 failures and 4 ignored doctests under managed job `egui037-ai-final-ci-20261025-1331`, with formatting that left the worktree unchanged and strict all-target clippy for `craft-ai-egui` and `craft-ai-engine` on Linux. Five deterministic kittest `wgpu`/snapshot goldens are committed and passed in that run. The historical 108-test checkpoint at `egui037-ai-regression-mutation-20261024-2205` also recorded a pure-background RGB(27,27,27) to RGB(30,30,30) change failing the expected snapshot with 75,486 changed pixels, and the two compiled connection-panel lifecycle mutants caught before the source and golden hashes were restored. A locked consumer-smoke test build and its one draw test passed with one `craft-ai-core` identity and the pinned egui family; that rendererless root does not use the winit patch. No app imports the widgets. Native X11/Wayland/Windows behavior, IME/clipboard/focus, physical GPU qualification and performance measurements remain open. Full remote workspace/platform CI also remains open. |
|
|
| Declarative adapter | `craft-ai-declarative` | Request/response and queue flows from a data file; prompt-only inputs. No launcher screen to add a spec yet. |
|
|
| Custom connections, model sources, node packs | `craft-ai-core`, `craft-ai-comfy-env` | See `docs/custom-connections-and-sources.md`. Own-host tokens only, no key sent across a redirect, pickle formats need consent, unverified checksums are flagged. |
|
|
| Model folders | `craft-ai-core`, `craft-ai-comfy-env`, `craft-ai-client` | A person's own folders of model files, kept as a map (which sub-folders hold which kind) in the shared store, never moved, renamed or deleted. Scanned read only with limits (depth, files, entries, header size), one default for downloads, folders linked from ComfyUI are read only, and the map can be written as a ComfyUI extra-paths file. Not wired yet: downloads and the installed-files scan still use ComfyUI's own folders, and no launcher screen is merged. |
|
|
| Consent, history, shared store, key handling | `craft-ai-core`, `craft-ai-native`, `craft-ai-client` | Consent is scoped to app, connection, destination and capability, per kind of data, and may expire. `craft-ai-egui` offers the original prompt and a timed prompt through `consent_prompt_with_terms`; no app has mounted it yet. |
|
|
|
|
## Run for real
|
|
|
|
Only the local SAM 3 clients (`Sam3Point`, `Sam3Session`): point, box, text, candidate masks, grounding
|
|
instances and prepared hover sessions, through a headless ComfyUI on one Windows machine. The live-test
|
|
evidence is not kept in this repository. This does not make the generic ComfyUI image or video workflows
|
|
live-tested.
|
|
|
|
## Not verified
|
|
|
|
- Any real hosted provider, in any adapter.
|
|
- TLS against an intercepting proxy or antivirus HTTPS scanner. `craft-ai-http` trusts the platform's
|
|
certificate store, so such software can read traffic, API keys included, and there is no switch back
|
|
to bundled roots.
|
|
- macOS (CI runs the tests on Linux and Windows), and the macOS and Linux keychains. The real keychain
|
|
test is `#[ignore]`d.
|
|
- Model download from Hugging Face and automatic node-pack installation against the real services.
|
|
- The built-in recommended-models catalogue (`catalog.json`) is empty. Which models to recommend per tool
|
|
is a curation decision, so "Recommended" never appears until someone makes it.
|
|
- App adoption. PhotoCraft's dev build uses the SAM 3 clients directly, not the shared role, connection
|
|
and consent flow. A route-resolution test or a Ready badge is not inference adoption.
|
|
- The keychain is no longer called while the shared file lock is held, but how long a slow real keychain
|
|
takes is unmeasured.
|
|
- The encrypted-file secret store is not in `ConnectionManager::standard()`. It layers the environment and
|
|
the OS keychain only, so on a machine with no keychain (a bare Linux session) a key cannot be saved
|
|
until an app supplies its own passphrase and builds a `LayeredStore` with `EncryptedFileStore`.
|
|
|
|
## Design limits
|
|
|
|
- The runtime is in-process: apps link `craft-ai-client` and run adapters themselves. A daemon with IPC
|
|
between apps and the adapters has been suggested by the maintainers and is a separate discussion; no
|
|
wire format is defined.
|
|
- A registered tool is a declaration, not installed code. No dynamic adapter registry for arbitrary user
|
|
code, no general tool executor and no automatic model installation.
|
|
- Interactive negative boxes are ignored, grounding exemplar boxes are not strict regions, hover JPEGs
|
|
are not exact masks, candidate order is not a semantic label, and refinement is not guaranteed to
|
|
improve quality. Runtime catalogue entries stay executable-only.
|