feat(client): offer SAM 2.1 on Burn as an optional engine after rten (#91)

* refactor(engine-rten): let another engine reuse the SAM 2.1 request handling

Loaded, Backend, Prediction, read_prompt, prompt_points and best_proposal become
public. Visibility only: the click and box mapping, the picture kept between
requests, the empty-mask answer and the size checks then live in one place for
every SAM 2.1 engine.

* feat(engine-burn): crate shell and the script that generates the SAM 2.1 architecture

tools/burn/generate_sam21.py checks the two published ONNX files against the
pinned size and SHA-256, runs the pinned burn-onnx 0.22.0 (an example of
craft-ai-burn-setup), applies the five asserted fixes, drops the generated
loaders that panic on a bad file, splits by submodule so every file is under
700 lines, formats and writes. --check proves the tree is what the script
produces. REVISION must equal the saved-weights cache revision. Cargo.lock
additions are Burn with wgpu and ndarray (generated).

* feat(burn-setup): prepare SAM 2.1 weights for Burn once, on this computer

Brings in craft-ai-burn-setup (feature `conversion`): checks the two published
ONNX files by size and SHA-256, converts them to Burn weights with the pinned
burn-onnx 0.22.0, and keeps one checked, locked, shared copy. Nothing is hosted
or downloaded. Mutation group bp (10 cases). Cargo.lock additions are the
optional burn-onnx tree (generated).

* feat(engine-burn): GENERATED SAM 2.1 tiny architecture (30 files)

Output of tools/burn/generate_sam21.py, revision sam21-tiny-burn-0.22.1-onnx-0.22.0-v1,
from the published encoder.onnx and decoder.onnx with burn-onnx 0.22.0 and its five
asserted fixes. Do not review line by line; rerun the script with --check instead.
Longest file 699 lines. Marked linguist-generated.

* feat(engine-burn): SAM 2.1 runner for image.segment on Burn

Sam21Runner (engine burn, model sam21) over the generated architecture. The request
handling, picture kept between requests, empty-mask answer and click/box mapping are
the shared Loaded from craft-ai-engine-rten, so only the model run lives here.
Weights load through BurnpackStore and come back as errors; every call into the
generated code is guarded against panics. Graphics card when one answers, else the
processor. An .onnx file is answered with 'prepare it first', not a guess.
16 unit tests on a fake backend: click and box mapping, empty mask, short mask,
a failing model, shared conformance run with Cancel, damaged and truncated files.

* test(engine-burn): live comparison with rten, and the shared revision check

The ignored live test runs the real SAM 2.1 files through rten (published ONNX) and
Burn (weights prepared on this computer) on the astronaut sample: one click and a box,
best-mask IoU, timings, and a truncated prepared file refused. The revision test fails
when the generator, the compiled architecture and the saved-weights cache key disagree.

* test+docs(engine-burn): mutation group be, weight-loading tests and the Burn engine guide

Group be: 15 cases, one target each, all caught (saved-weights marker, load completeness,
panic guard, answer length, best proposal, file names, declared card makers, the
revision shared by script, architecture and cache key, the catalogue route and the
.bpk binding rule). fully_applied is split out so the load checks can be tested without
the store's own refusals. Group nf's onnx case follows the rewritten extension check.
docs/BURN-ENGINE.md: how to regenerate, the five fixes, the revision bump rule, limits.

* feat(engine-burn): CRAFT_AI_BURN_DEVICE=cpu forces the processor, and the processor path is measured

* feat(engine-burn): prepare the saved copy when loading from the downloaded ONNX files

The host step: given the downloaded encoder.onnx with decoder.onnx beside it, the runner calls
craft-ai-burn-setup (exact source bytes checked, reused when already saved) and loads from the
burn-prepared folder beside the model folder. Behind the default 'prepare' feature; mutation
group be gains two cases (17/17); docs say what is measured.

* feat(client): optional burn feature and a catalogue route for SAM 2.1 on Burn

craft-ai-client gains the cargo feature burn (ProviderFactory::standard() then knows the
engine). SAM 2.1's recipe lists a burn NativeRoute after rten, with the same two published
ONNX files (one download for both, nothing converted is hosted). rten stays first and
the default. The binding accepts the prepared .bpk file. Engines now list rten, burn,
candle; the tests that assumed the Burn slot had no files, or that no built-in engine
exists whatever cargo features are on, say what they mean. burn_engine.rs runs two apps
through the client with the real prepared files (ignored).

* fix(burn-setup): sweep the scratch folders a killed run left, and keep craft-ai-features a dev dependency

CodeRabbit on #89: tempdir_in(cache) folders hold the two source copies (about 130 MB) and were
removed only by Drop, so a killed app leaked them for good. They now carry the cache key and a role
in their names, and after the key's lock is taken the same key's leftovers are removed (other keys'
folders, the lock file and links are left alone). Red first: leftovers_of_a_killed_run_are_swept_for_this_key_only.
Mutation cases burn-killed-run-leftovers-kept and burn-sweep-takes-other-keys, both caught.
craft-ai-features is only used by a unit test, so it moves to dev-dependencies.

* fix(engine-burn): review findings on #90

CodeRabbit: the generator now refuses an unknown option (a mistyped --chek used to select the run that
rewrites src/model; red first: tools/burn/test_generate_sam21.py failed to import parse_args); the
preparation live test needs the prepare feature and is ignored by default; the refusal test no longer
leaves burn-prepared/<key>.lock in the shared system temp folder (the model folder is nested one level
inside the test's own folder; confirmed the old layout leaked it on this PC); docs, STATUS and the
mutation README no longer describe the client feature and catalogue route, which arrive in the next
PR, as available.

* docs(client): say the Burn route and client feature exist, and what the status line really covers

CodeRabbit on #91: the STATUS row said the host step was not built. Given the downloaded
encoder.onnx the runner prepares the saved copy itself on first load; the row now says that, names
the limits (no progress or Cancel during that load, no app run through the shared binding, no live
run on macOS or Linux) and LOCAL-BURN-PREPARATION.md no longer calls engine registration separate
work. Also restores the client feature and catalogue route in the docs and rustdoc that the #90
review fix had marked as planned, and the three mutation-README claims for them.

* fix(burn-setup): sweep only the scratch folders, not a renamed copy of the key's folder

CodeRabbit on #89 (second round): sweep_stale removed every folder starting with <key>., so a damaged
copy a person renamed to <key>.backup would be deleted on the next preparation. It now matches only
<key>.work., <key>.conv. and <key>.out., the names the scratch folders are made with (one constant
for both). Red first: the sweep test gained a <key>.backup folder that must survive. Mutation group
bp 13/13 caught, including the new case burn-sweep-takes-a-renamed-copy.

* test(burn): the generator's usage errors must exit with code 1

CodeRabbit on #90 (second round): the parse_args tests asserted only SystemExit, which a SystemExit(0)
would also satisfy. They now assert the code. Checked against a mutant that exits 0: 6 failures.

* fix(engine-burn): a damaged saved copy says it is damaged and where it lives

CodeRabbit on #90 (third round) pointed out that a saved copy with a bad manifest or weights fails
the same way on every retry, behind the message 'could not be read or written. Try again.' It
proposed deleting the copy. The rule that a derived copy is refused, never repaired or deleted for
the person, stays (#89 and its tests pin it); the dead end was the wording. craft-ai-burn-setup now
answers every unusable saved copy with DAMAGED_COPY (remove it, prepare again, no 'try again'), and
the Burn engine adds where it lives: the burn-prepared folder beside the model folder.
Tests written first (they did not compile without DAMAGED_COPY and explain): a damaged copy of three
kinds gets that answer and is left untouched; the engine explains it and passes other errors on.
Mutation: bp 2 new cases and be 1 new case, all caught.

* fix(burn-setup): a missing or unreadable saved weight is a damaged copy too

CodeRabbit on #90 (fourth round): hash_file(...)? on a saved encoder.bpk or decoder.bpk that is gone
returned the generic 'try again' error before the damaged-copy branch could run. It now maps to
DAMAGED_COPY. Red first: the damaged-copy test gained 'missing encoder' and 'missing decoder' cases and
failed with the old message. New mutation case burn-missing-saved-weight-answered-with-try-again; the
saved-hash case was re-aimed at the new line. bp 16/16 caught.

* fix(burn-setup): a manifest that cannot be opened or read is a damaged copy too

CodeRabbit on #91 (round 5): File::open and read_to_end on an existing manifest returned the generic
'try again' error, so the engine gave no deletion guidance. Both now map to DAMAGED_COPY. Red first:
a_manifest_that_cannot_be_opened_is_a_damaged_copy (Windows: another handle holds it exclusively;
Unix: mode 0, ending quietly where permissions are not enforced) failed with the old message. The
Unix variant was not run here. New mutation case burn-unopenable-manifest-answered-with-try-again;
bp 17/17 caught (Windows).

* test(burn-setup): write the Unix permission as octal (clippy::non_octal_unix_permissions, CI only)

---------

Co-authored-by: scratch <s@s>
This commit is contained in:
Jeremy
2026-10-22 23:28:33 +01:00
committed by GitHub
parent fdad29cdd4
commit 1a02afd6be
22 changed files with 333 additions and 53 deletions
Generated
+1
View File
@@ -2010,6 +2010,7 @@ dependencies = [
"craft-ai-core",
"craft-ai-declarative",
"craft-ai-engine",
"craft-ai-engine-burn",
"craft-ai-engine-candle",
"craft-ai-engine-rten",
"craft-ai-features",
+2
View File
@@ -13,6 +13,7 @@ publish = false
# one on is the whole of adopting it: `ProviderFactory::standard()` then knows the engine.
candle = ["dep:craft-ai-engine-candle"]
rten = ["dep:craft-ai-engine-rten"]
burn = ["dep:craft-ai-engine-burn"]
[dependencies]
craft-ai-core.workspace = true
@@ -33,6 +34,7 @@ craft-ai-engine-candle = { path = "../ai-engine-candle", package = "craft-ai-eng
# The picker-preset "Released …" date goes through chrono (no hand-rolled calendar code).
chrono = { workspace = true }
craft-ai-engine-rten = { path = "../ai-engine-rten", package = "craft-ai-engine-rten", optional = true }
craft-ai-engine-burn = { path = "../ai-engine-burn", package = "craft-ai-engine-burn", optional = true }
[dev-dependencies]
serde_json.workspace = true
+2
View File
@@ -83,5 +83,7 @@ pub(crate) fn built_in() -> EngineRegistry {
registry.extend(craft_ai_engine_candle::runners());
#[cfg(feature = "rten")]
registry.extend(craft_ai_engine_rten::runners());
#[cfg(feature = "burn")]
registry.extend(craft_ai_engine_burn::runners());
registry
}
+8 -5
View File
@@ -333,18 +333,21 @@ fn local_connection(
}
}
/// An absolute path to an existing `.safetensors` or `.onnx` file, as text.
/// An absolute path to an existing `.safetensors`, `.onnx` or `.bpk` file, as text. The last is the
/// saved copy the Burn engine reads, made once on this computer from the published `.onnx` files.
fn weights_file(path: &Path) -> Result<String, AiError> {
let text = path
.to_str()
.filter(|t| !t.contains('\0'))
.ok_or_else(|| AiError::invalid("the weights path is not valid text"))?;
let known = path
.extension()
.is_some_and(|e| e.eq_ignore_ascii_case("safetensors") || e.eq_ignore_ascii_case("onnx"));
let known = path.extension().is_some_and(|e| {
["safetensors", "onnx", "bpk"]
.iter()
.any(|known| e.eq_ignore_ascii_case(known))
});
if !path.is_absolute() || !known || !path.is_file() {
return Err(AiError::invalid(
"the weights must be an existing .safetensors or .onnx file, given by its full path",
"the weights must be an existing .safetensors, .onnx or .bpk file, given by its full path",
));
}
Ok(text.to_owned())
+155
View File
@@ -0,0 +1,155 @@
//! The Burn engine through the client: the catalogue's route, the shared store, the cargo feature
//! and the engine. An app writes none of it, and two apps read the same choice.
//!
//! The first test needs no model files. The second (ignored) needs the real ones:
//! `CRAFT_TEST_SAM21_BURN_DIR` (the folder `craft-ai-burn-setup` saved, holding `encoder.bpk` and
//! `decoder.bpk`) and `CRAFT_TEST_SAM21_IMAGE`. Run:
//! `cargo test -p craft-ai-client --features burn --release --test burn_engine -- --ignored`.
#![cfg(feature = "burn")]
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use craft_ai_client::{AiClient, ToolDesk};
use craft_ai_core::{
AppId, ConsentLedger, Event, InputRole, JobState, Payload, Request, StoreFile, segment,
};
use craft_ai_features::Feature;
use craft_ai_native::secrets::MemoryStore;
use craft_ai_native::shared::SharedStore;
use serde_json::json;
/// The built-in `select-subject` feature.
fn feature() -> Feature {
craft_ai_features::builtin()
.into_iter()
.find(|f| f.id == "select-subject")
.unwrap()
}
/// A shared store in a temporary folder, with the launcher's desk over it.
struct World {
_dir: tempfile::TempDir,
shared: SharedStore,
}
impl World {
/// An empty shared store.
fn new() -> Self {
let dir = tempfile::tempdir().unwrap();
let shared = SharedStore::new(dir.path().join("connections.json"));
shared.replace(&StoreFile::default()).unwrap();
Self { _dir: dir, shared }
}
/// An `AiClient` for the app `app` over the shared store.
fn client(&self, app: &str) -> AiClient {
AiClient::new(
AppId::new(app).unwrap(),
self.shared.clone(),
Arc::new(MemoryStore::new()),
)
}
/// The launcher's part: choose Burn for the tool with `weights` as its file.
fn choose_burn(&self, weights: &Path) {
ToolDesk::new(self.client("launcher"), self.shared.clone(), None)
.enable_native_with(&feature(), "burn", weights)
.unwrap();
}
}
/// A one-click `image.segment` request for `image`.
fn request(image: Vec<u8>) -> Request {
Request::new(feature().capability.clone())
.with_input(
InputRole::Source,
Payload::inline("image/png", image).unwrap(),
)
.with_param("points", json!([{"x": 0.29, "y": 0.64, "positive": true}]))
}
/// The cargo feature is the whole of adopting the engine: a connection on the Burn engine is built,
/// and one that holds a download with no decoder is told, in plain words, what is missing instead
/// of being reported as an engine the app does not have.
#[test]
fn the_engine_is_known_and_an_incomplete_download_says_what_is_missing() {
let w = World::new();
let dir = tempfile::tempdir().unwrap();
let weights = dir.path().join("encoder.onnx");
std::fs::write(&weights, b"not the model").unwrap();
w.choose_burn(&weights);
let role = feature().role().unwrap();
let app = w.client("lightcraft");
// The refusal may come when the plan is made, when the job starts or as the job's own end.
let error = match app.plan(&role, request(vec![1, 2, 3])) {
Err(e) => e,
Ok(plan) => match app.start(&plan, &ConsentLedger::default()) {
Err(e) => e,
Ok(job) => {
match job
.wait(Duration::from_millis(20), Duration::from_secs(30))
.unwrap()
.state
{
JobState::Failed { error } => error,
other => panic!("expected a refusal, got {other:?}"),
}
}
},
};
// Only the encoder was written, so the answer names the missing decoder and where to go.
assert!(error.message.contains("decoder.onnx"), "{}", error.message);
assert!(error.message.contains("Set up AI"), "{}", error.message);
}
/// Two apps ask only for the capability and each gets the real model's mask of the suit at the
/// picture's size, with the shared result saying it matched.
#[test]
#[ignore = "needs the prepared SAM 2.1 files; see the module docs"]
fn two_apps_get_a_mask_from_the_real_model_by_asking_for_select_subject() {
let (Some(burn), Some(image)) = (
std::env::var_os("CRAFT_TEST_SAM21_BURN_DIR"),
std::env::var_os("CRAFT_TEST_SAM21_IMAGE"),
) else {
eprintln!("skipped: set CRAFT_TEST_SAM21_BURN_DIR and CRAFT_TEST_SAM21_IMAGE");
return;
};
let image = std::fs::read(image).unwrap();
let w = World::new();
w.choose_burn(&PathBuf::from(burn).join("encoder.bpk"));
let role = feature().role().unwrap();
for app in ["lightcraft", "photocraft"] {
let app = w.client(app);
let plan = app.plan(&role, request(image.clone())).unwrap();
let done = app
.start(&plan, &ConsentLedger::default())
.unwrap()
.wait(Duration::from_millis(50), Duration::from_secs(180))
.unwrap();
assert_eq!(done.state, JobState::Succeeded);
let outputs: Vec<_> = done
.events
.iter()
.filter_map(|e| match e {
Event::Output(o) => Some(o.clone()),
_ => None,
})
.collect();
assert_eq!(outputs.len(), 2, "the mask, then the shared result");
assert!(segment::read_found(&outputs).unwrap().matched);
let Some(Payload::Inline { data, .. }) = segment::mask(&outputs) else {
panic!("expected the mask itself");
};
let mut reader = png::Decoder::new(std::io::Cursor::new(data))
.read_info()
.unwrap();
let mut pixels = vec![0; reader.output_buffer_size().unwrap()];
let frame = reader.next_frame(&mut pixels).unwrap();
assert_eq!((frame.width, frame.height), (512, 512));
let selected = pixels.iter().filter(|v| **v > 127).count() as f64 / (512.0 * 512.0);
assert!((0.15..0.35).contains(&selected), "selected {selected}");
}
}
+6 -4
View File
@@ -59,7 +59,7 @@ fn without_engines() -> ProviderFactory {
}
/// An app with no built-in engine is advised ComfyUI, and every engine some usable recipe lists
/// (rten for SAM 2.1, Candle and Burn for SAM 3) is listed, neither usable nor recommended.
/// (rten and Burn for SAM 2.1, Candle for SAM 3) is listed, neither usable nor recommended.
#[test]
fn an_app_that_only_has_comfyui_is_advised_comfyui_and_the_built_in_ones_are_listed_unusable() {
let w = World::new();
@@ -67,7 +67,7 @@ fn an_app_that_only_has_comfyui_is_advised_comfyui_and_the_built_in_ones_are_lis
.client()
.engine_advice(&select_subject(), &windows_nvidia());
let providers: Vec<&str> = advice.iter().map(|a| a.provider.as_str()).collect();
assert_eq!(providers, ["comfyui", "rten", "candle", "burn"]);
assert_eq!(providers, ["comfyui", "rten", "burn", "candle"]);
assert!(advice[0].recommended && advice[0].usable);
assert!(advice[1..].iter().all(|a| !a.usable && !a.recommended));
}
@@ -77,7 +77,7 @@ fn an_app_that_only_has_comfyui_is_advised_comfyui_and_the_built_in_ones_are_lis
#[test]
fn an_app_with_a_candle_adapter_is_advised_candle_on_nvidia() {
let w = World::new();
let mut factory = ProviderFactory::standard();
let mut factory = without_engines();
factory.register("candle", |_, _| {
Err(AiError::unsupported("advice never builds a provider"))
});
@@ -244,10 +244,12 @@ fn a_recommended_provider_that_is_switched_off_is_not_marked() {
);
}
/// An engine the app registers without vouching for its build gets pace `Unsure`, and a provider that is
/// sure of the graphics card is listed before it.
#[test]
fn an_engine_the_app_registers_without_vouching_for_its_build_is_not_promised_the_card() {
let w = World::new();
let mut factory = ProviderFactory::standard();
let mut factory = without_engines();
factory.register("candle", |_, _| Err(AiError::unsupported("never built")));
let advice = w
.client()
+5
View File
@@ -34,8 +34,13 @@ impl World {
Self { dir, shared }
}
/// An app client over this world's shared store whose only engines are the ones registered here.
fn client(&self, app: &str, registry: EngineRegistry) -> AiClient {
let mut factory = ProviderFactory::standard();
// Only the engines a test registers exist, whatever cargo features are on.
for built_in in ["rten", "candle", "burn"] {
factory.unregister(built_in);
}
factory.register_engines(registry);
AiClient::new(
AppId::new(app).unwrap(),
+8 -4
View File
@@ -223,9 +223,9 @@ fn the_picker_then_lists_sam_2_1_as_ready_not_as_something_to_install() {
.unwrap();
let options = craft_ai_features::setup_options(&craft_ai_features::builtin());
let view = w.client("photocraft").picker(&role(), &options).unwrap();
// SAM 2.1 is ready; SAM 3 is not installed, so it keeps its way in; the two built-in engines
// (rten for SAM 2.1, Candle for SAM 3) are separate options.
assert_eq!(view.groups.len(), 4);
// SAM 2.1 is ready; SAM 3 is not installed, so it keeps its way in; the three built-in engines
// (rten and Burn for SAM 2.1, Candle for SAM 3) are separate options.
assert_eq!(view.groups.len(), 5);
let comfy = |state| {
view.groups
.iter()
@@ -242,7 +242,11 @@ fn the_picker_then_lists_sam_2_1_as_ready_not_as_something_to_install() {
);
// With an engine's adapter built in it can be set up; without it, it is only listed.
for (title, built) in [
("SAM 2.1, built in", cfg!(feature = "rten")),
("SAM 2.1, built in (the processor", cfg!(feature = "rten")),
(
"SAM 2.1, built in (any graphics card",
cfg!(feature = "burn"),
),
("SAM 3, built in", cfg!(feature = "candle")),
] {
let native = view
+11 -4
View File
@@ -1,10 +1,11 @@
//! Burn is a third local backend for the same tool, beside ComfyUI and Candle. This is the slot:
//! the route is registered, bound, read and refused exactly as the other two are. No inference runs
//! here, and no Burn runtime exists yet, so every app reports "no adapter" for it until one does.
//! Burn is a third local backend for the same tool, beside ComfyUI and Candle. These tests are for
//! an app that was built without it: the route is registered, bound, read and refused exactly as
//! the others are, and the app says plainly that it cannot use it (the `burn_engine` tests cover
//! an app that has the engine).
use std::path::PathBuf;
use std::sync::Arc;
use craft_ai_client::{AiClient, GroupState, RoleStatus, ToolDesk};
use craft_ai_client::{AiClient, GroupState, ProviderFactory, RoleStatus, ToolDesk};
use craft_ai_core::{AppId, Role, StoreFile};
use craft_ai_features::Feature;
use craft_ai_native::secrets::MemoryStore;
@@ -44,11 +45,17 @@ impl World {
}
/// An `AiClient` for the app `app` over the shared store of the test world.
fn client(&self, app: &str) -> AiClient {
// An app with no built-in engine, whatever cargo features are on.
let mut factory = ProviderFactory::standard();
for built_in in ["rten", "candle", "burn"] {
factory.unregister(built_in);
}
AiClient::new(
AppId::new(app).unwrap(),
self.shared.clone(),
Arc::new(MemoryStore::new()),
)
.with_factory(factory)
}
/// Writes a stand-in weights file in the world's folder and returns its path.
fn weights(&self) -> PathBuf {
+30
View File
@@ -101,6 +101,36 @@ fn the_native_route_is_registered_for_an_onnx_file() {
assert_eq!(route.file.as_deref(), file.to_str());
}
/// The Burn engine reads the saved copy made once on this computer: its encoder `.bpk` file is
/// accepted as the weights, registered under the Burn connection and bound, and an app reads that
/// exact file back. A file of any other kind is still refused.
#[test]
fn the_burn_route_is_registered_for_a_prepared_bpk_file() {
let w = World::new();
let file = w.weights("encoder.bpk");
let done = w
.desk("launcher")
.enable_native_with(&feature("select-subject"), "burn", &file)
.unwrap();
assert_eq!(done.connection.as_str(), "local-select-subject-burn");
assert_eq!(done.model.as_str(), "sam21");
let route = w
.client("lightcraft")
.local_route_for(&role())
.unwrap()
.unwrap();
assert_eq!(route.provider, "burn");
assert_eq!(route.file.as_deref(), file.to_str());
for other in ["encoder.bin", "encoder.pt", "encoder"] {
let refused = w.desk("launcher").enable_native_with(
&feature("select-subject"),
"burn",
&w.weights(other),
);
assert!(refused.is_err(), "{other} is not a weights file");
}
}
/// Two apps that share the store read the same registered weights file.
#[test]
fn two_apps_read_the_same_file() {
+2 -2
View File
@@ -3,8 +3,8 @@
//! engine for this model stays rten (`craft-ai-engine-rten`); this one is a choice.
//!
//! An app does not use this crate directly. It puts [`runners`] in an
//! [`EngineRegistry`](craft_ai_engine::EngineRegistry) and asks the client for a capability. See
//! `docs/ADD-AN-ENGINE.md`.
//! [`EngineRegistry`](craft_ai_engine::EngineRegistry), or turns on the `burn` feature of
//! `craft-ai-client`, and asks the client for a capability. See `docs/ADD-AN-ENGINE.md`.
//!
//! - **Same contract as rten**: clicks and a box in, a mask at the picture's size and a `Found`
//! result out. The request handling is shared with rten, not copied; this crate adds only the
+21 -7
View File
@@ -4,7 +4,8 @@ use super::florence2::florence2_large_files;
use crate::recipe::{Component, Licence, LocalRecipe, ModelFile, NativeRoute, NodePack, Source};
/// The published ONNX export of SAM 2.1 (tiny size) at a pinned revision. The built-in rten
/// engine reads these two files as they are: no conversion, nothing hosted by us. Both sit in
/// engine reads these two files as they are: no conversion, nothing hosted by us. The Burn
/// engine converts the same two once on the person's computer, so it lists them too. Both sit in
/// one folder, `encoder.onnx` first (the weights file the connection records).
fn sam21_onnx_files() -> Vec<ModelFile> {
const REVISION: &str = "12bdf0f712731bd0b52ccbff807683ce49b726b9";
@@ -91,12 +92,25 @@ pub(super) fn sam21() -> LocalRecipe {
min_vram_mb: None,
// Built-in engines that run SAM 2.1. It selects by clicking; a description needs the
// ComfyUI route above (Florence-2 finds the words).
native: vec![NativeRoute {
provider: "rten".to_owned(),
model: "sam21".to_owned(),
label: "SAM 2.1, built in (the processor, on any system)".to_owned(),
models: sam21_onnx_files(),
}],
native: vec![
NativeRoute {
provider: "rten".to_owned(),
model: "sam21".to_owned(),
label: "SAM 2.1, built in (the processor, on any system)".to_owned(),
models: sam21_onnx_files(),
},
// The same published files, converted once on this computer to the saved copy this
// engine reads (`craft-ai-burn-setup`). Nothing converted is hosted, so the download is
// the same two files and shares the folder with the route above. rten stays the first
// and default route; this one is a choice.
NativeRoute {
provider: "burn".to_owned(),
model: "sam21".to_owned(),
label: "SAM 2.1, built in (any graphics card, set up once on this computer)"
.to_owned(),
models: sam21_onnx_files(),
},
],
components: components(),
}
}
+26 -3
View File
@@ -56,12 +56,35 @@ fn a_route_without_files_of_its_own_has_no_weights() {
assert_eq!(route.total_bytes(), 0);
}
/// The Burn slot names no files, so it has no weights and nothing to download.
/// SAM 3's Burn slot still names no files, because no SAM 3 code exists for that engine and a
/// route that cannot run must not offer a 3.4 GB download.
#[test]
fn a_runtime_with_no_code_yet_offers_no_download() {
let feature = select_subject();
let recipe = feature.engine_recipe("burn").unwrap();
let burn = recipe.native.iter().find(|r| r.provider == "burn").unwrap();
let sam3 = feature
.locals
.iter()
.find(|r| r.workflow == "sam3-segment")
.unwrap();
let burn = sam3.native.iter().find(|r| r.provider == "burn").unwrap();
assert!(burn.models.is_empty() && burn.weights().is_none());
assert_eq!(burn.total_bytes(), 0);
}
/// SAM 2.1's Burn route reads the same two published files as rten, listed after it: one download
/// serves both, the encoder is the weights file, and Burn is never first. The files are not hosted
/// converted copies; the engine prepares its own on the computer.
#[test]
fn burns_sam_2_1_route_shares_the_published_files_and_comes_after_rten() {
let feature = select_subject();
let recipe = feature.engine_recipe("burn").unwrap();
assert_eq!(recipe.workflow, "sam21-segment");
let providers: Vec<_> = recipe.native.iter().map(|r| r.provider.as_str()).collect();
assert_eq!(providers, ["rten", "burn"]);
let (rten, burn) = (&recipe.native[0], &recipe.native[1]);
assert_eq!(burn.model, "sam21");
assert_eq!(burn.models, rten.models);
assert_eq!(burn.weights().unwrap().filename, "encoder.onnx");
assert_eq!(burn.total_bytes(), 130_129_714);
assert!(burn.models.iter().all(|f| f.filename.ends_with(".onnx")));
}
+25 -11
View File
@@ -70,23 +70,22 @@ fn missing_native_files_count_against_free_space() {
assert_eq!(plan.blockers.len(), 1);
}
/// A runtime with no files of its own (Burn today) or an unknown engine gets no plan and no download.
/// An unknown engine gets no plan and no download.
#[test]
fn unfinished_or_unknown_engines_offer_no_download() {
let feature = builtin()
.into_iter()
.find(|f| f.id == "select-subject")
.unwrap();
for engine in ["burn", "not-an-engine"] {
let plan = plan_native(&feature, engine, &Inventory::default());
assert_eq!(plan.state, State::NoLocalRoute);
assert!(plan.missing_files.is_empty());
}
let plan = plan_native(&feature, "not-an-engine", &Inventory::default());
assert_eq!(plan.state, State::NoLocalRoute);
assert!(plan.missing_files.is_empty());
}
/// The setup options list Candle with its own size, and leave out a runtime that names no files.
/// The setup options list Candle with its own size, and Burn with the size of SAM 2.1's two
/// published files (its SAM 3 slot names none, so it never offers that download).
#[test]
fn setup_options_offer_candle_with_its_own_size_but_not_unbuilt_burn() {
fn setup_options_offer_candle_with_its_own_size_and_burn_with_sam_2_1s() {
let options = setup_options(&builtin());
let candle = options
.iter()
@@ -98,7 +97,13 @@ fn setup_options_offer_candle_with_its_own_size_but_not_unbuilt_burn() {
);
assert_eq!(candle.size_bytes, Some(3_441_325_459));
assert!(candle.usable);
assert!(options.iter().all(|o| o.id != "native:select-subject:burn"));
let burn = options
.iter()
.find(|o| o.id == "native:select-subject:burn")
.unwrap();
assert!(matches!(&burn.route, Route::Native { provider, model, .. }
if provider == "burn" && model == "sam21"));
assert_eq!(burn.size_bytes, Some(130_129_714));
}
/// A recipe apps cannot run yet offers no native plan, even though its runtime names files.
@@ -214,7 +219,8 @@ fn each_engine_plans_from_the_first_usable_recipe_that_lists_it() {
.into_iter()
.find(|f| f.id == "select-subject")
.unwrap();
assert_eq!(feature.engines(), ["rten", "candle", "burn"]);
// Burn is listed on SAM 2.1 after rten, so it comes before Candle, which only SAM 3 lists.
assert_eq!(feature.engines(), ["rten", "burn", "candle"]);
let none = Inventory::default();
let rten = plan_native(&feature, "rten", &none);
assert_eq!(rten.recipe, "sam21-segment");
@@ -227,6 +233,10 @@ fn each_engine_plans_from_the_first_usable_recipe_that_lists_it() {
let candle = plan_native(&feature, "candle", &none);
assert_eq!(candle.recipe, "sam3-segment");
assert_eq!(candle.download_bytes, 3_441_325_459);
// Burn also plans from SAM 2.1 (its route is listed there), with the same two files.
let burn = plan_native(&feature, "burn", &none);
assert_eq!(burn.recipe, "sam21-segment");
assert_eq!(burn.missing_files, rten.missing_files);
let options = setup_options(std::slice::from_ref(&feature));
let native: Vec<_> = options
.iter()
@@ -235,6 +245,10 @@ fn each_engine_plans_from_the_first_usable_recipe_that_lists_it() {
.collect();
assert_eq!(
native,
["native:select-subject:rten", "native:select-subject:candle"]
[
"native:select-subject:rten",
"native:select-subject:burn",
"native:select-subject:candle"
]
);
}
+8 -3
View File
@@ -40,17 +40,22 @@ fn offer(profile: &SystemProfile, inventory: &Inventory, route: SetupRoute) -> O
setup_offer(&select_subject(), profile, inventory, route)
}
/// The kinds of the steps in an offer, in order.
fn kinds(o: &SetupOffer) -> Vec<StepKind> {
o.steps.iter().map(|s| s.kind).collect()
}
/// A native offer is for the runtime that was asked for: Burn names no files so it gets none, and
/// Candle gets its three own files.
/// A native offer is for the runtime that was asked for: Burn gets SAM 2.1's two published files
/// (the same download as rten, converted later on this computer), Candle gets its three own SAM 3
/// files, and a runtime no recipe lists gets nothing.
#[test]
fn a_native_offer_is_for_the_selected_provider_not_any_built_in_engine() {
let profile = pc(Os::Windows, vec![]);
let feature = select_subject();
assert!(setup_offer_native(&feature, "burn", &profile, &empty()).is_none());
assert!(setup_offer_native(&feature, "unknown-engine", &profile, &empty()).is_none());
let burn = setup_offer_native(&feature, "burn", &profile, &empty()).unwrap();
assert_eq!(kinds(&burn), [StepKind::Model; 2]);
assert_eq!(burn.download_bytes, 130_129_714);
let candle = setup_offer_native(&feature, "candle", &profile, &empty()).unwrap();
assert_eq!(kinds(&candle), [StepKind::Model; 3]);
assert_eq!(candle.download_bytes, 3_441_325_459);
+1 -1
View File
@@ -110,7 +110,7 @@ pub fn runners() -> Vec<std::sync::Arc<dyn craft_ai_engine::ModelRunner>> {
```
An engine that ships with this repository also gets a cargo feature on `craft-ai-client`
(`candle` and `rten` today). An app turns the feature on and `ProviderFactory::standard()` knows the engine:
(`candle`, `rten` and `burn` today). An app turns the feature on and `ProviderFactory::standard()` knows the engine:
nothing else. For any other engine, the app (or the launcher) puts the runners in a registry and
gives the client the registry:
+4 -2
View File
@@ -12,8 +12,8 @@ The pieces, and which crate owns each:
| The model run on Burn, weight loading, the panic guard | `craft-ai-engine-burn` |
| The compiled architecture (generated) | `crates/ai-engine-burn/src/model/` |
| One-time local conversion of the published ONNX files to saved weights | `craft-ai-burn-setup` (`docs/LOCAL-BURN-PREPARATION.md`) |
| The catalogue route (planned, the next change) | `crates/ai-features/src/catalogue/sam21.rs`: a `burn` route after `rten`, same two published files |
| The cargo feature (planned, the next change) | `craft-ai-client` feature `burn` |
| The catalogue route | `crates/ai-features/src/catalogue/sam21.rs`: a `burn` route after `rten`, same two published files |
| The cargo feature | `craft-ai-client` feature `burn` |
Nothing converted is hosted. The saved weights (`encoder.bpk`, `decoder.bpk`) are made on the
person's computer, once, and shared by every app. Given the downloaded `encoder.onnx` (with
@@ -69,6 +69,8 @@ cargo run -p craft-ai-burn-setup --features conversion --example prepare_sam21 -
CRAFT_TEST_SAM21_ONNX_DIR=<onnx dir> CRAFT_TEST_SAM21_BURN_DIR=<folder printed above> \
CRAFT_TEST_SAM21_IMAGE=<512 square sample> \
cargo test --release -p craft-ai-engine-burn --test live -j 8 -- --ignored --nocapture
# Through the client, two apps
cargo test -p craft-ai-client --features burn --test burn_engine -j 8 -- --ignored
python tools/mutation/mutation_check.py be
```
+3 -2
View File
@@ -60,8 +60,9 @@ choose a new global folder policy or migrate existing data.
explicit removal/rebuild of that derived copy. Originals are never removed or repaired.
- Cancel before starting writes nothing. Cancel during a synchronous converter call takes effect
when that component finishes, and its late result is discarded. Do not promise immediate Cancel.
- Preparation is not inference, readiness in an app, or permission to bind. Host UI and the generic
engine registration remain separate work.
- Preparation is not inference, readiness in an app, or permission to bind. The engine that reads the saved
weights is `craft-ai-engine-burn` (`docs/BURN-ENGINE.md`); the host's approval and progress screens
remain separate work.
## Exercise it
+1 -1
View File
@@ -16,7 +16,7 @@ No real hosted service, download host or repository was contacted by these tests
| ArtCraft Omni API key | `craft-ai-artcraft` | Prompt-only image and video. Account or subscription access has no adapter: no official programmatic route is documented. An API key is not evidence that a subscription covers the requests. |
| Hosted media copies | `craft-ai-media` | https only (loopback only for a connection marked local), every redirect re-checked, private addresses refused, size bounded, file type taken from the bytes. A failed copy keeps the original link with a notice. A busy host is retried with a bounded, doubling pause. Not built: retries for provider API calls, expiry tracking, resume, and an input-side upload cache. |
| Generic ComfyUI workflows | `craft-ai-comfyui` | Maps a prompt field; `image.inpaint` needs mapped `LoadImage` and `LoadImageMask` fields. A workflow is offered as a model only after `/object_info` confirms its nodes and checkpoint names; submission repeats the check. Output stays inline (32 MiB) unless the app gives a directory (1 GiB, local `file:` handle, app owns cleanup). Targeted cancel is off by default because an older server may treat it as a global interrupt. Imported image.segment workflows map a description and source to a checked, source-sized grayscale mask plus the shared result. Empty masks mean no match; source uploads are remembered for selection jobs. This is not calibrated absence detection: the example Florence graph can find a box for an absent subject. No input file handles or general parameter mapping yet. |
| Built-in engines | `craft-ai-engine`, `craft-ai-engine-candle`, `craft-ai-engine-rten`, `craft-ai-engine-burn` | Contract, registry and provider are tested with a fake runner and the shared conformance suite. SAM 3 on Candle (point and text prompts, processor, or Metal on macOS) ran on a real photograph and passes the same suite on Meta's checkpoint; the live tests skip without `CRAFT_SAM3_WEIGHTS`. It reads the Hugging Face checkpoint layout with tokenizer files, not the ComfyUI file. Pictures go in as PNG or JPEG (a JPEG's EXIF turn is applied). No Nvidia (CUDA) build, and the launcher does not download the checkpoint yet. SAM 2.1 on rten (processor only) reads the publisher's two `.onnx` files, pinned by size and SHA-256, with click and box prompts; a description is refused in plain words. Pictures are bounded by side and pixel count. With the real files in release on one Windows machine: a mask at the picture's size, and a hover through the ordinary job path with a median of about 35 ms after a first look of about 0.7 s (one picture, not a benchmark). SAM 2.1 is the first recipe for Select subject, so the plain built-in offer is SAM 2.1 on rten. No macOS or Linux run, and no app has run it through the shared binding yet. SAM 2.1 on Burn (`craft-ai-engine-burn`, optional, after rten in the recipe once its catalogue route lands) reads weights that `craft-ai-burn-setup` prepares once on the computer; on one Windows machine with a graphics card its mask matched rten's (best-mask IoU 1.0, one click and one box, one picture, not a benchmark). Given the downloaded `encoder.onnx`, the runner prepares the saved copy itself on first load (in `burn-prepared/` beside the model folder; no progress and no Cancel during that load). It has no catalogue route and no `craft-ai-client` feature yet, so no app can choose it. See `docs/BURN-ENGINE.md`. |
| Built-in engines | `craft-ai-engine`, `craft-ai-engine-candle`, `craft-ai-engine-rten`, `craft-ai-engine-burn` | Contract, registry and provider are tested with a fake runner and the shared conformance suite. SAM 3 on Candle (point and text prompts, processor, or Metal on macOS) ran on a real photograph and passes the same suite on Meta's checkpoint; the live tests skip without `CRAFT_SAM3_WEIGHTS`. It reads the Hugging Face checkpoint layout with tokenizer files, not the ComfyUI file. Pictures go in as PNG or JPEG (a JPEG's EXIF turn is applied). No Nvidia (CUDA) build, and the launcher does not download the checkpoint yet. SAM 2.1 on rten (processor only) reads the publisher's two `.onnx` files, pinned by size and SHA-256, with click and box prompts; a description is refused in plain words. Pictures are bounded by side and pixel count. With the real files in release on one Windows machine: a mask at the picture's size, and a hover through the ordinary job path with a median of about 35 ms after a first look of about 0.7 s (one picture, not a benchmark). SAM 2.1 is the first recipe for Select subject, so the plain built-in offer is SAM 2.1 on rten. No macOS or Linux run, and no app has run it through the shared binding yet. SAM 2.1 on Burn (`craft-ai-engine-burn`, cargo feature `burn`, optional, after rten in the recipe) reads weights that `craft-ai-burn-setup` prepares once on the computer; on one Windows machine with a graphics card its mask matched rten's (best-mask IoU 1.0, one click and one box, one picture, not a benchmark). Given the downloaded `encoder.onnx`, the runner prepares the saved copy itself on first load (in `burn-prepared/` beside the model folder; no progress and no Cancel during that load). No app has run it through the shared binding yet, and there has been no live run on macOS or Linux. See `docs/BURN-ENGINE.md`. |
| Declarative adapter | `craft-ai-declarative` | Request/response and queue flows from a data file; prompt-only inputs. No launcher screen to add a spec yet. |
| Custom connections, model sources, node packs | `craft-ai-core`, `craft-ai-comfy-env` | See `docs/custom-connections-and-sources.md`. Own-host tokens only, no key sent across a redirect, pickle formats need consent, unverified checksums are flagged. |
| Model folders | `craft-ai-core`, `craft-ai-comfy-env`, `craft-ai-client` | A person's own folders of model files, kept as a map (which sub-folders hold which kind) in the shared store, never moved, renamed or deleted. Scanned read only with limits (depth, files, entries, header size), one default for downloads, folders linked from ComfyUI are read only, and the map can be written as a ComfyUI extra-paths file. Not wired yet: downloads and the installed-files scan still use ComfyUI's own folders, and no launcher screen is merged. |
+1 -1
View File
@@ -12,7 +12,7 @@ bad mutant (fix the mutant). Cargo runs serially, targeted with `-p`, with `-j 8
| `mutation_check.py rt` | rten model runner, bounded picture decoding, resampling and prompt assembly. |
| `mutation_check.py rt` | rten engine picture handling: bounded PNG and JPEG decoding, mask encoding, resampling. |
| `mutation_check.py bp` | Local Burn weight preparation (`ai-burn-setup`): source hashes, cache key, damaged saved weights, cancel. |
| `mutation_check.py be` | Burn engine: saved-weights loading, the panic guard, answer checks, file names and the shared architecture revision. |
| `mutation_check.py be` | Burn engine: saved-weights loading, the panic guard, answer checks, file names, its catalogue route, the `.bpk` binding rule and the shared architecture revision. |
| `mutation_check.py gs` | Generic selection: scratch sinks, the refusal of clicks and mapped selection workflows. |
| `mutation_check.py gs` | Generic selection workflows: the scratch sinks, the refusal of clicks and mapped selection workflows. |
| `mutation_check.py gu` | Generic selection uploads: one upload per picture, the bounded memory, the resend after a cleared scratch area or a failed job. |
+12 -2
View File
@@ -1,6 +1,6 @@
"""be: the Burn engine's own rules (saved-weights loading, the panic guard, the answer checks, the
file names it accepts) and the shared revision. The catalogue route and the binding's `.bpk`
rule live with the client change and are checked there.
file names it accepts), the catalogue route it is listed by, the binding's `.bpk` rule and the
shared revision.
Not reached here, because they need the real model, a graphics card or a graph the converter
panics on (the ignored live tests in `crates/ai-engine-burn/tests/live.rs` and
@@ -58,4 +58,14 @@ BE = [
'onnx-0.22.0-v1";', 'onnx-0.22.0-v2";', PKG, 'revision', 'the_architecture_and_the_saved'),
('burn-revision-bumped-in-the-cache-key', 'crates/ai-burn-setup/src/sam21.rs',
'onnx-0.22.0-v1";', 'onnx-0.22.0-v2";', PKG, 'revision', 'the_architecture_and_the_saved'),
('burn-route-lists-no-files', 'crates/ai-features/src/catalogue/sam21.rs',
' models: sam21_onnx_files(),\n },\n ],',
' models: Vec::new(),\n },\n ],',
'craft-ai-features', 'native_files', 'burns_sam_2_1_route'),
('burn-route-named-rten', 'crates/ai-features/src/catalogue/sam21.rs',
'provider: "burn".to_owned(),', 'provider: "rten".to_owned(),',
'craft-ai-features', 'native_files', 'burns_sam_2_1_route'),
('burn-bpk-refused-when-binding', 'crates/ai-client/src/local.rs',
'["safetensors", "onnx", "bpk"]', '["safetensors", "onnx"]',
'craft-ai-client', 'local_native', 'burn_route_is_registered'),
]
+1 -1
View File
@@ -26,5 +26,5 @@ NF = [
('native-option-loses-its-model-id', F + 'options.rs', 'model: native.model.clone(),', 'model: String::new(),', FE, 'native_setup', 'candle_with_its_own_size'),
('onnx-is-not-weights', F + 'recipe.rs', ' name.ends_with(".safetensors") || name.ends_with(".onnx")', ' name.ends_with(".safetensors")', FE, 'native_onnx', 'an_onnx_file'),
('onnx-name-is-case-sensitive', F + 'recipe.rs', 'let name = m.filename.to_ascii_lowercase();', 'let name = m.filename.clone();', FE, 'native_onnx', 'an_onnx_file'),
('onnx-path-refused-when-binding', 'crates/ai-client/src/local.rs', '.is_some_and(|e| e.eq_ignore_ascii_case("safetensors") || e.eq_ignore_ascii_case("onnx"));', '.is_some_and(|e| e.eq_ignore_ascii_case("safetensors"));', 'craft-ai-client', 'local_native', 'the_native_route_is_registered'),
('onnx-path-refused-when-binding', 'crates/ai-client/src/local.rs', '["safetensors", "onnx", "bpk"]', '["safetensors", "bpk"]', 'craft-ai-client', 'local_native', 'the_native_route_is_registered'),
]