jyviko
17465a63b8
The seeded robustness goldens (tests/robustness/goldens.json, tests::robustness_goldens) recorded what truck built for seeds 0-95, so they now fail on every seed SMLib builds differently. The kernel-cases ladder (tests/kernel-cases/ladder, cargo xtask kernel-cases --ladder), whose cases each carry their own right answer, supersedes them. Gone with them: fuzz::golden, the CI goldens lane (the goldens output, SLOW_SEEDS, the re-record hint), their nextest filter and priority entries, the mutants sample and the tests/robustness/ test-data mapping in affected.py. The robustness seeds and tests::robustness stay. The docs point at the ladder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.5 KiB
Mutation testing
SolidCraft will move from truck to solidcore (#1597). The tests that have to catch regressions
across that switch are the ones that don't depend on the kernel: the reference tests
(docs/oracle.md) and the engine's command tests. The reference for what the kernel
builds is the kernel-cases ladder (tests/kernel-cases/ladder/,
docs/kernel-ladder.md). Most of the kernel's own unit tests call
truck-specific code and go away with it.
Mutation testing checks those suites. cargo-mutants makes small changes to
the source, one at a time (* becomes /, a condition is negated, a function returns a default
value), rebuilds, and runs the tests. A mutant the tests don't catch is a change in behaviour
nobody would notice, and solidcore could get the same thing wrong without anyone noticing.
What judges a mutant
.cargo/mutants.toml judges mutants by the black-box suites only:
oracle_suite(apps/solidcraft-cli/tests/oracle_cases.rs): every case intests/oraclewith expected values and noknown_failuremust pass, ascargo xtask oraclerequires. It is#[ignore]d so CI doesn't replay the oracle twice.
The kernel's unit tests are left out on purpose. A mutant that only they catch marks a behaviour the black-box suites would miss once the kernel is swapped.
Running it
cargo install cargo-mutants --locked
CARGO_TARGET_DIR=$PWD/target cargo mutants --in-place \
-f crates/kernel/src/measure.rs \
--test-package solidcraft-cli --test-package solidcraft-engine \
--baseline skip --timeout 900 --build-timeout 900
- Use a worktree you don't otherwise touch:
--in-placewrites each mutant into the source tree and restores it afterwards. --shard 0/200 --sharding round-robinwith-d crates/kerneltakes an even sample (every 200th mutant) of the whole crate.- The
mutantsprofile inCargo.tomlis theciprofile with incremental builds. It cuts the rebuild after a kernel change from about 3 minutes to about 15 seconds. - A mutant costs 1.5–5 minutes on a 12-core Mac, mostly the tests. The kernel has about 40,000 mutants, so run a sample or one file at a time, never the whole crate. This is a tool for finding gaps, not a CI gate.
Two problems with cargo-mutants 27.1 are behind the flags above:
- The baseline (the unmutated run) ignores
--test-packageand tests the mutated crate, which fails. Skip it with--baseline skip, set--timeoutyourself, and check the unmutated tree passes first. --reand--exclude-redon't filter "delete field" mutants. Saved diffs inmutants.out/diff/name the mutation, not the file, in their+++line: apply one withpatch <file> < <diff>.
Pilot results (October 2026, cb857f1f)
| Sample | Mutants that build | Caught by the black-box suites |
|---|---|---|
crates/kernel/src/measure.rs, every mutant |
85 | 85% |
| Whole kernel, every 200th mutant | 173 | 15% (95% interval 10–21%) |
Out of 49 of the kernel sample's survivors, the kernel's own unit tests caught 21 (43%) and
nothing caught 28 (57%). About 11% of survivors swap > and >= on floating-point values that
are never exactly equal, so they don't change behaviour.
The black-box suites are weakest on tessellation and the display mesh (0 of 19 mutants caught), offset, shell, emboss and lip (0 of 14), blends (15%) and STEP/IGES exchange (17%). For a survivor that a kernel unit test catches, that test's design is the starting point for a new oracle case or engine test.