mirror of
https://github.com/bherbruck/solidcraft.git
synced 2026-11-04 02:55:54 +00:00
ladder: cargo xtask kernel-cases --ladder, the complexity ladder's runner
The ladder orders kernel operations by complexity instead of by random goldens: a manifest (tests/kernel-cases/ladder/ladder.json) lists tiers in order and each tier's cases in order; every case is graded against its own answer like any rig case, and a golden (tests/kernel-cases/ladder/golden.json) records per case and kernel the status, the commands' time and what the kernel made. A run compares with the golden and flags slowdowns (over twice the time and 0.25 s more), status changes and changed counts. `--record REASON` re-measures one case at a time, rewrites the golden with its reason and sorts each tier by the first kernel's time. The report, docs/kernel-ladder.md, gives per tier and kernel the counts per status and the frontier: the first case in ladder order the kernel fails. It reuses the rig's process-per-case runner, timeout and cells; results now carry their path under tests/kernel-cases/ so cases from other areas can be listed in a tier. The README documents `base`, `missing` and the ladder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QWhMwdqJYQKQq8RCLSL1LT
This commit is contained in:
@@ -47,6 +47,9 @@ Each case has:
|
||||
files the case writes (an export).
|
||||
- `expected.json`: the right answer, graded by `solidcraft-cli kernel-cases`.
|
||||
|
||||
A script may start with `"base": "<case directory>"` (relative to its own directory): that case's
|
||||
commands, and its own base's, run first. The ladder's cases build on each other this way.
|
||||
|
||||
Each area also has `skipped.json` (or several, `skipped-<part>.json`, where an area is triaged in
|
||||
parts): every issue and PR looked at that did not become a case, with one line each saying why.
|
||||
|
||||
@@ -81,11 +84,38 @@ parts): every issue and PR looked at that did not become a case, with one line e
|
||||
| `body_count`, `faces`, `edges`, `vertices` | Exact counts, where the geometry fixes them. Faces and edges count whole analytic surfaces and curves once (a full cylinder side is one face). |
|
||||
| `volume_mm3`, `area_mm2`, `center_of_mass`, `bbox` | As in `tests/oracle`: totals over every body, relative tolerance 1e-3 for volume and area, 1e-3 of the box diagonal for positions. |
|
||||
| `tolerance` | Optional overrides: `rel`, `abs_mm`. |
|
||||
| `missing` | A capability SolidCraft has no command or parameter for yet (`"solid.thread has no start count"`): the case is reported **not supported yet** without running, so the gap stays visible. |
|
||||
|
||||
## The complexity ladder
|
||||
|
||||
`ladder/` orders kernel operations by complexity instead of by where they went wrong: tiers from
|
||||
primitives through degenerate booleans, analytic and spline profiles, holes and taps, threads,
|
||||
fillets, chamfers, shells, patterns, to whole parts. Later tiers build on earlier tiers' shapes (a
|
||||
running part grows one tier at a time through `base`), so the last case carries everything.
|
||||
|
||||
- `ladder/<NN-tier>/<NNN-name>/`: cases in the format above, each a test with its own answer.
|
||||
- `ladder/ladder.json`: the tiers in order and, per tier, its cases in order (paths under
|
||||
`tests/kernel-cases/`, so a tier may also list a rig case from another area).
|
||||
- `ladder/golden.json`: per case and kernel the status, the commands' time and what the kernel
|
||||
made (counts, volume, area). Runs compare with it and flag slowdowns (over twice the time and
|
||||
0.25 s more) and changed counts or statuses. It is re-recorded only with a reason.
|
||||
- `tools/kernel-ladder/cases.py` writes the recipes and answers (standard-library Python; each
|
||||
`method` gives the derivation).
|
||||
|
||||
```sh
|
||||
cargo xtask kernel-cases --ladder --kernel both --cli target/release/solidcraft-cli
|
||||
cargo xtask kernel-cases --ladder --kernel both --cli target/release/solidcraft-cli --record "why"
|
||||
```
|
||||
|
||||
`--record` measures one case at a time, rewrites the golden and sorts each tier by the first
|
||||
kernel's time, fastest first; commit both files. The report, `docs/kernel-ladder.md`, gives per
|
||||
tier and kernel the counts per status and the **frontier**: the first case in ladder order the
|
||||
kernel fails.
|
||||
|
||||
## Results
|
||||
|
||||
Per case and kernel: **pass**; **fail** with the numbers; **not supported yet** (the kernel says
|
||||
so); **crash** (a panic caught by the guards, or the process died); **hang** (no result before
|
||||
so, or the case names a `missing` capability); **crash** (a panic caught by the guards, or the process died); **hang** (no result before
|
||||
the timeout).
|
||||
|
||||
## Exporting to solidcore
|
||||
|
||||
@@ -20,17 +20,17 @@ const DEFAULT_TIMEOUT_S: u64 = 180;
|
||||
/// Per kernel, how many cases ended in each status.
|
||||
type StatusCounts = BTreeMap<String, BTreeMap<String, usize>>;
|
||||
|
||||
struct Options {
|
||||
kernels: Vec<String>,
|
||||
pub(crate) struct Options {
|
||||
pub(crate) kernels: Vec<String>,
|
||||
cli: Option<String>,
|
||||
area: Option<String>,
|
||||
timeout: Duration,
|
||||
jobs: usize,
|
||||
pub(crate) jobs: usize,
|
||||
/// Where to keep every case's full result as JSON.
|
||||
results: Option<String>,
|
||||
}
|
||||
|
||||
fn options(rest: &[&str]) -> Result<Options, String> {
|
||||
pub(crate) fn options(rest: &[&str]) -> Result<Options, String> {
|
||||
let arg = |name: &str| rest.iter().position(|a| *a == name).and_then(|i| rest.get(i + 1)).map(|s| s.to_string());
|
||||
let kernels = match arg("--kernel").as_deref() {
|
||||
None | Some("default") => vec!["default".to_string()],
|
||||
@@ -43,7 +43,7 @@ fn options(rest: &[&str]) -> Result<Options, String> {
|
||||
Ok(Options { kernels, cli: arg("--cli"), area: arg("--area"), timeout: Duration::from_secs(secs), jobs, results: arg("--results") })
|
||||
}
|
||||
|
||||
fn read(path: &Path) -> Value {
|
||||
pub(crate) fn read(path: &Path) -> Value {
|
||||
std::fs::read_to_string(path).ok().and_then(|t| serde_json::from_str(&t).ok()).unwrap_or(Value::Null)
|
||||
}
|
||||
|
||||
@@ -66,8 +66,16 @@ fn cases(dir: &Path, only: Option<&str>) -> Result<Vec<PathBuf>, String> {
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// One case on one kernel, in its own process, killed after the timeout.
|
||||
/// One case on one kernel, in its own process, killed after the timeout; `path` is the case's
|
||||
/// directory under `tests/kernel-cases/`.
|
||||
fn run_one(o: &Options, kernel: &str, case: &Path) -> Value {
|
||||
let mut r = run_case(o, kernel, case);
|
||||
let base = root().join("tests/kernel-cases");
|
||||
r["path"] = json!(case.strip_prefix(&base).unwrap_or(case).to_string_lossy().replace('\\', "/"));
|
||||
r
|
||||
}
|
||||
|
||||
fn run_case(o: &Options, kernel: &str, case: &Path) -> Value {
|
||||
let rel = case.strip_prefix(root()).unwrap_or(case).to_path_buf();
|
||||
let mut cmd = match &o.cli {
|
||||
Some(bin) => Command::new(bin),
|
||||
@@ -111,7 +119,8 @@ fn run_one(o: &Options, kernel: &str, case: &Path) -> Value {
|
||||
};
|
||||
match serde_json::from_slice::<Value>(&out.stdout).ok().and_then(|v| v.as_array().and_then(|a| a.first().cloned())) {
|
||||
Some(mut r) => {
|
||||
r["seconds"] = json!(start.elapsed().as_secs_f64());
|
||||
// `seconds` is the commands' own time, measured inside the case's process.
|
||||
r["process_seconds"] = json!(start.elapsed().as_secs_f64());
|
||||
r
|
||||
}
|
||||
// No JSON: the process died (a native crash, an abort).
|
||||
@@ -123,7 +132,7 @@ fn run_one(o: &Options, kernel: &str, case: &Path) -> Value {
|
||||
}
|
||||
}
|
||||
|
||||
fn run_all(o: &Options, list: &[PathBuf]) -> Vec<Value> {
|
||||
pub(crate) fn run_all(o: &Options, list: &[PathBuf]) -> Vec<Value> {
|
||||
let jobs: Vec<(String, PathBuf)> = o.kernels.iter().flat_map(|k| list.iter().map(move |c| (k.clone(), c.clone()))).collect();
|
||||
let next = Arc::new(Mutex::new(0usize));
|
||||
let results = Arc::new(Mutex::new(vec![Value::Null; jobs.len()]));
|
||||
@@ -157,7 +166,7 @@ fn run_all(o: &Options, list: &[PathBuf]) -> Vec<Value> {
|
||||
Arc::try_unwrap(results).ok().and_then(|m| m.into_inner().ok()).unwrap_or_default()
|
||||
}
|
||||
|
||||
fn cell(r: Option<&Value>) -> String {
|
||||
pub(crate) fn cell(r: Option<&Value>) -> String {
|
||||
let Some(r) = r else { return String::new() };
|
||||
let status = r["status"].as_str().unwrap_or("?");
|
||||
let mut notes: Vec<String> = r["checks"]
|
||||
@@ -318,6 +327,9 @@ fn write_report(o: &Options, dir: &Path, list: &[PathBuf], results: &[Value]) ->
|
||||
}
|
||||
|
||||
pub fn run(rest: &[&str]) -> Result<(), String> {
|
||||
if rest.contains(&"--ladder") {
|
||||
return crate::kernel_ladder::run(rest);
|
||||
}
|
||||
let o = options(rest)?;
|
||||
let dir = root().join("tests/kernel-cases");
|
||||
let list = cases(&dir, o.area.as_deref())?;
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
//! `cargo xtask kernel-cases --ladder [--record REASON]`: the kernel complexity ladder.
|
||||
//!
|
||||
//! `tests/kernel-cases/ladder/ladder.json` lists tiers in order of complexity and, per tier, its
|
||||
//! cases in order (paths under `tests/kernel-cases/`, so a tier may also list rig cases from other
|
||||
//! areas). Each case is a test like any rig case, graded against its own `expected.json`. On top,
|
||||
//! `tests/kernel-cases/ladder/golden.json` records per case and kernel the status, the commands'
|
||||
//! time and what the kernel made (counts, volume, area): the run compares with it and flags
|
||||
//! slowdowns and changes. `--record REASON` re-measures the golden (one case at a time unless
|
||||
//! `--jobs` says otherwise, so the times are not shared with other cases) and re-sorts each tier
|
||||
//! by the first kernel's time, fastest first; both files are committed.
|
||||
//!
|
||||
//! The report, `docs/kernel-ladder.md`, gives per tier and kernel the counts per status and the
|
||||
//! robustness frontier: the first case, in ladder order, the kernel fails.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::kernel_cases::{Options, cell, options, read, run_all};
|
||||
use crate::root;
|
||||
|
||||
/// A case is flagged slower when it takes more than this many times its golden time…
|
||||
const SLOWDOWN_FACTOR: f64 = 2.0;
|
||||
/// …and at least this many seconds more (timer noise on fast cases is not a slowdown).
|
||||
const SLOWDOWN_MIN_S: f64 = 0.25;
|
||||
|
||||
struct Tier {
|
||||
tier: String,
|
||||
what: String,
|
||||
cases: Vec<String>,
|
||||
}
|
||||
|
||||
fn manifest_path() -> PathBuf {
|
||||
root().join("tests/kernel-cases/ladder/ladder.json")
|
||||
}
|
||||
|
||||
fn golden_path() -> PathBuf {
|
||||
root().join("tests/kernel-cases/ladder/golden.json")
|
||||
}
|
||||
|
||||
fn tiers(manifest: &Value) -> Result<Vec<Tier>, String> {
|
||||
let list = manifest["tiers"].as_array().ok_or("ladder.json: `tiers` must be an array")?;
|
||||
let mut out = Vec::new();
|
||||
for t in list {
|
||||
let tier = t["tier"].as_str().ok_or("ladder.json: a tier without `tier`")?.to_string();
|
||||
let cases = t["cases"]
|
||||
.as_array()
|
||||
.ok_or_else(|| format!("ladder.json: tier {tier} without `cases`"))?
|
||||
.iter()
|
||||
.map(|c| c.as_str().map(str::to_string).ok_or_else(|| format!("ladder.json: tier {tier}: a case that is not a path")))
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
out.push(Tier { tier, what: t["what"].as_str().unwrap_or("").to_string(), cases });
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Every case directory under `ladder/` is listed once in the manifest, and every listed case
|
||||
/// exists: a case left out would never run.
|
||||
fn check_listed(tiers: &[Tier]) -> Result<(), String> {
|
||||
let base = root().join("tests/kernel-cases");
|
||||
let mut listed: BTreeMap<&str, usize> = BTreeMap::new();
|
||||
for t in tiers {
|
||||
for c in &t.cases {
|
||||
*listed.entry(c.as_str()).or_default() += 1;
|
||||
let d = base.join(c);
|
||||
if !(d.join("recipe.json").is_file() && d.join("expected.json").is_file()) {
|
||||
return Err(format!("ladder.json: {c} has no recipe.json and expected.json"));
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some((c, _)) = listed.iter().find(|(_, n)| **n > 1) {
|
||||
return Err(format!("ladder.json: {c} is listed more than once"));
|
||||
}
|
||||
let ladder = base.join("ladder");
|
||||
for tier in std::fs::read_dir(&ladder).map_err(|e| format!("{}: {e}", ladder.display()))?.flatten() {
|
||||
if !tier.path().is_dir() {
|
||||
continue;
|
||||
}
|
||||
for case in std::fs::read_dir(tier.path()).map_err(|e| e.to_string())?.flatten() {
|
||||
let rel = case.path().strip_prefix(&base).map(|p| p.to_string_lossy().replace('\\', "/")).unwrap_or_default();
|
||||
if case.path().join("recipe.json").is_file() && !listed.contains_key(rel.as_str()) {
|
||||
return Err(format!("ladder.json: {rel} is not listed in any tier"));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn rel_of(r: &Value) -> String {
|
||||
r["path"].as_str().unwrap_or("").to_string()
|
||||
}
|
||||
|
||||
/// The golden entry for one result: status, the commands' time, what was made.
|
||||
fn golden_entry(r: &Value) -> Value {
|
||||
let mut g = json!({"status": r["status"], "seconds": r["seconds"]});
|
||||
if let Some(m) = r.get("measured").filter(|m| m.is_object()) {
|
||||
g["measured"] = m.clone();
|
||||
}
|
||||
g
|
||||
}
|
||||
|
||||
/// What changed against the golden: a slowdown, a status change, other counts.
|
||||
fn flags(r: &Value, g: Option<&Value>) -> Vec<String> {
|
||||
let Some(g) = g else { return vec!["no golden".into()] };
|
||||
let mut out = Vec::new();
|
||||
if g["status"] != r["status"] {
|
||||
out.push(format!("status was {}", g["status"].as_str().unwrap_or("?")));
|
||||
}
|
||||
if let (Some(now), Some(then)) = (r["seconds"].as_f64(), g["seconds"].as_f64())
|
||||
&& now > then * SLOWDOWN_FACTOR
|
||||
&& now - then > SLOWDOWN_MIN_S
|
||||
{
|
||||
out.push(format!("slower: {now:.2} s (golden {then:.2} s)"));
|
||||
}
|
||||
for k in ["body_count", "faces", "edges", "vertices"] {
|
||||
let (now, then) = (&r["measured"][k], &g["measured"][k]);
|
||||
if !now.is_null() && !then.is_null() && now != then {
|
||||
out.push(format!("{k} {now} (golden {then})"));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn secs(v: &Value) -> String {
|
||||
v.as_f64().map(|s| format!("{s:.3}")).unwrap_or_else(|| "–".into())
|
||||
}
|
||||
|
||||
fn write_report(o: &Options, tiers: &[Tier], results: &BTreeMap<(String, String), Value>, golden: &Value) -> Result<String, String> {
|
||||
let mut md = String::from(
|
||||
"# Kernel complexity ladder\n\nGenerated by `cargo xtask kernel-cases --ladder`. The ladder in [`tests/kernel-cases/ladder/`](../tests/kernel-cases/ladder/) orders kernel operations by complexity: each tier builds on the shapes of the tiers before it, so the last case carries everything. Every case is a test with its own right answer (analysis or invariants, as in the [regression rig](kernel-cases.md)); the golden (`tests/kernel-cases/ladder/golden.json`) records per kernel its time and what it made, orders each tier by time and flags slowdowns.\n\nThe **frontier** is the first case of a tier, in ladder order, that the kernel fails (fail, crash or hang).\n\n## Tiers\n\n| Tier | What | Cases |",
|
||||
);
|
||||
for k in &o.kernels {
|
||||
md += &format!(" {k} | {k} frontier |");
|
||||
}
|
||||
md += "\n|---|---|---|";
|
||||
md += &"---|---|".repeat(o.kernels.len());
|
||||
md += "\n";
|
||||
let mut rows = String::new();
|
||||
let mut totals: BTreeMap<String, BTreeMap<String, usize>> = BTreeMap::new();
|
||||
for t in tiers {
|
||||
md += &format!("| {} | {} | {} |", t.tier, t.what.replace('|', "/"), t.cases.len());
|
||||
for k in &o.kernels {
|
||||
let mut counts: BTreeMap<String, usize> = BTreeMap::new();
|
||||
let mut frontier = None;
|
||||
for c in &t.cases {
|
||||
let status = results.get(&(c.clone(), k.clone())).and_then(|r| r["status"].as_str()).unwrap_or("not run").to_string();
|
||||
if frontier.is_none() && matches!(status.as_str(), "fail" | "crash" | "hang") {
|
||||
frontier = Some(c.rsplit('/').next().unwrap_or(c).to_string());
|
||||
}
|
||||
*totals.entry(k.clone()).or_default().entry(status.clone()).or_default() += 1;
|
||||
*counts.entry(status).or_default() += 1;
|
||||
}
|
||||
let counts = counts.iter().map(|(s, n)| format!("{n} {s}")).collect::<Vec<_>>().join(", ");
|
||||
md += &format!(" {counts} | {} |", frontier.unwrap_or_else(|| "–".into()));
|
||||
}
|
||||
md += "\n";
|
||||
for c in &t.cases {
|
||||
let want = read(&root().join("tests/kernel-cases").join(c).join("expected.json"));
|
||||
rows += &format!("| {} | {} | {} |", c, want["what"].as_str().unwrap_or("").replace('|', "/"), want["source"].as_str().unwrap_or("?"));
|
||||
for k in &o.kernels {
|
||||
let r = results.get(&(c.clone(), k.clone()));
|
||||
let g = golden["cases"][c].get(k);
|
||||
let f = r.map(|r| flags(r, g)).unwrap_or_default();
|
||||
rows += &format!(
|
||||
" {} | {} | {} | {} |",
|
||||
cell(r),
|
||||
r.map(|r| secs(&r["seconds"])).unwrap_or_default(),
|
||||
g.map(|g| secs(&g["seconds"])).unwrap_or_else(|| "–".into()),
|
||||
f.join("; ").replace('|', "/")
|
||||
);
|
||||
}
|
||||
rows += "\n";
|
||||
}
|
||||
}
|
||||
// Cases whose bodies SMLib's check gave no verdict on (it can't take the record yet).
|
||||
let unchecked: Vec<String> = o
|
||||
.kernels
|
||||
.iter()
|
||||
.map(|k| {
|
||||
let n = results
|
||||
.iter()
|
||||
.filter(|((_, kk), r)| {
|
||||
kk == k && r["checks"].as_array().into_iter().flatten().any(|c| c["got"].as_str().is_some_and(|g| g.starts_with("no verdict")))
|
||||
})
|
||||
.count();
|
||||
format!("{k}: {n}")
|
||||
})
|
||||
.collect();
|
||||
md += &format!(
|
||||
"\nCases where SMLib's check gave no verdict on a body (SMLib can't take the record yet), so only the record's checks graded validity: {}.\n",
|
||||
unchecked.join(", ")
|
||||
);
|
||||
md += "\n## Cases, in ladder order\n\nTimes are the commands' own, in seconds, from the case's process; **golden** is the recorded time.\n\n| Case | What | Answer |";
|
||||
for k in &o.kernels {
|
||||
md += &format!(" {k} | s | golden s | changes |");
|
||||
}
|
||||
md += "\n|---|---|---|";
|
||||
md += &"---|---|---|---|".repeat(o.kernels.len());
|
||||
md += "\n";
|
||||
md += &rows;
|
||||
std::fs::write(root().join("docs/kernel-ladder.md"), md).map_err(|e| format!("docs/kernel-ladder.md: {e}"))?;
|
||||
let n: usize = tiers.iter().map(|t| t.cases.len()).sum();
|
||||
let summary: Vec<String> = o
|
||||
.kernels
|
||||
.iter()
|
||||
.map(|k| {
|
||||
let pass = totals.get(k).and_then(|m| m.get("pass")).copied().unwrap_or(0);
|
||||
format!("{k}: {pass}/{n} pass")
|
||||
})
|
||||
.collect();
|
||||
Ok(format!("kernel ladder: {} → docs/kernel-ladder.md", summary.join(", ")))
|
||||
}
|
||||
|
||||
/// The golden from this run's results (keeping what it holds for kernels not run), and the
|
||||
/// manifest with each tier sorted by the first kernel's time.
|
||||
fn record(
|
||||
o: &Options,
|
||||
reason: &str,
|
||||
manifest: &Value,
|
||||
tiers: &[Tier],
|
||||
results: &BTreeMap<(String, String), Value>,
|
||||
golden: &Value,
|
||||
) -> Result<(), String> {
|
||||
let mut cases = golden["cases"].as_object().cloned().unwrap_or_default();
|
||||
for t in tiers {
|
||||
for c in &t.cases {
|
||||
let entry = cases.entry(c.clone()).or_insert_with(|| json!({}));
|
||||
for k in &o.kernels {
|
||||
if let Some(r) = results.get(&(c.clone(), k.clone())) {
|
||||
entry[k.as_str()] = golden_entry(r);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let new_golden = json!({"reason": reason, "kernels": o.kernels, "cases": cases});
|
||||
let mut m = manifest.clone();
|
||||
let lead = o.kernels.first().cloned().unwrap_or_else(|| "default".into());
|
||||
if let Some(list) = m["tiers"].as_array_mut() {
|
||||
for (t, tier) in list.iter_mut().zip(tiers) {
|
||||
let mut sorted: Vec<(f64, usize, &String)> = tier
|
||||
.cases
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, c)| (results.get(&(c.clone(), lead.clone())).and_then(|r| r["seconds"].as_f64()).unwrap_or(f64::INFINITY), i, c))
|
||||
.collect();
|
||||
// Fastest first; equal times keep the manifest's order.
|
||||
sorted.sort_by(|a, b| a.0.total_cmp(&b.0).then(a.1.cmp(&b.1)));
|
||||
t["cases"] = json!(sorted.iter().map(|(_, _, c)| c.as_str()).collect::<Vec<_>>());
|
||||
}
|
||||
}
|
||||
let pretty = |v: &Value| serde_json::to_string_pretty(v).map(|s| s + "\n").map_err(|e| e.to_string());
|
||||
std::fs::write(golden_path(), pretty(&new_golden)?).map_err(|e| format!("golden.json: {e}"))?;
|
||||
std::fs::write(manifest_path(), pretty(&m)?).map_err(|e| format!("ladder.json: {e}"))?;
|
||||
println!("kernel ladder: golden recorded ({reason}); tiers re-sorted by {lead} time");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn run(rest: &[&str]) -> Result<(), String> {
|
||||
let reason = rest.iter().position(|a| *a == "--record").map(|i| rest.get(i + 1).copied().filter(|r| !r.starts_with("--")));
|
||||
let reason = match reason {
|
||||
None => None,
|
||||
Some(None) => return Err("--record needs a reason: --record \"why the golden changes\"".into()),
|
||||
Some(Some(r)) => Some(r.to_string()),
|
||||
};
|
||||
let mut o = options(rest)?;
|
||||
if reason.is_some() && !rest.contains(&"--jobs") {
|
||||
o.jobs = 1;
|
||||
}
|
||||
let manifest = read(&manifest_path());
|
||||
if manifest.is_null() {
|
||||
return Err(format!("{}: missing or not JSON", manifest_path().display()));
|
||||
}
|
||||
let tiers = tiers(&manifest)?;
|
||||
check_listed(&tiers)?;
|
||||
let base = root().join("tests/kernel-cases");
|
||||
let list: Vec<PathBuf> = tiers.iter().flat_map(|t| t.cases.iter().map(|c| base.join(c))).collect();
|
||||
let results: BTreeMap<(String, String), Value> =
|
||||
run_all(&o, &list).into_iter().map(|r| ((rel_of(&r), r["kernel"].as_str().unwrap_or("").to_string()), r)).collect();
|
||||
let golden = read(&golden_path());
|
||||
if let Some(reason) = &reason {
|
||||
record(&o, reason, &manifest, &tiers, &results, &golden)?;
|
||||
let tiers = self::tiers(&read(&manifest_path()))?;
|
||||
println!("{}", write_report(&o, &tiers, &results, &read(&golden_path()))?);
|
||||
} else {
|
||||
println!("{}", write_report(&o, &tiers, &results, &golden)?);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn flags_slowdowns_status_and_count_changes_only() {
|
||||
let golden = json!({"status": "pass", "seconds": 1.0, "measured": {"faces": 6, "edges": 12}});
|
||||
let same = json!({"status": "pass", "seconds": 1.2, "measured": {"faces": 6, "edges": 12}});
|
||||
assert!(flags(&same, Some(&golden)).is_empty());
|
||||
// Twice as slow but within timer noise of a fast case.
|
||||
let fast = json!({"status": "pass", "seconds": 0.03, "measured": {}});
|
||||
assert!(flags(&fast, Some(&json!({"status": "pass", "seconds": 0.01}))).is_empty());
|
||||
let worse = json!({"status": "fail", "seconds": 3.0, "measured": {"faces": 7, "edges": 12}});
|
||||
let f = flags(&worse, Some(&golden));
|
||||
assert_eq!(f.len(), 3, "{f:?}");
|
||||
assert_eq!(flags(&same, None), ["no golden"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tiers_need_paths() {
|
||||
let m = json!({"tiers": [{"tier": "01-a", "what": "w", "cases": ["ladder/01-a/001-x"]}]});
|
||||
let t = tiers(&m).unwrap();
|
||||
assert_eq!((t[0].tier.as_str(), t[0].cases.len()), ("01-a", 1));
|
||||
assert!(tiers(&json!({"tiers": [{"tier": "01-a", "cases": [3]}]})).is_err());
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,7 @@ mod features;
|
||||
mod i18n;
|
||||
mod icons;
|
||||
mod kernel_cases;
|
||||
mod kernel_ladder;
|
||||
mod layers;
|
||||
mod licences;
|
||||
mod lists;
|
||||
@@ -51,6 +52,10 @@ commands:
|
||||
process each, killed after --timeout, default 180 s) with each kernel asked for,
|
||||
grade it against the case's own answer and write docs/kernel-cases.md (--results:
|
||||
every result as JSON too)
|
||||
--ladder [--record REASON]: the complexity ladder instead
|
||||
(tests/kernel-cases/ladder/ladder.json, tiers in order), graded and compared
|
||||
with its timing golden, into docs/kernel-ladder.md; --record re-measures the
|
||||
golden one case at a time and re-sorts each tier by time
|
||||
action-matrix [--bless] [--results FILE] [--from DIR]
|
||||
try every ordinary action on every face, edge, vertex and body of the design
|
||||
corpus and compare with tests/action_matrix/baseline.json (--bless rewrites it;
|
||||
|
||||
Reference in New Issue
Block a user