masi/solvecraft
1
0
Fork 0
mirror of https://github.com/bherbruck/solidcraft.git synced 2026-11-08 04:52:10 +00:00
Code Issues Packages Projects Releases 1 Wiki Activity

Merge pull request #773 from fdalmaso/fix/448-reference-platform

Browse Source 
fix: skip architecture-dependent C++ comparisons on non-x86-64 (#448)
This commit is contained in:
bherbruck
2026-10-25 14:11:16 -04:00
committed by GitHub
parent cfd42044fa 1d5e32002d
commit 4fe6f4d516
15 changed files with 447 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/difftest/README.md
+21
View File
@@ -54,6 +54,27 @@ PASS|PARTIAL|FAIL <unit> cases=… passed=… failed=… deferred=… max_rel_er
```
then the first few failing cases and `reports/difftest/<unit>.json`.
**Native reference restrictions ([#448](https://github.com/tangency-labs/solidcore/issues/448)).**
`geom.hash_table`, `heal.heal_data_p1`, `heal.heal_data_p2`, `heal.heal_data_p3`
and `surface.twist_volume` require an x86-64 C++ reference. Their out-of-range
floating-point-to-integer conversions are undefined upstream; ARM64 saturates
where the pinned x86-64 reference wraps. Rust deliberately reproduces the latter
on every target. On other architectures these units print `SKIP ...
skipped(platform)` (or `FUZZ-SKIP`), before building fixtures or calling C++.
The twist unit is included because its period reduction has the same dependency,
even though the current committed cases do not expose it.
Skipped units are counted separately from passes, partials and failures. Their
JSON reports have `status: "skipped(platform)"`, `pass: false`, a reason in
`skipped_platform`, and no compared cases. `--status-of` prints `skipped(platform)`;
ordinary comparison and fuzz runs exit successfully when they contain only passes,
partials and platform skips. `--baseline` fails if any selected reference is
unsupported, and skipped reports are invalid baselines: this cannot waive the CI
coverage ratchet. Plan statuses and tolerances are unchanged. Rust-only
`--digest-dir` runs remain enabled on every target. On Apple Silicon, an x86-64
harness and difftest binary under Rosetta can compare these units; this does not
address independent libc++ reader or libm differences.
`with_values` / `no_values` split the passing cases into those that compared
at least one float, integer or topology entry and those that compared nothing
(both sides returned the same error, or empty outputs). `vacuous` counts the
crates/difftest/src/baseline.rs
+1
View File
@@ -755,6 +755,7 @@ mod tests {
unit: "a.b".into(),
cases,
harness_error: None,
skipped_platform: None,
abs_floors: Vec::new(),
seeds: 16,
timing: crate::runner::UnitTiming::default(),
crates/difftest/src/fuzz.rs
+17 -2
View File
@@ -635,6 +635,8 @@ pub struct FuzzResult {
pub notes: Vec<Failure>,
/// Failure to build or run the cases.
pub harness_error: Option<String>,
/// Reason the native reference is unsupported; no mutations or workers ran.
pub skipped_platform: Option<String>,
}
impl FuzzResult {
@@ -643,15 +645,21 @@ impl FuzzResult {
self.verdicts.iter().filter(|&&x| x == v).count()
}
/// No failure and no harness error.
/// At least one verdict, no failure, no harness error and no platform skip.
pub fn pass(&self) -> bool {
self.harness_error.is_none() && self.failures.is_empty() && !self.verdicts.is_empty()
self.skipped_platform.is_none()
&& self.harness_error.is_none()
&& self.failures.is_empty()
&& !self.verdicts.is_empty()
}
}
/// One-line summary, e.g. `FUZZ-PASS harness.selftest cases=300 rejected=81 malformed=40
/// ran=179 differs=2 crashes=0 panics=0 timeouts=0` (`ran` includes `differs`).
pub fn summary_line(r: &FuzzResult) -> String {
if let Some(reason) = &r.skipped_platform {
return format!("FUZZ-SKIP {} skipped(platform): {reason}", r.unit);
}
let mut s = format!(
"{} {} cases={} rejected={} malformed={} ran={} differs={} crashes={} panics={} timeouts={}",
if r.pass() { "FUZZ-PASS" } else { "FUZZ-FAIL" },
@@ -732,8 +740,15 @@ fn clear_old(dir: &Path) {
pub fn run_fuzz(unit: &dyn UnitDiff, ctx: &CaseContext, opts: &FuzzOptions) -> FuzzResult {
let mut res = FuzzResult {
unit: unit.id().to_string(),
skipped_platform: unit
.reference_platform()
.and_then(|p| p.skip_reason(std::env::consts::ARCH))
.map(str::to_string),
..FuzzResult::default()
};
if res.skipped_platform.is_some() {
return res;
}
if let Err(e) = fuzz_unit(unit, ctx, opts, &mut res) {
res.harness_error = Some(e.0);
}
crates/difftest/src/jobs.rs
+61 -24
View File
@@ -161,10 +161,8 @@ pub struct UnitRun {
pub stdout: String,
/// The child's stderr, forwarded as is.
pub stderr: String,
/// The unit counts as failed.
pub failed: bool,
/// The unit counts as partial.
pub partial: bool,
/// Comparison outcome, including unsupported native references.
pub status: crate::UnitStatus,
/// The child hit a setup error (exit status 2), e.g. an unwritable report.
pub setup_error: bool,
}
@@ -181,14 +179,20 @@ fn exit_detail(status: ExitStatus) -> String {
format!("exit code {}", status.code().unwrap_or(-1))
}
/// Parses `difftest: 1 unit(s), P passed, Q partial, F failed[...]` into `(Q, F)`.
fn parse_summary(line: &str) -> Option<(usize, usize)> {
/// Parses the aggregate counts `(partial, failed, skipped)` of one child.
fn parse_summary(line: &str) -> Option<(usize, usize, usize)> {
let rest = line.strip_prefix("difftest: 1 unit(s), ")?;
let mut parts = rest.split(", ");
let _passed = parts.next()?.strip_suffix(" passed")?;
let partial = parts.next()?.strip_suffix(" partial")?.parse().ok()?;
let failed = parts.next()?.split(' ').next()?.parse().ok()?;
Some((partial, failed))
let skipped = parts
.find_map(|p| p.strip_suffix(" skipped(platform)"))
.map(str::parse)
.transpose()
.ok()?
.unwrap_or(0);
Some((partial, failed, skipped))
}
/// Turns a finished child's output into what the parent prints and counts. A child that
@@ -197,7 +201,7 @@ pub fn interpret(id: &str, status: Option<ExitStatus>, stdout: &str, stderr: &st
let code = status.and_then(|s| s.code());
let body = stdout.trim_end_matches('\n');
let (block, last) = body.rsplit_once('\n').unwrap_or(("", body));
if let (Some(c @ 0..=2), Some((partial, failed))) = (code, parse_summary(last)) {
if let (Some(c @ 0..=2), Some((partial, failed, skipped))) = (code, parse_summary(last)) {
let mut out = block.to_string();
if !out.is_empty() {
out.push('\n');
@@ -205,8 +209,15 @@ pub fn interpret(id: &str, status: Option<ExitStatus>, stdout: &str, stderr: &st
return UnitRun {
stdout: out,
stderr: stderr.to_string(),
failed: failed > 0,
partial: partial > 0,
status: if failed > 0 {
crate::UnitStatus::Fail
} else if partial > 0 {
crate::UnitStatus::Partial
} else if skipped > 0 {
crate::UnitStatus::SkippedPlatform
} else {
crate::UnitStatus::Green
},
setup_error: c == 2,
};
}
@@ -226,8 +237,7 @@ pub fn interpret(id: &str, status: Option<ExitStatus>, stdout: &str, stderr: &st
UnitRun {
stdout: out,
stderr: String::new(),
failed: true,
partial: false,
status: crate::UnitStatus::Fail,
setup_error: false,
}
}
@@ -317,6 +327,7 @@ pub fn run_parallel(
let mut printed = 0usize;
let mut pool = Pool(Vec::new());
let (mut failed, mut partial, mut setup_error) = (0usize, 0usize, false);
let mut skipped = 0usize;
let stdout = std::io::stdout();
while printed < units.len() {
while pool.0.len() < jobs.max(1) {
@@ -361,8 +372,9 @@ pub fn run_parallel(
let _ = so.flush();
drop(so);
let _ = std::io::stderr().write_all(r.stderr.as_bytes());
failed += usize::from(r.failed);
partial += usize::from(r.partial && !r.failed);
failed += usize::from(r.status == crate::UnitStatus::Fail);
partial += usize::from(r.status == crate::UnitStatus::Partial);
skipped += usize::from(r.status == crate::UnitStatus::SkippedPlatform);
setup_error |= r.setup_error;
printed += 1;
}
@@ -371,9 +383,8 @@ pub fn run_parallel(
}
}
println!(
"difftest: {} unit(s), {} passed, {partial} partial, {failed} failed",
units.len(),
units.len() - failed - partial
"{}",
crate::report::run_summary(units.len(), partial, failed, skipped)
);
// Timing goes to stderr so stdout stays identical to a serial run.
let sum: Duration = times.iter().sum();
@@ -467,17 +478,42 @@ mod tests {
#[test]
fn summary_lines() {
assert_eq!(
parse_summary(
"difftest: 1 unit(s), 0 passed, 0 partial, 0 failed, 1 skipped(platform)"
),
Some((0, 0, 1))
);
assert_eq!(
parse_summary("difftest: 1 unit(s), 0 passed, 1 partial, 0 failed"),
Some((1, 0))
Some((1, 0, 0))
);
assert_eq!(
parse_summary("difftest: 1 unit(s), 0 passed, 0 partial, 1 failed, 1 dropped"),
Some((0, 1))
Some((0, 1, 0))
);
assert_eq!(parse_summary("PASS a cases=1"), None);
}
#[test]
#[cfg(unix)]
fn skipped_child_is_neither_a_pass_nor_a_failure() {
use std::os::unix::process::ExitStatusExt;
let summary = crate::report::run_summary(1, 0, 0, 1);
let stdout = format!("SKIP u skipped(platform): unsupported reference\n{summary}\n");
let r = interpret("u", Some(ExitStatus::from_raw(0)), &stdout, "");
assert_eq!(r.status, crate::UnitStatus::SkippedPlatform);
assert!(!r.setup_error);
assert_eq!(
r.stdout,
"SKIP u skipped(platform): unsupported reference\n"
);
assert_eq!(
crate::report::run_summary(3, 1, 0, 1),
"difftest: 3 unit(s), 1 passed, 1 partial, 0 failed, 1 skipped(platform)"
);
}
#[cfg(unix)]
#[test]
fn interprets_children() {
@@ -490,31 +526,32 @@ mod tests {
"",
);
assert_eq!(r.stdout, "PASS u cases=1\n report: r/u.json\n");
assert!(!r.failed && !r.partial && !r.setup_error);
assert_eq!(r.status, crate::UnitStatus::Green);
assert!(!r.setup_error);
let partial = interpret(
"u",
Some(ok),
"PARTIAL u\ndifftest: 1 unit(s), 0 passed, 1 partial, 0 failed\n",
"",
);
assert!(partial.partial && !partial.failed);
assert_eq!(partial.status, crate::UnitStatus::Partial);
let fail = interpret(
"u",
Some(ExitStatus::from_raw(1 << 8)),
"FAIL u\ndifftest: 1 unit(s), 0 passed, 0 partial, 1 failed\n",
"",
);
assert!(fail.failed);
assert_eq!(fail.status, crate::UnitStatus::Fail);
// SIGSEGV
let crash = interpret("u", Some(ExitStatus::from_raw(11)), "", "a\nboom\n");
assert!(crash.failed);
assert_eq!(crash.status, crate::UnitStatus::Fail);
assert_eq!(
crash.stdout,
"FAIL u (worker process killed by signal 11)\n stderr: a\n stderr: boom\n"
);
// A clean exit without the summary line is a crash too.
let cut = interpret("u", Some(ok), "PASS u\n", "");
assert!(cut.failed);
assert_eq!(cut.status, crate::UnitStatus::Fail);
assert!(cut
.stdout
.starts_with("PASS u\nFAIL u (worker process exit code 0)"));
crates/difftest/src/lib.rs
+1
View File
@@ -19,6 +19,7 @@ pub mod json;
pub mod known;
pub mod output;
pub mod plan;
pub mod platform;
pub mod report;
pub mod rng;
pub mod runner;
crates/difftest/src/main.rs
+61 -21
View File
@@ -267,9 +267,7 @@ fn main() -> ExitCode {
let res = run_unit(unit.as_ref(), &ctx, &opts);
// The value-share floor is part of the gate (`--green` enforces it too).
match baseline::value_share(&res) {
Some(v)
if res.status() != UnitStatus::Fail && waivers.covering(&v).is_none() =>
{
Some(v) if res.pass() && waivers.covering(&v).is_none() => {
eprintln!("difftest: {id}: {}", v.detail);
UnitStatus::Fail
}
@@ -395,6 +393,7 @@ fn main() -> ExitCode {
let mut failed = 0usize;
let mut partial = 0usize;
let mut skipped = 0usize;
for id in &wanted {
let Some(unit) = reg.iter().find(|u| u.id() == id.as_str()) else {
println!("FAIL {id} (no difftest runner registered for this unit)");
@@ -402,7 +401,7 @@ fn main() -> ExitCode {
continue;
};
let res = run_unit(unit.as_ref(), &ctx, &opts);
if args.perturb {
if args.perturb && res.status() != UnitStatus::SkippedPlatform {
let (n, caught) = res.perturb_stats();
let ok = n > 0 && n == caught;
let unperturbed = res
@@ -470,18 +469,12 @@ fn main() -> ExitCode {
.iter()
.find(|p| p.id == id.as_str())
.map(|p| p.status.as_str());
let mut unit_failed = match res.status() {
UnitStatus::Fail => true,
UnitStatus::Partial if plan_status == Some("green") => {
// Green units cannot contain deferred or known mismatches.
println!(
" FAIL {id}: plan status is green but cases are deferred or known mismatches"
);
true
}
UnitStatus::Partial | UnitStatus::Green => false,
};
if !unit_failed {
let gate_error = selection_gate(res.status(), plan_status, args.baseline.is_some());
if let Some(reason) = gate_error {
println!(" FAIL {id}: {reason}");
}
let mut unit_failed = res.status() == UnitStatus::Fail || gate_error.is_some();
if !unit_failed && res.status() != UnitStatus::SkippedPlatform {
let mut violations: Vec<Violation> = baseline::value_share(&res).into_iter().collect();
if let (Some(dir), None) = (&args.baseline, &args.filter) {
match baseline::load_baseline(dir, id) {
@@ -499,8 +492,10 @@ fn main() -> ExitCode {
failed += 1;
} else if res.status() == UnitStatus::Partial {
partial += 1;
} else if res.status() == UnitStatus::SkippedPlatform {
skipped += 1;
}
if !args.no_report {
if !args.no_report && !args.perturb {
match report::write_unit_report(&report_dir, &res) {
Ok(p) => println!(
" report: {}",
@@ -519,9 +514,8 @@ fn main() -> ExitCode {
String::new()
};
println!(
"difftest: {} unit(s), {} passed, {partial} partial, {failed} failed{dropped_note}",
wanted.len(),
wanted.len() - failed - partial
"{}{dropped_note}",
report::run_summary(wanted.len(), partial, failed, skipped)
);
if setup_error {
ExitCode::from(2)
@@ -532,6 +526,24 @@ fn main() -> ExitCode {
}
}
/// Selection constraints are not case-level waivers. An unsupported oracle cannot
/// satisfy an exact-base run, including when that baseline has no earlier unit report.
fn selection_gate(
status: UnitStatus,
plan_status: Option<&str>,
has_baseline: bool,
) -> Option<&'static str> {
match status {
UnitStatus::Partial if plan_status == Some("green") => {
Some("plan status is green but cases are deferred or known mismatches")
}
UnitStatus::SkippedPlatform if has_baseline => {
Some("skipped reference cannot validate an exact-base comparison")
}
_ => None,
}
}
/// Prints gate violations (waived ones with their reason); true if any is not waived.
/// `--compare-reports DIR --baseline BASE`: the regression ratchet applied to the finished
/// reports of a sweep run elsewhere (CI runs the sweep while the exact-base artifact is
@@ -724,6 +736,7 @@ fn fuzz_main(args: &Args, ctx: &CaseContext, root: &Path) -> ExitCode {
let mut seen = std::collections::BTreeSet::new();
wanted.retain(|w| seen.insert(w.clone()));
let mut failed = 0usize;
let mut skipped = 0usize;
for id in &wanted {
let Some(unit) = reg.iter().find(|u| u.id() == id.as_str()) else {
println!("FUZZ-FAIL {id} (no difftest runner registered for this unit)");
@@ -732,6 +745,10 @@ fn fuzz_main(args: &Args, ctx: &CaseContext, root: &Path) -> ExitCode {
};
let res = fuzz::run_fuzz(unit.as_ref(), ctx, &opts);
println!("{}", fuzz::summary_line(&res));
if res.skipped_platform.is_some() {
skipped += 1;
continue;
}
for f in res.failures.iter().take(MAX_FUZZ_LINES) {
let file = f.file.as_deref().map_or_else(String::new, |p| {
format!(" -> {}", p.strip_prefix(root).unwrap_or(p).display())
@@ -757,8 +774,13 @@ fn fuzz_main(args: &Args, ctx: &CaseContext, root: &Path) -> ExitCode {
failed += 1;
}
}
let skipped_note = if skipped > 0 {
format!(", {skipped} skipped(platform)")
} else {
String::new()
};
println!(
"difftest --fuzz: {} unit(s), {failed} failed, {:.1} s",
"difftest --fuzz: {} unit(s), {failed} failed, {:.1} s{skipped_note}",
wanted.len(),
start.elapsed().as_secs_f64()
);
@@ -768,3 +790,21 @@ fn fuzz_main(args: &Args, ctx: &CaseContext, root: &Path) -> ExitCode {
ExitCode::SUCCESS
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn platform_skips_are_not_exact_base_comparisons() {
for plan in [None, Some("green"), Some("partial")] {
assert!(selection_gate(UnitStatus::SkippedPlatform, plan, false).is_none());
assert!(selection_gate(UnitStatus::SkippedPlatform, plan, true).is_some());
assert!(selection_gate(UnitStatus::Green, plan, true).is_none());
}
// A real deferral in a green unit must still fail, on every platform.
assert!(selection_gate(UnitStatus::Partial, Some("green"), false).is_some());
assert!(selection_gate(UnitStatus::Partial, Some("green"), true).is_some());
assert!(selection_gate(UnitStatus::Partial, Some("partial"), true).is_none());
}
}
crates/difftest/src/platform.rs
+185
View File
@@ -0,0 +1,185 @@
//! Platform requirements of the native reference, independent of Rust portability.
//!
//! Some pinned C++ operations convert out-of-range floats to integers (undefined
//! behavior). The Rust port deliberately reproduces the x86-64 reference on every
//! target. A different native instruction cannot act as its parity oracle (#448).
/// A documented architecture requirement of a unit's C++ reference.
#[derive(Debug, Clone, Copy)]
pub struct ReferencePlatform {
/// Architecture name, using Rust's `std::env::consts::ARCH` spelling.
pub arch: &'static str,
/// Why other architectures cannot supply this unit's reference results.
pub reason: &'static str,
}
impl ReferencePlatform {
/// Reason to skip comparison on `arch`; `None` when it is supported.
pub fn skip_reason(self, arch: &str) -> Option<&'static str> {
(arch != self.arch).then_some(self.reason)
}
}
/// Native voxel mapping and edge-length histograms depend on x86-64 float casts.
pub const X86_64_HASH_CASTS: ReferencePlatform = ReferencePlatform {
arch: "x86_64",
reason: "C++ float-to-ULONG conversions require x86-64 (UB-35/UB-194, #448)",
};
/// Native twist-period reduction uses an out-of-range float-to-LONG conversion.
pub const X86_64_TWIST_CASTS: ReferencePlatform = ReferencePlatform {
arch: "x86_64",
reason: "C++ float-to-LONG conversion requires x86-64 (TwistVolume::EvaluateBoundingBoxSimple, #448)",
};
#[cfg(test)]
mod tests {
use super::*;
use std::cell::Cell;
use crate::{Case, CaseContext, CaseSource, HarnessError, Output, RunOptions, UnitDiff};
#[test]
fn reference_requirements_are_architecture_specific() {
for platform in [X86_64_HASH_CASTS, X86_64_TWIST_CASTS] {
assert_eq!(platform.skip_reason("x86_64"), None);
for arch in ["aarch64", "arm", "x86", "wasm32"] {
assert_eq!(platform.skip_reason(arch), Some(platform.reason));
}
}
}
#[derive(Default)]
struct Restricted {
cases: Cell<usize>,
rust: Cell<usize>,
}
impl UnitDiff for Restricted {
fn id(&self) -> &'static str {
"test.platform"
}
fn reference_platform(&self) -> Option<ReferencePlatform> {
Some(ReferencePlatform {
arch: "unsupported-test-architecture",
reason: "test reference is unavailable",
})
}
fn cases(&self, _: &CaseContext) -> Result<Vec<Case>, HarnessError> {
self.cases.set(self.cases.get() + 1);
Ok(vec![Case {
unit: self.id().into(),
name: "value".into(),
source: CaseSource::Inline,
inputs: serde_json::Value::Null,
}])
}
fn run_cpp(&self, _: &Case) -> Result<Output, HarnessError> {
panic!("unsupported C++ must never run")
}
fn run_rust(&self, _: &Case) -> Result<Output, HarnessError> {
self.rust.set(self.rust.get() + 1);
Ok(Output::new().scalar("value", 42.0))
}
}
#[test]
fn unsupported_comparisons_skip_before_fixture_construction_and_never_pass() {
let unit = Restricted::default();
let ctx = CaseContext {
root: crate::case::repo_root(),
seeds: 16,
};
for perturb in [false, true] {
let r = crate::run_unit(
&unit,
&ctx,
&RunOptions {
perturb,
..RunOptions::default()
},
);
assert_eq!(r.status(), crate::UnitStatus::SkippedPlatform);
assert!(!r.pass());
assert!(r.cases.is_empty());
assert_eq!((unit.cases.get(), unit.rust.get()), (0, 0));
let report = crate::report::unit_report(&r);
assert_eq!(report["status"], "skipped(platform)");
assert_eq!(report["pass"], false);
assert_eq!(report["cases_passed"], 0);
assert_eq!(report["skipped_platform"], "test reference is unavailable");
assert!(crate::report::summary_line(&r)
.starts_with("SKIP test.platform skipped(platform):"));
// A platform-only report must never replace a trusted CI baseline.
assert!(crate::baseline::parse_baseline(&report.to_string()).is_err());
let baseline = crate::baseline::Baseline {
unit: unit.id().into(),
status: Some("green".into()),
seeds: Some(16),
cases: vec![crate::baseline::BaselineCase {
name: "value".into(),
outcome: "pass".into(),
generated: false,
compared_values: true,
}],
};
assert!(!crate::baseline::ratchet(&baseline, &r).is_empty());
}
}
#[test]
fn fuzz_skips_without_workers_but_rust_digests_still_run() {
let unit = Restricted::default();
let ctx = CaseContext {
root: crate::case::repo_root(),
seeds: 16,
};
let r = crate::fuzz::run_fuzz(
&unit,
&ctx,
&crate::fuzz::FuzzOptions {
cases: 10,
out_dir: ctx.root.join("bin/platform-test-unused"),
filter: None,
},
);
assert!(r.skipped_platform.is_some());
assert!(!r.pass());
assert!(r.verdicts.is_empty());
assert_eq!((unit.cases.get(), unit.rust.get()), (0, 0));
assert!(crate::fuzz::summary_line(&r).starts_with("FUZZ-SKIP test.platform"));
let digest = crate::runner::digest_unit(&unit, &ctx, &RunOptions::default(), false);
assert!(digest.harness_error.is_none());
assert_eq!(digest.lines.len(), 1);
assert_eq!((unit.cases.get(), unit.rust.get()), (1, 1));
}
#[test]
#[cfg(feature = "cpp")]
fn registry_marks_only_the_documented_units() {
let marked: Vec<_> = crate::registry()
.iter()
.filter_map(|u| {
u.reference_platform().map(|p| {
assert_eq!(p.arch, "x86_64");
u.id()
})
})
.collect();
assert_eq!(
marked,
[
"geom.hash_table",
"heal.heal_data_p1",
"heal.heal_data_p2",
"heal.heal_data_p3",
"surface.twist_volume"
]
);
}
}
crates/difftest/src/report.rs
+28 -2
View File
@@ -87,6 +87,9 @@ pub fn unit_report(r: &UnitResult) -> Value {
m.insert("unit".into(), json!(r.unit));
m.insert("pass".into(), json!(r.pass()));
m.insert("status".into(), json!(r.status().as_str()));
if let Some(reason) = &r.skipped_platform {
m.insert("skipped_platform".into(), json!(reason));
}
m.insert("rel_tol".into(), num(REL_TOL));
m.insert("abs_floors".into(), Value::Array(floors));
m.insert("seeds".into(), json!(r.seeds));
@@ -134,7 +137,7 @@ pub fn write_unit_report(dir: &Path, r: &UnitResult) -> Result<PathBuf, HarnessE
/// Aggregate counts recorded in a report (including vacuous and known outcomes).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ReportCounts {
/// `green`, `partial` or `fail`.
/// `green`, `partial`, `fail` or `skipped(platform)`.
pub status: String,
/// All cases.
pub total: u64,
@@ -200,14 +203,35 @@ impl ReportCounts {
}
}
/// Aggregate CLI summary. Skipped units never contribute to the passed count.
/// The optional suffix preserves existing output when all references are supported.
pub fn run_summary(total: usize, partial: usize, failed: usize, skipped: usize) -> String {
let mut s = format!(
"difftest: {total} unit(s), {} passed, {partial} partial, {failed} failed",
total.saturating_sub(partial + failed + skipped)
);
if skipped > 0 {
let _ = write!(s, ", {skipped} skipped(platform)");
}
s
}
/// One-line summary, e.g.
/// `PASS harness.selftest cases=40 passed=40 failed=0 deferred=0 max_rel_err=1.1e-16 with_values=36 no_values=4 vacuous=1`.
/// The status word is `PASS` (green), `PARTIAL` or `FAIL`; `with_values` / `no_values`
/// The status word is `PASS` (green), `PARTIAL`, `FAIL` or `SKIP`; a skip prints
/// only the unit and its platform reason. Otherwise, `with_values` / `no_values`
/// split the passing cases into those that compared at least one value and those that
/// compared nothing (the same error on both sides, or empty outputs); `vacuous` counts
/// the `no_values` cases where both sides succeeded, so only the status agreed; with deferred
/// cases the line ends with ` blocked_by=<unit>(<cases>),...`.
pub fn summary_line(r: &UnitResult) -> String {
if r.status() == UnitStatus::SkippedPlatform {
return format!(
"SKIP {} skipped(platform): {}",
r.unit,
r.skipped_platform.as_deref().unwrap_or_default()
);
}
let with_values = r.passed_with_values();
let mut s = format!(
"{} {} cases={} passed={} failed={} deferred={} max_rel_err={:e} with_values={with_values} no_values={} vacuous={}",
@@ -215,6 +239,7 @@ pub fn summary_line(r: &UnitResult) -> String {
UnitStatus::Green => "PASS",
UnitStatus::Partial => "PARTIAL",
UnitStatus::Fail => "FAIL",
UnitStatus::SkippedPlatform => "SKIP",
},
r.unit,
r.cases.len(),
@@ -262,6 +287,7 @@ mod tests {
unit: "a.b".into(),
cases: Vec::new(),
harness_error: None,
skipped_platform: None,
abs_floors: Vec::new(),
seeds: 16,
timing: crate::runner::UnitTiming::default(),
crates/difftest/src/runner.rs
+29 -9
View File
@@ -18,6 +18,13 @@ pub trait UnitDiff {
/// Unit id as in `plan/units/<id>.toml` (e.g. `nlib.knots`).
fn id(&self) -> &'static str;
/// Architecture restriction of the C++ oracle, if any. This does not restrict
/// Rust-only digest runs. Unsupported comparisons are skipped before fixtures
/// are constructed and never count as parity passes.
fn reference_platform(&self) -> Option<crate::platform::ReferencePlatform> {
None
}
/// Runs the upstream C++ (via `solidcore_ffi::units::*`).
/// Return `Ok(Output::err(..))` when the *operation* fails and `Err` only
/// for harness problems (bad inputs); `Err` always fails the case. Native fixture
@@ -224,15 +231,18 @@ pub enum UnitStatus {
Partial,
/// Any failing case, a harness error, no cases, or only rejected fixtures.
Fail,
/// The C++ reference is unsupported on this architecture; no cases ran.
SkippedPlatform,
}
impl UnitStatus {
/// `green`, `partial` or `fail`.
/// `green`, `partial`, `fail` or `skipped(platform)`.
pub fn as_str(self) -> &'static str {
match self {
Self::Green => "green",
Self::Partial => "partial",
Self::Fail => "fail",
Self::SkippedPlatform => "skipped(platform)",
}
}
}
@@ -246,6 +256,8 @@ pub struct UnitResult {
pub cases: Vec<CaseResult>,
/// Failure to build the case list.
pub harness_error: Option<String>,
/// Reason the C++ reference cannot run on this architecture (no cases ran).
pub skipped_platform: Option<String>,
/// Documented absolute floors used.
pub abs_floors: Vec<AbsFloor>,
/// `--seeds` value used.
@@ -307,13 +319,13 @@ impl UnitTiming {
}
impl UnitResult {
/// Green, partial or fail (see [`UnitStatus`]).
/// Comparison status, including an unsupported reference (see [`UnitStatus`]).
pub fn status(&self) -> UnitStatus {
if self.harness_error.is_some()
|| self.cases.is_empty()
|| self.failed() > 0
|| self.cases.len() == self.rejected()
{
if self.harness_error.is_some() || self.failed() > 0 {
UnitStatus::Fail
} else if self.skipped_platform.is_some() {
UnitStatus::SkippedPlatform
} else if self.cases.is_empty() || self.cases.len() == self.rejected() {
UnitStatus::Fail
} else if self.deferred() > 0 || self.known_mismatches() > 0 {
UnitStatus::Partial
@@ -324,9 +336,9 @@ impl UnitResult {
}
}
/// No failure: the unit is green or partial (the exit status is 0).
/// Parity was checked: the unit is green or partial. Skipped units do not pass.
pub fn pass(&self) -> bool {
self.status() != UnitStatus::Fail
matches!(self.status(), UnitStatus::Green | UnitStatus::Partial)
}
/// Number of passing cases.
@@ -489,10 +501,17 @@ pub fn run_unit(unit: &dyn UnitDiff, ctx: &CaseContext, opts: &RunOptions) -> Un
unit: unit.id().to_string(),
cases: Vec::new(),
harness_error: None,
skipped_platform: unit
.reference_platform()
.and_then(|p| p.skip_reason(std::env::consts::ARCH))
.map(str::to_string),
abs_floors: unit.abs_floors().to_vec(),
seeds: ctx.seeds,
timing: UnitTiming::default(),
};
if res.skipped_platform.is_some() {
return res;
}
let digest = std::env::var_os("DIFFTEST_RUST_DIGEST").is_some();
let t0 = Instant::now();
let cases = unit.cases(ctx);
@@ -799,6 +818,7 @@ mod tests {
unit: "u".into(),
cases: vec![case(None, true), case(d(), true)],
harness_error: None,
skipped_platform: None,
abs_floors: Vec::new(),
seeds: 0,
timing: UnitTiming::default(),
crates/difftest/src/units/geom_hash_table.rs
+4
View File
@@ -637,6 +637,10 @@ fn apply(m: &mut Model, upd: &mut Vec<usize>, op: &Op) -> solidcore::Result<()>
}
impl UnitDiff for Runner {
fn reference_platform(&self) -> Option<crate::platform::ReferencePlatform> {
Some(crate::platform::X86_64_HASH_CASTS)
}
fn id(&self) -> &'static str {
"geom.hash_table"
}
crates/difftest/src/units/heal_heal_data_p1.rs
+4
View File
@@ -348,6 +348,10 @@ fn gen_api(rng: &mut Rng) -> Inputs {
}
impl UnitDiff for Runner {
fn reference_platform(&self) -> Option<crate::platform::ReferencePlatform> {
Some(crate::platform::X86_64_HASH_CASTS)
}
fn id(&self) -> &'static str {
"heal.heal_data_p1"
}
crates/difftest/src/units/heal_heal_data_p2.rs
+4
View File
@@ -2054,6 +2054,10 @@ fn gen_prob(rng: &mut Rng) -> Inputs {
}
impl UnitDiff for Runner {
fn reference_platform(&self) -> Option<crate::platform::ReferencePlatform> {
Some(crate::platform::X86_64_HASH_CASTS)
}
fn id(&self) -> &'static str {
"heal.heal_data_p2"
}
crates/difftest/src/units/heal_heal_data_p3.rs
+4
View File
@@ -104,6 +104,10 @@ fn script(rng: &mut Rng) -> Inputs {
}
impl UnitDiff for Runner {
fn reference_platform(&self) -> Option<crate::platform::ReferencePlatform> {
Some(crate::platform::X86_64_HASH_CASTS)
}
fn id(&self) -> &'static str {
"heal.heal_data_p3"
}
crates/difftest/src/units/surface_twist_volume.rs
+4
View File
@@ -767,6 +767,10 @@ fn gen_case(rng: &mut Rng) -> Inputs {
}
impl UnitDiff for Runner {
fn reference_platform(&self) -> Option<crate::platform::ReferencePlatform> {
Some(crate::platform::X86_64_TWIST_CASTS)
}
fn id(&self) -> &'static str {
"surface.twist_volume"
}
plan/findings/general--448-x86-64-reference-requirement.md
+23
View File
@@ -0,0 +1,23 @@
# general: Issue #448: architecture requirements of the differential reference (2026-10-24)
The pinned C++ converts out-of-range floating-point values to integers in
`SmHashTable.h:1957-1959` (UB-35), `SmTArray.h:2395-2397` (UB-194), and
`SmTwistVolume.cpp:1202`. The Rust ports intentionally emulate the x86-64
conversions on every target. The ARM64 differences and macOS reproduction are
recorded in #448; this change does not alter either kernel or the reference pin.
The five runners named in #448 now declare an x86-64 reference requirement.
On other architectures, comparison and fuzz runs stop before fixture construction
with `skipped(platform)`, separate from passes, deferrals and native rejections.
Serial and parallel summaries exclude these units from their passed counts; JSON
reports have `pass: false`. Rust-only digests remain enabled. Skipped reports are
not accepted as baselines, and an exact-base comparison with an unsupported
reference fails even for a newly selected unit. No plan status, case, tolerance or
coverage waiver changed.
Validation of the skip paths uses an unsupported-reference test double, so it runs
on x86-64 as well. It checks that neither fixture construction nor C++ execution
happens, fuzz workers are not launched, Rust digests still run, skipped reports
cannot replace baselines, and previously passing cases remain ratcheted. The
architecture predicates also cover ARM64 explicitly. No fresh ARM64 native run is
claimed; this machine is Linux x86-64.