diff --git a/tests/kernel-cases/EXEMPT b/tests/kernel-cases/EXEMPT new file mode 100644 index 0000000000..5bd7ed2b39 --- /dev/null +++ b/tests/kernel-cases/EXEMPT @@ -0,0 +1,399 @@ +# Registry commands with no kernel rig case (`cargo xtask kernel-cases --coverage`). Every other +# command of `solidcraft-cli commands` (planned ones aside) must be run by some recipe.json under +# tests/kernel-cases/. One line a command: ` `; `#` starts a comment. +# Each section is sorted, with no id twice. +# +# Commands with no geometry to grade: the view, rendering, appearance, materials, selection, the +# browser, files and saving, documents, diagnostics, and component, parameter and timeline +# bookkeeping. Each is covered by the engine's and the UI's own tests. + +appearance.assign appearance only: colours and textures +appearance.library appearance only: colours and textures +appearance.list appearance only: colours and textures +assembly.bom a bill of materials report +body.rename a body name +browser.collapse_all the browser tree: order, groups and folding +browser.expand_all the browser tree: order, groups and folding +browser.group the browser tree: order, groups and folding +browser.move the browser tree: order, groups and folding +browser.order the browser tree: order, groups and folding +browser.rename_group the browser tree: order, groups and folding +browser.ungroup the browser tree: order, groups and folding +canvas.calibrate reference images, and sketches traced from them (sketch only) +canvas.decal reference images, and sketches traced from them (sketch only) +canvas.delete reference images, and sketches traced from them (sketch only) +canvas.edit reference images, and sketches traced from them (sketch only) +canvas.insert reference images, and sketches traced from them (sketch only) +canvas.to_sketch reference images, and sketches traced from them (sketch only) +component.break_link component links +component.copies a listing of identical copies +component.cycle_color component colours +component.delete removes a component: no geometry built +component.links component links +component.list a listing +component.metadata component metadata +component.metadata.set component metadata +component.properties component properties +component.relink component links +component.rename a component name +component.replace_with_link component links +component.unlock component locking +component.update_links component links +contact.list a listing of contact sets +diagnostics.crash_report crash reports +diagnostics.crash_reports crash reports +diagnostics.crash_reports.clear crash reports +diagnostics.crash_test crash reports +diagnostics.profile timing: where a command's time goes +doc.open opening and recovering documents +doc.recover opening and recovering documents +doc.recovery_discard opening and recovering documents +doc.recovery_list opening and recovering documents +document.metadata document metadata and revisions +document.metadata.remove document metadata and revisions +document.metadata.set document metadata and revisions +document.revision document metadata and revisions +edit.checkpoint undo, redo and checkpoints: an earlier state restored, no new geometry +edit.redo undo, redo and checkpoints: an earlier state restored, no new geometry +edit.restore_checkpoint undo, redo and checkpoints: an earlier state restored, no new geometry +edit.undo undo, redo and checkpoints: an earlier state restored, no new geometry +engine.commands the command registry itself +engine.describe the command registry itself +explode.create exploded views: display offsets +explode.delete exploded views: display offsets +explode.list exploded views: display offsets +explode.show exploded views: display offsets +expr.evaluate expression evaluation +file.open_in_easyprint hands the file to another program +file.save saving the design +file.save_as saving the design +file.save_mesh saving a display mesh +file.send_to_slicer hands the file to a slicer +file.slicers a listing of slicers +frame.profile.list a listing of the profile library +frame.profile.properties a profile's section properties from the library +inspect.accessibility display analysis: shading +inspect.colored_components display: component colours +inspect.curvature_map display analysis: shading +inspect.environment_map display analysis: shading +inspect.silhouette display analysis: outline +inspect.zebra display analysis: shading +joint.list a listing of joints +library.apply material and appearance libraries +library.export material and appearance libraries +library.inspect material and appearance libraries +material.assign materials and their properties +material.create materials and their properties +material.delete materials and their properties +material.edit materials and their properties +material.inspect materials and their properties +material.list materials and their properties +material.properties.directional materials and their properties +material.properties.validate materials and their properties +model.edges a listing of the model's topology +model.faces a listing of the model's topology +model.threads a listing of the model's topology +motion.delete removes a motion study +motion.export writes a study's results to a file +motion.export_animation writes an animation +motion.list a listing of motion studies +motion.play plays a study in the view +occurrence.properties occurrence properties +parameters.comment parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.complete parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.delete parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.export parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.favorite parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.graph parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.import parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.list parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.move parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.rename parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parameters.users parameter bookkeeping; a value change that rebuilds is parameters.change, which cases use +parts.library a listing of the parts library +plastic.rules a listing of plastic rules +render.aging.describe rendering: images, cameras, stages +render.backends rendering: images, cameras, stages +render.camera.fit rendering: images, cameras, stages +render.camera.get rendering: images, cameras, stages +render.camera.set rendering: images, cameras, stages +render.cancel rendering: images, cameras, stages +render.effects.describe rendering: images, cameras, stages +render.export rendering: images, cameras, stages +render.hdr.inspect rendering: images, cameras, stages +render.library.appearance.apply rendering: images, cameras, stages +render.library.describe rendering: images, cameras, stages +render.library.environment.apply rendering: images, cameras, stages +render.library.list rendering: images, cameras, stages +render.library.physical.prepare rendering: images, cameras, stages +render.library.preview rendering: images, cameras, stages +render.library.studio.apply rendering: images, cameras, stages +render.library.studio.prepare rendering: images, cameras, stages +render.library.surface.bind rendering: images, cameras, stages +render.library.surface.prepare rendering: images, cameras, stages +render.library.surface.unbind rendering: images, cameras, stages +render.library.targets rendering: images, cameras, stages +render.optics.bind rendering: images, cameras, stages +render.optics.unbind rendering: images, cameras, stages +render.preflight rendering: images, cameras, stages +render.preset.apply rendering: images, cameras, stages +render.preset.resolve rendering: images, cameras, stages +render.quality.prepare rendering: images, cameras, stages +render.result.pin rendering: images, cameras, stages +render.result.regrade rendering: images, cameras, stages +render.scene rendering: images, cameras, stages +render.settings.apply rendering: images, cameras, stages +render.settings.capture_hdr rendering: images, cameras, stages +render.settings.get rendering: images, cameras, stages +render.settings.set rendering: images, cameras, stages +render.shot.duplicate rendering: images, cameras, stages +render.shot.remove rendering: images, cameras, stages +render.shot.rename rendering: images, cameras, stages +render.stage.backdrop rendering: images, cameras, stages +render.stage.clear rendering: images, cameras, stages +render.stage.ground rendering: images, cameras, stages +render.start rendering: images, cameras, stages +render.status rendering: images, cameras, stages +select.clear the selection +select.set the selection +selection.delete deletes what is selected through the commands that own each kind +sheet.rules a listing of sheet metal rules +sketch.arc.tangent 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.auto_constrain 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.auto_constrain_from_datum 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.break 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.break_link 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.cancel 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.centerline 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.chain 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.chamfer.distance_angle 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.chamfer.equal_distance 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.chamfer.two_distance 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.circle.three_point 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.circle.three_tangent 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.circle.two_point 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.circle.two_tangent 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constrainer 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.coincident 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.collinear 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.concentric 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.curvature 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.equal 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.midpoint 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.parallel 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.perpendicular 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.polygon 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.constraint.symmetry 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.construction 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.copy 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.cut 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.delete 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.dimension_text 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.drag 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.drag_end 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.draw.dimensioned 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.drawing_bounds 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.edit_text 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.export_dxf 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.extend 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.fillet 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.finish_auto_constrain 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.glyphs 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.infinite 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.insert_dxf 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.insert_svg 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.inspect 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.line.midpoint 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.loose_ends 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.loose_ends.tie 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.mirror 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.move 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.move3d 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.offset 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.offset.break 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.paste 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.pattern.rectangular 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.point3d 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.polygon.edge 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.rectangle.three_point 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.redefine 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.scale 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.slot.arc_center 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.slot.arc_three_point 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.slot.center_point 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.snap 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.solve 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.toggle_driven 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.transform 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.trim 2D sketch only (the sketch solver and engine tests); no kernel operation +sketch.trim.path 2D sketch only (the sketch solver and engine tests); no kernel operation +timeline.dependents a listing of dependents +timeline.export_script writes the history as a script +timeline.inspect a listing of the timeline +timeline.rename a feature name +timeline.rename_group a group name +view.ambient_occlusion the view: camera, layout, display +view.apply the view: camera, layout, display +view.back the view: camera, layout, display +view.bottom the view: camera, layout, display +view.delete the view: camera, layout, display +view.fit the view: camera, layout, display +view.front the view: camera, layout, display +view.graphics_preset the view: camera, layout, display +view.grid the view: camera, layout, display +view.ground_shadow the view: camera, layout, display +view.history_layout the view: camera, layout, display +view.home the view: camera, layout, display +view.isometric the view: camera, layout, display +view.layout the view: camera, layout, display +view.left the view: camera, layout, display +view.list the view: camera, layout, display +view.navigate the view: camera, layout, display +view.navigation_bar the view: camera, layout, display +view.normal_to the view: camera, layout, display +view.perspective the view: camera, layout, display +view.rename the view: camera, layout, display +view.right the view: camera, layout, display +view.save the view: camera, layout, display +view.spin the view: camera, layout, display +view.tool the view: camera, layout, display +view.top the view: camera, layout, display +view.visual_style the view: camera, layout, display + +# Commands that make or change geometry and have no case yet. This section only shrinks: +# the pull request that adds a case for one removes its line (the check fails on a listed command +# that a case uses), and a new geometry command gets a case, not a line here. +[to cover] +body.repair repair: the solid a refused or open part becomes +component.copy_bodies assemblies: bodies copied into a component +component.extract assemblies: bodies extracted to a design +component.from_bodies assemblies: a component made from bodies +component.insert_design assemblies: a design inserted +component.move_bodies assemblies: bodies moved between components +component.move_sketches assemblies: sketches moved between components +component.paste_new assemblies: a pasted new component +component.revert_position assemblies: a placement reverted +config.activate configurations: a configured part rebuilt per row +config.column configurations: a configured part rebuilt per row +config.configure configurations: a configured part rebuilt per row +config.preview configurations: a configured part rebuilt per row +config.row configurations: a configured part rebuilt per row +config.rule configurations: a configured part rebuilt per row +config.rules configurations: a configured part rebuilt per row +config.table configurations: a configured part rebuilt per row +construct.axis.edge construction: the plane, axis or point exactly +construct.axis.normal_to_face construction: the plane, axis or point exactly +construct.axis.through_cylinder construction: the plane, axis or point exactly +construct.axis.two_planes construction: the plane, axis or point exactly +construct.plane.at_angle construction: the plane, axis or point exactly +construct.plane.midplane construction: the plane, axis or point exactly +construct.plane.perpendicular construction: the plane, axis or point exactly +construct.plane.tangent construction: the plane, axis or point exactly +construct.plane.three_points construction: the plane, axis or point exactly +construct.plane.two_edges construction: the plane, axis or point exactly +construct.point.along_path construction: the plane, axis or point exactly +construct.point.center construction: the plane, axis or point exactly +construct.point.edge_plane construction: the plane, axis or point exactly +construct.point.three_planes construction: the plane, axis or point exactly +construct.point.two_edges construction: the plane, axis or point exactly +construct.point.vertex construction: the plane, axis or point exactly +contact.check assemblies: contact sets and what they report +contact.create assemblies: contact sets and what they report +contact.disable_all assemblies: contact sets and what they report +contact.edit assemblies: contact sets and what they report +contact.enable_all assemblies: contact sets and what they report +contact.enable_sets assemblies: contact sets and what they report +file.export_view_dxf drawings: the projection of the model a view DXF holds +frame.member frames: member bodies +frame.profile.insert frames: member bodies +inspect.center_of_mass inspect: the number or curves it reports +inspect.clearance inspect: the number or curves it reports +inspect.compare inspect: the number or curves it reports +inspect.continuity inspect: the number or curves it reports +inspect.curvature_comb inspect: the number or curves it reports +inspect.draft inspect: the number or curves it reports +inspect.interference inspect: the number or curves it reports +inspect.isocurve inspect: the number or curves it reports +inspect.minimum_radius inspect: the number or curves it reports +inspect.parameter_sweep inspect: results rebuilt over a parameter range +inspect.section inspect: the number or curves it reports +joint.as_built assemblies: joint placements and values +joint.cam assemblies: joint placements and values +joint.create assemblies: joint placements and values +joint.delete assemblies: joint placements and values +joint.drive assemblies: joint placements and values +joint.edit assemblies: joint placements and values +joint.gear assemblies: joint placements and values +joint.inspect assemblies: joint placements and values +joint.limits assemblies: joint placements and values +joint.motion_link assemblies: joint placements and values +joint.origin assemblies: joint placements and values +joint.rack_pinion assemblies: joint placements and values +joint.reach assemblies: joint placements and values +joint.rigid_group assemblies: joint placements and values +joint.screw assemblies: joint placements and values +joint.solve assemblies: joint placements and values +joint.sweep assemblies: joint placements and values +motion.interference assemblies: motion results +motion.plot assemblies: motion results +motion.study assemblies: motion results +motion.trace assemblies: motion results +motion.trace.edit assemblies: motion results +motion.trace.keep assemblies: motion results +occurrence.copy assemblies: an occurrence copied +occurrence.drag assemblies: an occurrence dragged to a placement +occurrence.reparent assemblies: an occurrence moved in the tree +parts.fastener standard parts: a fastener body +parts.insert standard parts: a part body +parts.resize standard parts: a part resized +plastic.assign_rule plastic: the feature body +plastic.boss plastic: the feature body +plastic.lip plastic: the feature body +plastic.manage_rules plastic: rules that rebuild the features using them +plastic.rest plastic: the feature body +sheet.export_bend_table sheet metal: the bend table of a flat pattern +sheet.export_dxf sheet metal: the flat pattern as DXF +sheet.hem sheet metal: the feature body +sheet.path_flange sheet metal: the feature body +sheet.perforate sheet metal: the feature body +sketch.arc3d sketch: a curve kind no case profiles yet +sketch.auto_project sketch from the model: curves the kernel computes from bodies +sketch.conic sketch: a curve kind no case profiles yet +sketch.fit_curves_to_section sketch from the model: curves the kernel computes from bodies +sketch.include_3d sketch from the model: curves the kernel computes from bodies +sketch.intersect sketch from the model: curves the kernel computes from bodies +sketch.intersection_curve sketch from the model: curves the kernel computes from bodies +sketch.isoparametric_curve sketch from the model: curves the kernel computes from bodies +sketch.project sketch from the model: curves the kernel computes from bodies +sketch.project_to_surface sketch from the model: curves the kernel computes from bodies +sketch.spline.control_point_5 sketch: a curve kind no case profiles yet +sketch.spline3d sketch: a curve kind no case profiles yet +sketch.spun_profile sketch from the model: curves the kernel computes from bodies +solid.align solid modelling: the body it makes +solid.boundary_fill solid modelling: the body it makes +solid.bounding_solid solid modelling: the body it makes +solid.cone solid modelling: the body it makes +solid.pattern.path solid modelling: the body it makes +solid.replace_face solid modelling: the body it makes +solid.scale solid modelling: the body it makes +solid.web solid modelling: the body it makes +surface.blend surfaces: the surface body it makes +surface.boundary surfaces: the surface body it makes +surface.fair surfaces: the surface body it makes +surface.match surfaces: the surface body it makes +surface.offset surfaces: the surface body it makes +surface.reverse_normal surfaces: the surface body it makes +surface.ruled surfaces: the surface body it makes +surface.unstitch surfaces: the surface body it makes +surface.untrim surfaces: the surface body it makes +timeline.compute_all timeline: the bodies after the rebuild +timeline.delete timeline: the bodies after the rebuild +timeline.edit timeline: the bodies after the rebuild +timeline.group timeline: the bodies after the rebuild +timeline.move_many timeline: the bodies after the rebuild +timeline.move_to_group timeline: the bodies after the rebuild +timeline.reorder timeline: the bodies after the rebuild +timeline.reorder_many timeline: the bodies after the rebuild +timeline.roll_to timeline: the bodies after the rebuild +timeline.suppress timeline: the bodies after the rebuild +timeline.ungroup timeline: the bodies after the rebuild diff --git a/xtask/src/kernel_cases.rs b/xtask/src/kernel_cases.rs index 7121ea2edd..ce3d2e9834 100644 --- a/xtask/src/kernel_cases.rs +++ b/xtask/src/kernel_cases.rs @@ -335,6 +335,10 @@ pub fn run(rest: &[&str]) -> Result<(), String> { } let o = options(rest, "target/kernel-cases")?; let dir = root().join("tests/kernel-cases"); + if rest.contains(&"--coverage") { + println!("{}", run_coverage(&o, &dir)?); + return Ok(()); + } let list = cases(&dir, o.area.as_deref())?; if list.is_empty() { return Err(format!("no cases (recipe.json + expected.json) under {}", dir.display())); @@ -410,9 +414,145 @@ fn gate(text: &str, cases: &[String], statuses: &[&str]) -> Result Result, String> { + let mut cmd = match &o.cli { + Some(bin) => Command::new(bin), + None => { + let mut c = cargo(); + c.args(["run", "-q", "--release", "-p", "solidcraft-cli", "--"]); + c + } + }; + let out = cmd.current_dir(root()).arg("commands").output().map_err(|e| format!("solidcraft-cli: {e}"))?; + if !out.status.success() { + return Err(format!("solidcraft-cli commands failed:\n{}", String::from_utf8_lossy(&out.stderr))); + } + let list: Value = serde_json::from_slice(&out.stdout).map_err(|e| format!("commands JSON: {e}"))?; + let list = list.as_array().ok_or("commands JSON: not a list")?; + Ok(list.iter().filter_map(|c| Some((c["id"].as_str()?.to_string(), c["planned"].as_str().is_some_and(|p| !p.is_empty())))).collect()) +} + +/// Every command id the command scripts (`recipe.json`) under `dir` run, at any depth. +fn used_commands(dir: &Path) -> Result, String> { + let mut out = Vec::new(); + let mut todo = vec![dir.to_path_buf()]; + while let Some(d) = todo.pop() { + for e in std::fs::read_dir(&d).map_err(|e| format!("{}: {e}", d.display()))?.flatten() { + let p = e.path(); + if p.is_dir() { + todo.push(p); + } else if p.file_name().is_some_and(|n| n == "recipe.json") { + let text = std::fs::read_to_string(&p).map_err(|e| format!("{}: {e}", p.display()))?; + out.extend(recipe_commands(&text).map_err(|e| format!("{}: {e}", p.display()))?); + } + } + } + out.sort(); + out.dedup(); + Ok(out) +} + +/// The command ids a recipe runs: each `commands[].command` of a command script. An oracle +/// recipe (no `commands`) names features, not commands, and counts none. +fn recipe_commands(text: &str) -> Result, String> { + let v: Value = serde_json::from_str(text).map_err(|e| e.to_string())?; + let Some(list) = v.get("commands") else { return Ok(Vec::new()) }; + let list = list.as_array().ok_or("`commands` is not a list")?; + list.iter() + .enumerate() + .map(|(i, c)| c["command"].as_str().map(str::to_string).ok_or_else(|| format!("commands[{i}] has no `command` id"))) + .collect() +} + +/// One line of `tests/kernel-cases/EXEMPT`: a command with no case (its reason is for readers). +#[derive(Debug, PartialEq)] +struct Exempt<'a> { + id: &'a str, + /// Listed under `[to cover]`: a geometry command still waiting for its case. + to_cover: bool, +} + +/// The section header of the commands that make geometry but have no case yet. +const TO_COVER: &str = "[to cover]"; + +/// The commands `tests/kernel-cases/EXEMPT` exempts from the coverage check: one per line, +/// ` `; `#` starts a comment line. The lines before `[to cover]` have no +/// geometry to grade; those after it make geometry and wait for a case, so that section only +/// shrinks. Each section is sorted, with no id twice in the file, and every line gives a reason. +fn exempt(text: &str) -> Result>, String> { + let mut out: Vec = Vec::new(); + let mut to_cover = false; + let mut last: Option<&str> = None; + for (n, line) in text.lines().enumerate() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + if line == TO_COVER { + if to_cover { + return Err(format!("line {}: a second {TO_COVER} section", n + 1)); + } + to_cover = true; + last = None; + continue; + } + let (id, why) = line.split_once(char::is_whitespace).map_or((line, ""), |(c, w)| (c, w.trim())); + if why.is_empty() { + return Err(format!("line {}: {id} gives no reason (` `)", n + 1)); + } + if out.iter().any(|e| e.id == id) { + return Err(format!("line {}: {id} is listed twice", n + 1)); + } + if let Some(prev) = last + && prev > id + { + return Err(format!("line {}: {id} is out of order after {prev}: keep each section sorted", n + 1)); + } + last = Some(id); + out.push(Exempt { id, to_cover }); + } + Ok(out) +} + +/// The coverage check: every built command of `registry` (id, planned) is in `used` (run by a +/// case) or exempted by `exempt_text`; an exemption names a registered command that no case +/// uses, so a case that comes removes its line. +fn coverage(registry: &[(String, bool)], used: &[String], exempt_text: &str) -> Result { + let listed = exempt(exempt_text)?; + let mut problems = Vec::new(); + for e in &listed { + if !registry.iter().any(|(id, _)| id == e.id) { + problems.push(format!("EXEMPT lists {}, which is not a registry command: remove its line", e.id)); + } else if used.iter().any(|u| u == e.id) { + problems.push(format!("EXEMPT lists {}, which a case now uses: remove its line", e.id)); + } + } + let mut covered = 0; + for (id, _) in registry.iter().filter(|(_, planned)| !planned) { + if used.iter().any(|u| u == id) { + covered += 1; + } else if !listed.iter().any(|e| e.id == id) { + problems.push(format!("NO CASE: {id} (add a case under tests/kernel-cases/, or a line to EXEMPT)")); + } + } + if !problems.is_empty() { + return Err(format!("kernel-cases coverage:\n {}", problems.join("\n "))); + } + let to_cover = listed.iter().filter(|e| e.to_cover).count(); + Ok(format!("kernel-cases coverage: {covered} commands have a case, {} have no geometry, {to_cover} still to cover", listed.len() - to_cover)) +} + +fn run_coverage(o: &Options, dir: &Path) -> Result { + let file = dir.join("EXEMPT"); + let text = std::fs::read_to_string(&file).map_err(|e| format!("{}: {e}", file.display()))?; + coverage(®istry(o)?, &used_commands(dir)?, &text).map_err(|e| format!("{}: {e}", file.display())) +} + #[cfg(test)] mod tests { - use super::gate; + use super::{Exempt, coverage, exempt, gate, recipe_commands}; fn run(text: &str, cases: &[(&str, &str)]) -> Result { let names: Vec = cases.iter().map(|(c, _)| c.to_string()).collect(); @@ -438,4 +578,43 @@ mod tests { assert!(run("a x\na y\n", &[("a", "fail")]).unwrap_err().contains("a repeat of")); assert!(run("a\n", &[("a", "fail")]).unwrap_err().contains("gives no cause")); } + + #[test] + fn a_recipe_names_the_commands_of_its_script() { + let script = r#"{"base": "../a", "commands": [{"command": "solid.box", "params": {}}, {"command": "solid.fillet"}]}"#; + assert_eq!(recipe_commands(script).unwrap(), ["solid.box", "solid.fillet"]); + assert!(recipe_commands(r#"{"id": "01-box", "features": []}"#).unwrap().is_empty()); + assert!(recipe_commands(r#"{"commands": [{"params": {}}]}"#).unwrap_err().contains("no `command` id")); + assert!(recipe_commands(r#"{"commands": {}}"#).unwrap_err().contains("not a list")); + } + + #[test] + fn the_exempt_list_has_two_sorted_sections_with_a_reason_each() { + let text = "# header\nview.fit camera only\nview.top camera only\n\n[to cover]\nsolid.cone surfaces area\n"; + assert_eq!( + exempt(text).unwrap(), + [Exempt { id: "view.fit", to_cover: false }, Exempt { id: "view.top", to_cover: false }, Exempt { id: "solid.cone", to_cover: true },] + ); + assert!(exempt("view.top x\nview.fit x\n").unwrap_err().contains("out of order")); + assert!(exempt("view.fit x\n[to cover]\nview.fit y\n").unwrap_err().contains("listed twice")); + assert!(exempt("view.fit\n").unwrap_err().contains("gives no reason")); + assert!(exempt("[to cover]\n[to cover]\n").unwrap_err().contains("second")); + } + + #[test] + fn coverage_fails_on_a_command_with_no_case_and_on_a_stale_exemption() { + let registry: Vec<(String, bool)> = + [("solid.box", false), ("solid.cone", false), ("view.fit", false), ("surface.fillet", true)].map(|(i, p)| (i.to_string(), p)).to_vec(); + let used = ["solid.box".to_string()]; + let list = "view.fit camera only\n[to cover]\nsolid.cone surfaces area\n"; + let ok = coverage(®istry, &used, list).unwrap(); + assert!(ok.contains("1 commands have a case, 1 have no geometry, 1 still to cover"), "{ok}"); + let missing = coverage(®istry, &used, "view.fit camera only\n").unwrap_err(); + assert!(missing.contains("NO CASE: solid.cone"), "{missing}"); + assert!(!missing.contains("surface.fillet"), "a planned command needs no case: {missing}"); + let covered = coverage(®istry, &["solid.box".into(), "solid.cone".into()], list).unwrap_err(); + assert!(covered.contains("EXEMPT lists solid.cone, which a case now uses"), "{covered}"); + let gone = coverage(®istry, &used, "view.fit camera only\nview.gone removed\n[to cover]\nsolid.cone x\n").unwrap_err(); + assert!(gone.contains("view.gone, which is not a registry command"), "{gone}"); + } } diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 13be4f8d77..3d13136d96 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -47,7 +47,7 @@ commands: replay the reference tests (tests/oracle/*/recipe.json), compare with each case's expected.json and write docs/oracle.md (--cli: a solidcraft-cli already built, instead of cargo run --release) - kernel-cases [--area AREA] [--cli BIN] [--timeout S] [--jobs N] [--out DIR] [--known-failing FILE] + kernel-cases [--area AREA] [--cli BIN] [--timeout S] [--jobs N] [--out DIR] [--known-failing FILE] [--coverage] the kernel regression rig: run every case in tests/kernel-cases/// on SMLib (one process each, killed after --timeout, default 180 s), grade it against the case's own answer and write docs/kernel-cases.md; DIR (default @@ -62,6 +62,9 @@ commands: docs/kernel-ladder.md (DIR default target/kernel-ladder); --record re-measures the golden and re-sorts each tier by time; --gate fails on any case whose status or counts differ from the golden, instead of the report + --coverage: instead of running cases, fail on a registry command (the CLI's + `commands`, planned ones aside) that no recipe.json under tests/kernel-cases/ + uses and tests/kernel-cases/EXEMPT does not list, and on a stale EXEMPT line action-matrix [--bless] [--results FILE] [--from DIR] try every ordinary action on every face, edge, vertex and body of the design corpus and compare with tests/action_matrix/baseline.json (--bless rewrites it;