diff --git a/tests/kernel-cases/README.md b/tests/kernel-cases/README.md index cef8d505b..2c3a95c0d 100644 --- a/tests/kernel-cases/README.md +++ b/tests/kernel-cases/README.md @@ -107,7 +107,7 @@ cargo xtask kernel-cases --ladder --kernel both --cli target/release/solidcraft- cargo xtask kernel-cases --ladder --kernel both --cli target/release/solidcraft-cli --record "why" ``` -`--record` measures one case at a time, rewrites the golden and sorts each tier by the first +Ladder runs measure one case at a time (unless `--jobs`); `--record` rewrites the golden and sorts each tier by the first kernel's time, fastest first; commit both files. The report, `docs/kernel-ladder.md`, gives per tier and kernel the counts per status and the **frontier**: the first case in ladder order the kernel fails. diff --git a/xtask/src/kernel_ladder.rs b/xtask/src/kernel_ladder.rs index 97f23a211..39fa07243 100644 --- a/xtask/src/kernel_ladder.rs +++ b/xtask/src/kernel_ladder.rs @@ -5,8 +5,8 @@ //! areas). Each case is a test like any rig case, graded against its own `expected.json`. On top, //! `tests/kernel-cases/ladder/golden.json` records per case and kernel the status, the commands' //! time and what the kernel made (counts, volume, area): the run compares with it and flags -//! slowdowns and changes. `--record REASON` re-measures the golden (one case at a time unless -//! `--jobs` says otherwise, so the times are not shared with other cases) and re-sorts each tier +//! slowdowns and changes; cases run one at a time unless `--jobs` says otherwise, so the times are +//! not shared with other cases. `--record REASON` re-measures the golden and re-sorts each tier //! by the first kernel's time, fastest first; both files are committed. //! //! The report, `docs/kernel-ladder.md`, gives per tier and kernel the counts per status and the @@ -264,7 +264,9 @@ pub fn run(rest: &[&str]) -> Result<(), String> { Some(Some(r)) => Some(r.to_string()), }; let mut o = options(rest)?; - if reason.is_some() && !rest.contains(&"--jobs") { + // Times are compared with the golden: cases run one at a time unless --jobs says otherwise, + // so they don't share the machine with each other. + if !rest.contains(&"--jobs") { o.jobs = 1; } let manifest = read(&manifest_path()); diff --git a/xtask/src/main.rs b/xtask/src/main.rs index ee0c5926f..3784bb785 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -54,8 +54,8 @@ commands: every result as JSON too) --ladder [--record REASON]: the complexity ladder instead (tests/kernel-cases/ladder/ladder.json, tiers in order), graded and compared - with its timing golden, into docs/kernel-ladder.md; --record re-measures the - golden one case at a time and re-sorts each tier by time + with its timing golden (one case at a time unless --jobs), into + docs/kernel-ladder.md; --record re-measures the golden and re-sorts each tier by time action-matrix [--bless] [--results FILE] [--from DIR] try every ordinary action on every face, edge, vertex and body of the design corpus and compare with tests/action_matrix/baseline.json (--bless rewrites it;