diff --git a/skills/solidcraft-tests/SKILL.md b/skills/solidcraft-tests/SKILL.md index 63ed8d6e9..7c2ec4abd 100644 --- a/skills/solidcraft-tests/SKILL.md +++ b/skills/solidcraft-tests/SKILL.md @@ -8,25 +8,28 @@ description: "Use when writing, placing or running tests in SolidCraft: a regres The long form is `docs/book/src/dev/testing.md`. ## Placement -- A regression test goes into the existing test file for its area (booleans, blends, shell, sweep, STEP, naming, ...), never a new file named after the bug (#3226). -- Engine tests sit next to the code or in `*_tests.rs` files; they measure the result (volume, area, face/edge/vertex counts) against a value you can derive. -- UI scenarios are JSON in `crates/ui-egui/tests/scenarios/*.json`; a new file is picked up automatically. Put `{"pending": "why"}` in a scenario that records a bug before it is fixed, and remove it in the fixing PR. -- Reference tests are `tests/oracle//recipe.json` with an `expected.json` beside it, run by `cargo xtask oracle`. Expected values are worked out in closed form (`tools/oracle-analytic/`) or computed by FreeCAD run as a separate program (`tools/freecad-oracle/`). A case that fails on a known bug names its issue (`"known_failure": "#123"`). Never change an expected value to match SolidCraft's output, and never take one from a commercial CAD product. -- Crash fixes: a regression test, started from the crash report's `design.solidcraft` and `replay.json` (`skills/solidcraft-crash-safety/SKILL.md`). +- A regression test goes into its area's existing test file (booleans, blends, shell, STEP, naming, ...), never a new file named after the bug (#3226). +- Engine tests sit next to the code or in `*_tests.rs` files, measured against a value you can derive. +- UI scenarios are JSON in `crates/ui-egui/tests/scenarios/*.json`. Put `{"pending": "why"}` in a scenario that records a bug before it is fixed, and remove it in the fixing PR. +- Reference tests are `tests/oracle//recipe.json` with an `expected.json` beside it, run by `cargo xtask oracle`, expected values in closed form (`tools/oracle-analytic/`) or from FreeCAD as a separate program (`tools/freecad-oracle/`). A known failure names its issue (`"known_failure": "#123"`). Never change an expected value to match SolidCraft's output, and never take one from a commercial CAD product. +- Crash fixes: a regression test from the report's `design.solidcraft` and `replay.json` (`skills/solidcraft-crash-safety/SKILL.md`). + +## Unit test or rig case +- Unit tests assert the code's logic: which edges a chain picks, name resolution, history labels, refusal wording, hostile input, parsing and saving (`corner_rounds_larger_than_their_faces_are_too_large`: per-edge radii, which no command takes, refused as `TooLarge`). +- The rig (`tests/kernel-cases/`) asserts geometry: exact volumes, shapes, seam continuity, geometric refusals, through commands with a hand-derived answer, so it survives a kernel swap (`fillets/190-fillet-disc-rim-past-its-2mm-wall`). A test doing both keeps only its logic. ## A failing test needs its fix -A failing test without its fix cannot merge: it turns `main` red for everyone. Include the fix in the same PR, or mark the test `#[ignore = "fails until #123 is fixed"]` and remove that line in the PR that fixes it. +A failing test cannot merge without its fix (it turns `main` red): include the fix, or mark the test `#[ignore = "fails until #123 is fixed"]` and remove that line in the PR that fixes it. ## Running -- `cargo xtask test` builds with the `ci` profile and runs every test binary at once; arguments after `--` go to each binary (`cargo xtask test -- shell`). Use the `ci` profile: it builds several times faster than `--release`, and debug assertions are off because truck's reject valid geometry. If a test fails only in a plain debug build, say so in the PR. -- `SOLIDCRAFT_TEST_JOBS=N` bounds concurrent commands on a shared machine. -- `cargo xtask ci` is the gate before a commit: fmt, assets, icons, licences, lists, layers, ui-containers, clippy `-D warnings`, tests and the wasm check. -- Every test runs in its own process: do not rely on state left by another test, or on wall-clock timing. +- `cargo xtask test` builds with the `ci` profile and runs every test binary at once; arguments after `--` go to each binary (`cargo xtask test -- shell`). It builds several times faster than `--release`; a test failing only in a debug build: say so in the PR. +- `SOLIDCRAFT_TEST_JOBS=N` bounds concurrent commands. `cargo xtask ci` is the gate before a commit (fmt, lints, clippy `-D warnings`, tests, wasm). +- Tests run in their own processes: no shared state, no wall-clock timing. ## Goldens, matrices and skipped tests -- The reference for what the kernel builds is the kernel-cases ladder (`tests/kernel-cases/ladder/`, `cargo xtask kernel-cases --ladder`, `docs/kernel-ladder.md`). -- Kernel PRs run `kernel-rig.yml`: the areas the diff reaches (`PATHS`) against `KNOWN_FAILING`, the ladder against its golden. In the same PR a fix removes its line, a new failing case adds one, a changed ladder result is re-recorded (`--record "why"`), a new command gets a case or an `EXEMPT` line; see `tests/kernel-cases/README.md`. -- Maintainers' PRs run the native, GPU path and reference tests on the affected packages; others wait for `approved-for-ci`. `release-gate.yml` runs everything in full before develop replaces main. -- The action matrix is a ratchet: record a newly passing action with `cargo xtask action-matrix --bless` (`docs/action-matrix.md`). -- Pull requests skip a few heavy tests, which run in the merge queue, nightly and on `main`. Check that validation after merging. The list is in `.config/nextest.toml` and the testing page. -- Mutation testing finds what the suites miss: `docs/mutation-testing.md`. +- The kernel's reference is the ladder (`tests/kernel-cases/ladder/`, `cargo xtask kernel-cases --ladder`, `docs/kernel-ladder.md`). +- Kernel PRs run `kernel-rig.yml` (areas by `PATHS` against `KNOWN_FAILING`, the ladder against its golden). In the same PR a fix drops its line, a new failing case adds one, a changed ladder result is re-recorded (`--record "why"`), a new command gets a case or an `EXEMPT` line (`tests/kernel-cases/README.md`). +- Maintainers' PRs run native, GPU and reference tests on affected packages; others wait for `approved-for-ci`. `release-gate.yml` runs all before develop replaces main. +- The action matrix is a ratchet: bless a newly passing action (`cargo xtask action-matrix --bless`, `docs/action-matrix.md`). +- PRs skip a few heavy tests, run in the merge queue, nightly and on `main`. Check them after merging (`.config/nextest.toml`). +- Mutation testing: `docs/mutation-testing.md`.