Files
solidcraft/crates/engine/src/schema/coverage_tests.rs
T
816bcf10de Every new command comes with input and output schemas (#2829) (#4035)
* Every new command comes with input and output schemas

A test fails for a command without a contract or an output type, unless it is a planned!
placeholder or listed in schema/unconverted.txt, which holds the 400 commands that predate
the rule and only shrinks: the test also fails when a listed command has its schemas, when
the list names a command that doesn't exist, or when it isn't sorted.

`checked!` now takes only the command's id and checks the contract that id publishes
(`run_checked_as`), so the contract enforced is always the one described. Another test runs
every command that has a contract in strict with an unknown key, so a contract is never
published without `checked!`, under the command's own id, enforcing it.

A new book page, Command schemas, explains how to write a contract and a reply type, the
tests to add, the shared field types and how to add one, with a checklist; the failure
messages point to it. "Adding a command", CONTRIBUTING.md and AGENTS.md now include the
schemas, and the MCP tool description and docs/mcp.md no longer list the converted commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* List the commands added on main since: view.layout, view.history_layout, view.normal_to

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Command schemas guide: what participants and targets do, checking the evaluation, warning-free replies

- `participants` changes the result for some features and nothing for others; where it and
  another key override each other, or where it has no effect, say so in that key's
  description or in the summary.
- The example primitive's `targets` default to every body in the feature's component, not the
  bodies it meets; they're ignored when `participants` lists bodies.
- Check what each value does where the feature is evaluated, not only how the handler reads it.
- A reply test runs every form in Warn and checks for no `param_warnings` (Strict for a reply
  that isn't an object).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* List the 43 commands added on main since: frame, component, render and others

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* List the 11 commands added on main since: browser folding, standard views, parameter sweep, path flange

browser.collapse_all, browser.expand_all, inspect.parameter_sweep, sheet.path_flange and the
seven view.* standard views (back, bottom, front, isometric, left, right, top) landed on main
without schemas; the gate flags them, as intended. 457 commands listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Schema rule warns instead of failing for now

A command without input and output schemas prints a warning listing it
(test output and a CI annotation). SCHEMA_RULE_STRICT in coverage_tests.rs
(or SOLIDCRAFT_SCHEMA_RULE_STRICT=1) makes it fail again. Docs say "should,
and will become required".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: bherbruck <brennen.herbruck@gmail.com>
2026-10-26 01:43:02 +00:00

107 lines
5.1 KiB
Rust

//! New commands come with schemas: every command has a contract (its input schema) and an
//! output schema, apart from `planned!` placeholders and the commands listed in
//! `unconverted.txt`. That list holds the commands that predate the rule; it only shrinks.
use std::collections::BTreeSet;
use serde_json::json;
use crate::Session;
use crate::cmd::command_specs;
use crate::schema::ParamCheck;
use crate::schema::check::MAX_KEYS;
/// Whether a command without schemas fails the test. Off for now: the test warns instead. The
/// owner will flip this to `true` when the PR backlog is down. `SOLIDCRAFT_SCHEMA_RULE_STRICT=1`
/// turns it on for one run.
const SCHEMA_RULE_STRICT: bool = false;
const UNCONVERTED: &str = include_str!("unconverted.txt");
/// The ids in `unconverted.txt`, in file order (`#` lines are comments).
fn unconverted() -> Vec<&'static str> {
UNCONVERTED.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect()
}
#[test]
fn every_command_has_input_and_output_schemas() {
let listed: BTreeSet<&str> = unconverted().into_iter().collect();
let mut problems = Vec::new();
let mut missing = Vec::new();
for c in command_specs() {
let has = (c.contract.is_some(), c.output.is_some());
let listed = listed.contains(c.id);
if c.planned.is_some() {
if listed {
problems.push(format!("`{}` is a planned! placeholder, which needs no line in unconverted.txt", c.id));
}
continue;
}
match (has, listed) {
((true, true), true) => problems.push(format!("`{}` has its schemas now: remove its line from unconverted.txt", c.id)),
((true, true), false) | (_, true) => {}
((input, output), false) => missing.push(format!(
"`{}` has no {} schema. Every new command needs a contract (`.contract(…)`, run through `checked!`) \
and an output type (`.output(…)`): see docs/book/src/dev/schemas.md",
c.id,
match (input, output) {
(true, _) => "output",
(false, true) => "input",
(false, false) => "input or output",
}
)),
}
}
assert!(problems.is_empty(), "{}", problems.join("\n"));
if !missing.is_empty() {
let strict = SCHEMA_RULE_STRICT || std::env::var_os("SOLIDCRAFT_SCHEMA_RULE_STRICT").is_some_and(|v| v != "0");
let report = format!(
"{} command(s) lack an input or output schema. This is a warning for now; it will become required:\n{}",
missing.len(),
missing.join("\n")
);
assert!(!strict, "{report}");
eprintln!("warning: {report}");
// A GitHub Actions annotation, so the warning shows on the CI run page too.
println!("::warning title=Commands without schemas::{}", missing.len());
}
}
/// A contract that is published but not checked would describe calls nobody refuses: every
/// command with a contract runs through `checked!` with its own id, which checks the contract
/// that id publishes. Calls `run` itself (not `Session::execute`, which first asks `enabled`)
/// in strict with an unknown key, which must be refused under the command's own id before the
/// handler sees it.
#[test]
fn every_contract_is_checked_in_front_of_its_handler() {
const PROBE: &str = "not_a_key_of_any_command";
for c in command_specs() {
let Some(contract) = c.contract else { continue };
// The check reads at most `MAX_KEYS` keys of a call, so a contract can't take more.
let keys = contract.allowed_keys().len();
assert!(keys <= MAX_KEYS, "`{}` takes {keys} keys; a call may have at most {MAX_KEYS}, so no call could use them all", c.id);
let mut s = Session { param_check: ParamCheck::Strict, ..Session::default() };
let run = c.run;
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| run(&mut s, &json!({PROBE: 1}))));
let fix = format!("give its CommandSpec `crate::checked!(\"{}\", <its handler>)`", c.id);
let error = match result {
Ok(Err(e)) => e.to_string(),
Ok(Ok(v)) => panic!("`{}` ran with an unknown key in strict ({v}): {fix}", c.id),
Err(_) => panic!("`{}` panicked on an unknown key in strict: {fix}", c.id),
};
let expected = format!("invalid parameters for `{}`: unknown key `{PROBE}`", c.id);
assert!(error.starts_with(&expected), "`{}` should refuse with \"{expected}\", not \"{error}\": {fix}", c.id);
}
}
#[test]
fn the_unconverted_list_is_sorted_and_names_real_commands() {
let ids = unconverted();
for pair in ids.windows(2) {
assert!(pair[0] < pair[1], "unconverted.txt must be sorted, each id once: `{}` comes before `{}`", pair[0], pair[1]);
}
let registered: BTreeSet<&str> = command_specs().iter().map(|c| c.id).collect();
let stale: Vec<&str> = ids.into_iter().filter(|id| !registered.contains(id)).collect();
assert!(stale.is_empty(), "unconverted.txt names commands that don't exist (remove their lines): {stale:?}");
}