CI: build and run the tests in one job, sccache on a Blacksmith disk (#3941)

* CI: build and run the tests in one job, sccache on a Blacksmith sticky disk

The tests job compiles and runs the tests itself on the 32 vCPU runner, so a pull request or merge
group uploads and downloads no 3 GB archive (the download took 45 to 385 s per shard, and once
14 minutes) and starts beside the repository checks instead of after them. The archive is built
only for the nightly and manual jobs that need it. sccache moves from GitHub's 10 GB cache, where
about 1100 per-ref entries were evicting each other, to a sticky disk that only main commits.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* TEMP: let this branch save caches so the warm run can be measured

* Revert "TEMP: let this branch save caches so the warm run can be measured"

This reverts commit 1bbbfcf5ff.

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
bherbruckandClaude Sonnet 5.5 authored and GitHub committed 2026-10-20 21:25:35 -04:00
1 parent 6e87fb1a25
commit 10d7bd045e
1 file changed
+94 -33
+94 -33
View File
@@ -8,12 +8,13 @@
# whole workspace, as do pushes to main that change code, the nightly run and Run workflow.
# Documentation, packaging, the other workflows and the GitHub bots' scripts select nothing:
# no native job runs for them, on pull requests or on main.
# Tests are built once and run in one job on a big runner, slowest first. A few heavy tests (see .config/nextest.toml) are
# Tests are built and run in one job on a big runner, slowest first, starting beside the repository
# checks (no archive to upload and download). A few heavy tests (see .config/nextest.toml) are
# skipped on pull requests and run on pushes to main and nightly. The Windows jobs (D3D12 shaders
# and app startup), the reference tests and the doc tests run on main and nightly only, not on
# pull requests (the fast lane, in Select checks); a break there is fixed forward on main. Only
# main saves the build caches; pull requests restore them. On pull requests the test build and the
# shards start only after the format and repository checks pass.
# main saves the build caches; pull requests restore them. sccache is on a Blacksmith sticky disk that
# only main commits.
# A push to main runs only the repository checks when a passing merge-queue run tested that very
# commit, or when nothing a build reads changed since the last push to main whose run passed.
# Otherwise it runs everything.
@@ -29,13 +30,13 @@
# reads `ci` from the newest run of a commit, so a run without it would block a PR that passed.
# Cross-compilation is opt-in through Run workflow; release builds remain in release.yml.
# Every Linux job runs on Blacksmith's Ubuntu 24.04 runners (blacksmith-<N>vcpu-ubuntu-2404), sized
# per job: 32 vCPU for the tests, 16 for Build tests and Clippy, 8 for compiles, doc and reference tests, 4 for the light checks, 2 for Select checks and ci.
# per job: 32 vCPU for the tests, 16 for Clippy, the archive build (nightly) and the merge queue's, 8 for compiles, doc and reference tests, 4 for the light checks, 2 for Select checks and ci.
# The Windows shader job runs on Blacksmith's Windows Server 2025 runners (blacksmith-8vcpu-windows-2025);
# the Windows app startup smoke stays on windows-latest, which has a desktop session that presents frames.
# Pull requests, forks included, the merge queue, pushes to main and the nightly run all use them:
# the runners are ephemeral, fork runs get no secrets, and main saves the build caches that every
# other run restores read-only. The tests are built once (Build tests) into an archive the shards
# run; Blacksmith's image matches ubuntu-24.04, so the archive and the recorded goldens behave the
# other run restores read-only. Nightly and by hand, Build tests also makes an archive the matrix
# jobs run; Blacksmith's image matches ubuntu-24.04, so the archive and the recorded goldens behave the
# same.
name: CI
@@ -339,25 +340,33 @@ jobs:
build-tests:
name: Build tests
# Fail fast on pull requests: the test archive (and so the shards) waits for the repository
# checks, the most common cheap failure; heavy jobs used to run on to the end beside a failed
# format check. Main, nightly and Run workflow build regardless, so the action matrix and the
# property tests that hang off this job keep their signal.
needs: [changes, checks]
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.native == 'true' && (github.event_name != 'pull_request' || needs.checks.result == 'success') }}
# The archive is for the action matrix and the property tests, which run only nightly and by
# hand. Pull requests, the merge queue and main build and run their tests in the `tests` job.
needs: changes
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.native == 'true' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') }}
runs-on: blacksmith-16vcpu-ubuntu-2404
# Fork builds start from main's cache, read-only; give them room to finish and upload.
timeout-minutes: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 60 || 45 }}
env:
CARGO_INCREMENTAL: "0"
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
SCCACHE_DIR: /home/runner/.cache/sccache
SCCACHE_CACHE_SIZE: 20G
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Size parallel compiles to the job's memory
run: scripts/ci/build-jobs.sh
- uses: dtolnay/rust-toolchain@stable
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
# forks mount a clone and never write back, so nothing they build can reach main's cache.
- name: Mount the compiler cache
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
continue-on-error: true
with:
key: ${{ github.repository }}-sccache-archive
path: /home/runner/.cache/sccache
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
key: native-tests
@@ -390,12 +399,12 @@ jobs:
tests:
name: Native Linux tests
needs: [changes, checks, build-tests]
if: ${{ !cancelled() && needs.changes.outputs.native == 'true' && (github.event_name != 'pull_request' || needs.checks.result == 'success') && needs.build-tests.result == 'success' }}
# One job on a big runner: nextest hands tests to its threads as they free up, the slowest
# first (priority in .config/nextest.toml), so they start at once and no core idles at the
# end. Four shards cut by `slice` each paid the setup (about 6 minutes) and a shard that drew
# several slow tests ran for 20 minutes while the others waited.
needs: changes
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.native == 'true' }}
# Builds the tests and runs them in one job on a big runner: there is no archive to upload
# and download (3 GB on a full run), and the job starts beside the repository checks, not
# after them. nextest hands tests to its threads as they free up, the slowest first (priority
# in .config/nextest.toml), so they start at once and no core idles at the end.
runs-on: blacksmith-32vcpu-ubuntu-2404
timeout-minutes: 60
env:
@@ -405,23 +414,47 @@ jobs:
SOLIDCRAFT_GPU: software
SOLIDCRAFT_REQUIRE_GPU: "1"
ACTION_MATRIX_JOBS: "12"
CARGO_INCREMENTAL: "0"
RUSTC_WRAPPER: sccache
SCCACHE_DIR: /home/runner/.cache/sccache
SCCACHE_CACHE_SIZE: 20G
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Install a software Vulkan driver (lavapipe)
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends mesa-vulkan-drivers libvulkan1
- name: Size parallel compiles to the job's memory
run: scripts/ci/build-jobs.sh
- uses: dtolnay/rust-toolchain@stable
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
# forks mount a clone and never write back, so nothing they build can reach main's cache.
- name: Mount the compiler cache
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
continue-on-error: true
with:
key: ${{ github.repository }}-sccache-tests
path: /home/runner/.cache/sccache
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
# Main saves the cache; every other run restores it, read-only.
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
key: native-tests
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Set up compiler cache
uses: mozilla-actions/sccache-action@fc920bf0ec8de6ee65d409111f7ec508035751ba # v0.0.11
with:
version: v0.16.0
- uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2.87.28
with:
tool: cargo-nextest
# Each shard runs the archive the Build tests job made.
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: nextest-archive
- name: Run tests
env:
EVENT_NAME: ${{ github.event_name }}
GOLDENS: ${{ needs.changes.outputs.goldens }}
FULL: ${{ needs.changes.outputs.full }}
PACKAGES: ${{ needs.changes.outputs.packages }}
# The robustness golden seeds a pull request skips: each takes one to twenty minutes under
# CI load, together about four fifths of the goldens' time. Pushes to main, the merge
# queue and the nightly run every seed.
@@ -446,11 +479,12 @@ jobs:
fi
junit=${CARGO_TARGET_DIR:-target}/nextest/ci/junit.xml
rm -f "$junit"
# Extract into ./target: tests that run a built binary (env!("CARGO_BIN_EXE_*")) have
# the build job's path, which is the same checkout path here.
run=(cargo nextest run --archive-file tests.tar.zst --workspace-remap . --extract-to . --extract-overwrite)
# Pull requests test the affected packages (scripts/ci/affected.py).
scope=(--workspace)
if [ "$FULL" != true ]; then scope=(); for p in $PACKAGES; do scope+=(-p "$p"); done; fi
status=0
"${run[@]}" --profile ci --no-tests=pass --ignore-default-filter -E "$base" || status=1
cargo nextest run "${scope[@]}" --cargo-profile ci --locked --profile ci --no-tests=pass \
--ignore-default-filter -E "$base" || status=1
if [ "$status" != 0 ]; then
# A failed golden gets the hint to re-record it (junit.xml lists the failures).
if [ -f "$junit" ] && tr '\n' ' ' < "$junit" | grep -q '<testcase name="tests::robustness_goldens::[^"]*"[^>]*>[[:space:]]*<failure'; then
@@ -459,6 +493,9 @@ jobs:
fi
exit 1
fi
- name: Compiler cache statistics
if: always()
run: sccache --show-stats
- name: Upload test timings
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
@@ -568,13 +605,23 @@ jobs:
env:
CARGO_INCREMENTAL: "0"
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
SCCACHE_DIR: /home/runner/.cache/sccache
SCCACHE_CACHE_SIZE: 20G
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Size parallel compiles to the job's memory
run: scripts/ci/build-jobs.sh
- uses: dtolnay/rust-toolchain@stable
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
# forks mount a clone and never write back, so nothing they build can reach main's cache.
- name: Mount the compiler cache
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
continue-on-error: true
with:
key: ${{ github.repository }}-sccache-doc
path: /home/runner/.cache/sccache
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
# Main saves the cache; every other run restores it, read-only.
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
@@ -616,13 +663,23 @@ jobs:
env:
CARGO_INCREMENTAL: "0"
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
SCCACHE_DIR: /home/runner/.cache/sccache
SCCACHE_CACHE_SIZE: 20G
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Size parallel compiles to the job's memory
run: scripts/ci/build-jobs.sh
- uses: dtolnay/rust-toolchain@stable
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
# forks mount a clone and never write back, so nothing they build can reach main's cache.
- name: Mount the compiler cache
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
continue-on-error: true
with:
key: ${{ github.repository }}-sccache-oracle
path: /home/runner/.cache/sccache
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
# Main saves the cache; every other run restores it, read-only.
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
@@ -754,6 +811,7 @@ jobs:
DOCTESTS: ${{ needs.changes.outputs.doctests }}
CLIPPY: ${{ needs.clippy.result }}
BUILD: ${{ needs.build-tests.result }}
EVENT_NAME: ${{ github.event_name }}
TESTS: ${{ needs.tests.result }}
DOCS: ${{ needs.doc-tests.result }}
ORACLE: ${{ needs.oracle.result }}
@@ -768,7 +826,10 @@ jobs:
fi
if [ "$NATIVE" = true ]; then
test "$CLIPPY" = success
test "$BUILD" = success
# The archive for the action matrix and property tests exists nightly and by hand only.
if [ "$EVENT_NAME" = schedule ] || [ "$EVENT_NAME" = workflow_dispatch ]; then
test "$BUILD" = success
fi
test "$TESTS" = success
fi
if [ "$DOCTESTS" = true ]; then