CI: build and run the tests in one job, sccache on a Blacksmith disk (#3941)
* CI: build and run the tests in one job, sccache on a Blacksmith sticky disk
The tests job compiles and runs the tests itself on the 32 vCPU runner, so a pull request or merge
group uploads and downloads no 3 GB archive (the download took 45 to 385 s per shard, and once
14 minutes) and starts beside the repository checks instead of after them. The archive is built
only for the nightly and manual jobs that need it. sccache moves from GitHub's 10 GB cache, where
about 1100 per-ref entries were evicting each other, to a sticky disk that only main commits.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* TEMP: let this branch save caches so the warm run can be measured
* Revert "TEMP: let this branch save caches so the warm run can be measured"
This reverts commit 1bbbfcf5ff.
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6e87fb1a25
commit
10d7bd045e
1 file changed
+94
-33
+94
-33
@@ -8,12 +8,13 @@
|
||||
# whole workspace, as do pushes to main that change code, the nightly run and Run workflow.
|
||||
# Documentation, packaging, the other workflows and the GitHub bots' scripts select nothing:
|
||||
# no native job runs for them, on pull requests or on main.
|
||||
# Tests are built once and run in one job on a big runner, slowest first. A few heavy tests (see .config/nextest.toml) are
|
||||
# Tests are built and run in one job on a big runner, slowest first, starting beside the repository
|
||||
# checks (no archive to upload and download). A few heavy tests (see .config/nextest.toml) are
|
||||
# skipped on pull requests and run on pushes to main and nightly. The Windows jobs (D3D12 shaders
|
||||
# and app startup), the reference tests and the doc tests run on main and nightly only, not on
|
||||
# pull requests (the fast lane, in Select checks); a break there is fixed forward on main. Only
|
||||
# main saves the build caches; pull requests restore them. On pull requests the test build and the
|
||||
# shards start only after the format and repository checks pass.
|
||||
# main saves the build caches; pull requests restore them. sccache is on a Blacksmith sticky disk that
|
||||
# only main commits.
|
||||
# A push to main runs only the repository checks when a passing merge-queue run tested that very
|
||||
# commit, or when nothing a build reads changed since the last push to main whose run passed.
|
||||
# Otherwise it runs everything.
|
||||
@@ -29,13 +30,13 @@
|
||||
# reads `ci` from the newest run of a commit, so a run without it would block a PR that passed.
|
||||
# Cross-compilation is opt-in through Run workflow; release builds remain in release.yml.
|
||||
# Every Linux job runs on Blacksmith's Ubuntu 24.04 runners (blacksmith-<N>vcpu-ubuntu-2404), sized
|
||||
# per job: 32 vCPU for the tests, 16 for Build tests and Clippy, 8 for compiles, doc and reference tests, 4 for the light checks, 2 for Select checks and ci.
|
||||
# per job: 32 vCPU for the tests, 16 for Clippy, the archive build (nightly) and the merge queue's, 8 for compiles, doc and reference tests, 4 for the light checks, 2 for Select checks and ci.
|
||||
# The Windows shader job runs on Blacksmith's Windows Server 2025 runners (blacksmith-8vcpu-windows-2025);
|
||||
# the Windows app startup smoke stays on windows-latest, which has a desktop session that presents frames.
|
||||
# Pull requests, forks included, the merge queue, pushes to main and the nightly run all use them:
|
||||
# the runners are ephemeral, fork runs get no secrets, and main saves the build caches that every
|
||||
# other run restores read-only. The tests are built once (Build tests) into an archive the shards
|
||||
# run; Blacksmith's image matches ubuntu-24.04, so the archive and the recorded goldens behave the
|
||||
# other run restores read-only. Nightly and by hand, Build tests also makes an archive the matrix
|
||||
# jobs run; Blacksmith's image matches ubuntu-24.04, so the archive and the recorded goldens behave the
|
||||
# same.
|
||||
name: CI
|
||||
|
||||
@@ -339,25 +340,33 @@ jobs:
|
||||
|
||||
build-tests:
|
||||
name: Build tests
|
||||
# Fail fast on pull requests: the test archive (and so the shards) waits for the repository
|
||||
# checks, the most common cheap failure; heavy jobs used to run on to the end beside a failed
|
||||
# format check. Main, nightly and Run workflow build regardless, so the action matrix and the
|
||||
# property tests that hang off this job keep their signal.
|
||||
needs: [changes, checks]
|
||||
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.native == 'true' && (github.event_name != 'pull_request' || needs.checks.result == 'success') }}
|
||||
# The archive is for the action matrix and the property tests, which run only nightly and by
|
||||
# hand. Pull requests, the merge queue and main build and run their tests in the `tests` job.
|
||||
needs: changes
|
||||
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.native == 'true' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') }}
|
||||
runs-on: blacksmith-16vcpu-ubuntu-2404
|
||||
# Fork builds start from main's cache, read-only; give them room to finish and upload.
|
||||
timeout-minutes: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 60 || 45 }}
|
||||
env:
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTC_WRAPPER: sccache
|
||||
SCCACHE_GHA_ENABLED: "true"
|
||||
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
|
||||
SCCACHE_DIR: /home/runner/.cache/sccache
|
||||
SCCACHE_CACHE_SIZE: 20G
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- name: Size parallel compiles to the job's memory
|
||||
run: scripts/ci/build-jobs.sh
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
|
||||
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
|
||||
# forks mount a clone and never write back, so nothing they build can reach main's cache.
|
||||
- name: Mount the compiler cache
|
||||
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
|
||||
continue-on-error: true
|
||||
with:
|
||||
key: ${{ github.repository }}-sccache-archive
|
||||
path: /home/runner/.cache/sccache
|
||||
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
|
||||
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
key: native-tests
|
||||
@@ -390,12 +399,12 @@ jobs:
|
||||
|
||||
tests:
|
||||
name: Native Linux tests
|
||||
needs: [changes, checks, build-tests]
|
||||
if: ${{ !cancelled() && needs.changes.outputs.native == 'true' && (github.event_name != 'pull_request' || needs.checks.result == 'success') && needs.build-tests.result == 'success' }}
|
||||
# One job on a big runner: nextest hands tests to its threads as they free up, the slowest
|
||||
# first (priority in .config/nextest.toml), so they start at once and no core idles at the
|
||||
# end. Four shards cut by `slice` each paid the setup (about 6 minutes) and a shard that drew
|
||||
# several slow tests ran for 20 minutes while the others waited.
|
||||
needs: changes
|
||||
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.native == 'true' }}
|
||||
# Builds the tests and runs them in one job on a big runner: there is no archive to upload
|
||||
# and download (3 GB on a full run), and the job starts beside the repository checks, not
|
||||
# after them. nextest hands tests to its threads as they free up, the slowest first (priority
|
||||
# in .config/nextest.toml), so they start at once and no core idles at the end.
|
||||
runs-on: blacksmith-32vcpu-ubuntu-2404
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
@@ -405,23 +414,47 @@ jobs:
|
||||
SOLIDCRAFT_GPU: software
|
||||
SOLIDCRAFT_REQUIRE_GPU: "1"
|
||||
ACTION_MATRIX_JOBS: "12"
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTC_WRAPPER: sccache
|
||||
SCCACHE_DIR: /home/runner/.cache/sccache
|
||||
SCCACHE_CACHE_SIZE: 20G
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- name: Install a software Vulkan driver (lavapipe)
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y --no-install-recommends mesa-vulkan-drivers libvulkan1
|
||||
- name: Size parallel compiles to the job's memory
|
||||
run: scripts/ci/build-jobs.sh
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
|
||||
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
|
||||
# forks mount a clone and never write back, so nothing they build can reach main's cache.
|
||||
- name: Mount the compiler cache
|
||||
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
|
||||
continue-on-error: true
|
||||
with:
|
||||
key: ${{ github.repository }}-sccache-tests
|
||||
path: /home/runner/.cache/sccache
|
||||
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
|
||||
# Main saves the cache; every other run restores it, read-only.
|
||||
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
key: native-tests
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
- name: Set up compiler cache
|
||||
uses: mozilla-actions/sccache-action@fc920bf0ec8de6ee65d409111f7ec508035751ba # v0.0.11
|
||||
with:
|
||||
version: v0.16.0
|
||||
- uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2.87.28
|
||||
with:
|
||||
tool: cargo-nextest
|
||||
# Each shard runs the archive the Build tests job made.
|
||||
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
||||
with:
|
||||
name: nextest-archive
|
||||
- name: Run tests
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
GOLDENS: ${{ needs.changes.outputs.goldens }}
|
||||
FULL: ${{ needs.changes.outputs.full }}
|
||||
PACKAGES: ${{ needs.changes.outputs.packages }}
|
||||
# The robustness golden seeds a pull request skips: each takes one to twenty minutes under
|
||||
# CI load, together about four fifths of the goldens' time. Pushes to main, the merge
|
||||
# queue and the nightly run every seed.
|
||||
@@ -446,11 +479,12 @@ jobs:
|
||||
fi
|
||||
junit=${CARGO_TARGET_DIR:-target}/nextest/ci/junit.xml
|
||||
rm -f "$junit"
|
||||
# Extract into ./target: tests that run a built binary (env!("CARGO_BIN_EXE_*")) have
|
||||
# the build job's path, which is the same checkout path here.
|
||||
run=(cargo nextest run --archive-file tests.tar.zst --workspace-remap . --extract-to . --extract-overwrite)
|
||||
# Pull requests test the affected packages (scripts/ci/affected.py).
|
||||
scope=(--workspace)
|
||||
if [ "$FULL" != true ]; then scope=(); for p in $PACKAGES; do scope+=(-p "$p"); done; fi
|
||||
status=0
|
||||
"${run[@]}" --profile ci --no-tests=pass --ignore-default-filter -E "$base" || status=1
|
||||
cargo nextest run "${scope[@]}" --cargo-profile ci --locked --profile ci --no-tests=pass \
|
||||
--ignore-default-filter -E "$base" || status=1
|
||||
if [ "$status" != 0 ]; then
|
||||
# A failed golden gets the hint to re-record it (junit.xml lists the failures).
|
||||
if [ -f "$junit" ] && tr '\n' ' ' < "$junit" | grep -q '<testcase name="tests::robustness_goldens::[^"]*"[^>]*>[[:space:]]*<failure'; then
|
||||
@@ -459,6 +493,9 @@ jobs:
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
- name: Compiler cache statistics
|
||||
if: always()
|
||||
run: sccache --show-stats
|
||||
- name: Upload test timings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||
@@ -568,13 +605,23 @@ jobs:
|
||||
env:
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTC_WRAPPER: sccache
|
||||
SCCACHE_GHA_ENABLED: "true"
|
||||
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
|
||||
SCCACHE_DIR: /home/runner/.cache/sccache
|
||||
SCCACHE_CACHE_SIZE: 20G
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- name: Size parallel compiles to the job's memory
|
||||
run: scripts/ci/build-jobs.sh
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
|
||||
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
|
||||
# forks mount a clone and never write back, so nothing they build can reach main's cache.
|
||||
- name: Mount the compiler cache
|
||||
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
|
||||
continue-on-error: true
|
||||
with:
|
||||
key: ${{ github.repository }}-sccache-doc
|
||||
path: /home/runner/.cache/sccache
|
||||
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
|
||||
# Main saves the cache; every other run restores it, read-only.
|
||||
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
@@ -616,13 +663,23 @@ jobs:
|
||||
env:
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTC_WRAPPER: sccache
|
||||
SCCACHE_GHA_ENABLED: "true"
|
||||
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
|
||||
SCCACHE_DIR: /home/runner/.cache/sccache
|
||||
SCCACHE_CACHE_SIZE: 20G
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- name: Size parallel compiles to the job's memory
|
||||
run: scripts/ci/build-jobs.sh
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
# sccache lives on a Blacksmith sticky disk (an ext4 volume cloned from the last snapshot,
|
||||
# no 10 GB limit and no download). Only main commits it; pull requests, the merge queue and
|
||||
# forks mount a clone and never write back, so nothing they build can reach main's cache.
|
||||
- name: Mount the compiler cache
|
||||
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
|
||||
continue-on-error: true
|
||||
with:
|
||||
key: ${{ github.repository }}-sccache-oracle
|
||||
path: /home/runner/.cache/sccache
|
||||
commit: ${{ github.ref == 'refs/heads/main' && 'on-change' || 'false' }}
|
||||
# Main saves the cache; every other run restores it, read-only.
|
||||
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
@@ -754,6 +811,7 @@ jobs:
|
||||
DOCTESTS: ${{ needs.changes.outputs.doctests }}
|
||||
CLIPPY: ${{ needs.clippy.result }}
|
||||
BUILD: ${{ needs.build-tests.result }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
TESTS: ${{ needs.tests.result }}
|
||||
DOCS: ${{ needs.doc-tests.result }}
|
||||
ORACLE: ${{ needs.oracle.result }}
|
||||
@@ -768,7 +826,10 @@ jobs:
|
||||
fi
|
||||
if [ "$NATIVE" = true ]; then
|
||||
test "$CLIPPY" = success
|
||||
test "$BUILD" = success
|
||||
# The archive for the action matrix and property tests exists nightly and by hand only.
|
||||
if [ "$EVENT_NAME" = schedule ] || [ "$EVENT_NAME" = workflow_dispatch ]; then
|
||||
test "$BUILD" = success
|
||||
fi
|
||||
test "$TESTS" = success
|
||||
fi
|
||||
if [ "$DOCTESTS" = true ]; then
|
||||
|
||||
Reference in new issue
Block a user