Benchmarks and reports

The repository benchmark harness builds real projects in separate checkouts. This tests wall time and whether path normalization produces portable keys.

Nightly benchmark matrix

The Bench workflow is scheduled every night at 04:37 UTC. It runs:

Firefox cold/warm builds on Linux and Windows
LLVM, Substrate, SurrealDB, Lance, and OpenDAL cold/warm builds
Firefox through sccache for comparison
Firefox source-update builds on Linux and Windows

Jobs upload their reports, Perfetto traces, and logs even when a benchmark fails. GitHub retains those artifacts for 30 days.

A scheduled run is not automatically a valid result. Check that the individual job succeeded and that its benchmark verdict is ok before using its timing or hit-rate numbers.

Run a scenario

just bench                 # list Kache scenarios
just bench firefox
just bench substrate
just bench lance
just bench opendal
just bench llvm
just bench-retry firefox   # reuse the saved cold state
just bench-trace firefox   # include key-diff artifacts

just bench-sccache
just bench-sccache firefox

Firefox is a large mixed Rust/C++ workload and needs about 50 GB of scratch space. Run benchmark builds sequentially on an otherwise quiet host.

Trust the verdict first

Each summary has a verdict. Use timing and hit-rate results only when it is ok. A degraded verdict can mean checkout-path leakage, an unexpected miss, or inconsistent cache placement.

Compare medians from several runs. Keep repository revision, toolchain, machine, target, profile, and remote location fixed.

Output

Latest artifacts are under tmp/bench/<scenario>/:

cold and warm JSON/Markdown reports
Perfetto-compatible trace files
build and wrapper logs
scenario summary with tool version
optional key-diff JSON and Markdown
sccache reports for comparison runs

Completed runs are archived below tmp/bench/<scenario>/runs/ so later runs do not erase the evidence.

Reports outside the harness

kache report --format text --since 24h
kache report --format json --output report.json
kache report --format trace --output trace.json
kache report --format markdown --root "$PWD"

trace, perfetto, and chrome-trace produce the same trace format. Open it in Perfetto or Chrome's trace viewer.

Add a workload

Benchmark definitions live under scenarios/bench-*. A scenario declares its repository, revision, setup, wrapper wiring, build command, and optional patches. Start from bench-firefox, bench-substrate, or bench-llvm and read the scenario format.

For choosing a cache before benchmarking, see Kache or sccache?.