SmartCore runs a continuous benchmark harness on every push to the development branch. The harness lives in the smartcore-benches repository, which covers the five hot paths flagged in smartcore#407 as well as the legacy algorithm benches.
Trend charts and JSON history are published to GitHub Pages by benchmark-action/github-action-benchmark:
| Tool | Chart | What it plots | Role |
|---|---|---|---|
| ⏱️ criterion | smartcorelib.github.io/smartcore-benches/dev/ | Wall-clock time per bench (ns/iter) over time. Lower is better. | Advisory — 200% alert threshold. Wall-clock numbers on shared GitHub runners are noisy (20–60% variance is normal), so this chart is informational, not a merge gate. |
| 📊 iai-callgrind | smartcorelib.github.io/smartcore-benches/iai-dev/ | Instructions retired (Ir) per bench — deterministic and machine-independent. Lower is better. |
Deterministic gate — 120% alert threshold, fails the job on a >20% regression. Posts a success/failure status check back to the triggering smartcore commit. |
Open either chart in a browser. Each page shows a searchable line chart with one series per benchmark name (e.g. matmul/1024, iai_matmul::matmul::bench_matmul_256). Hover for the value and the commit that produced each point.
Instruction counts from Valgrind’s Callgrind are machine-independent — the same binary reports the same instruction count on every Linux x86_64 host, including GitHub-hosted runners. This makes a tight 120% threshold reliable: a >20% jump is a real code change, not measurement noise. When the iai job fails, it means a smartcore commit regressed the instruction count of one of the hot paths beyond the threshold — the smartcore commit receives a smartcore-benches / iai-callgrind failed status check, which branch protection can require.
In addition to the rolling trend, the criterion job records one frozen criterion.json per published smartcore version under benches-results/v<version>/ in the benches repo — useful for diffing two smartcore releases against each other rather than tracking the rolling trend.
Each CI run uploads its raw bench output as a downloadable artifact:
criterion-json — bench-output/criterion.json (bencher format)iai-json — bench-output/iai.ndjson (raw iai-callgrind NDJSON) + bench-output/iai.json (the customSmallerIsBetter array the adapter produces)Download from a run page, or via the gh CLI:
gh run download <run-id> --repo smartcorelib/smartcore-benches -n iai-json -D ./iai-out
gh run download <run-id> --repo smartcorelib/smartcore-benches -n criterion-json -D ./crit-out
The full guide on how to read the results, the dispatch contract that fires from smartcore’s CI, and the required secrets are documented in the smartcore-benches README.