Reproducing
Run the benchmark harness on your own hardware or contribute an engine adapter.
The evaluation harness, engine recipes, and aggregation scripts are open source in pie-project/benchmarks. You can reproduce any cell locally with the same controls used in CI or add a new inference engine via pull request.
Running the harness locally
Install Rust and uv, then build the runner and execute a job from the repository root:
uv venv && uv pip install -e ".[dev,orchestrate]"
pie-evals-node build --pie-commit <sha> --features cuda
pie-evals jobs --tier nightly --pie-commit <sha> --platform <id> --engine pie --engine <id> --out jobs
pie-evals-node run --job jobs/<job>.json --out out/<job> --download
pie-evals collect --tier nightly out/<job>
pie-evals report --tier nightlyPass --platform <id> matching an entry in matrix/platforms.yaml (or add a new host definition there), and substitute --features rocm on AMD GPUs.
When running outside CI, ensure one engine has exclusive access to the accelerator, use unmodified checkpoints and KV precision matching the matrix definition, keep prefix caching disabled on throughput sweeps, and sweep batched concurrency until aggregate tok/s gains fall below 5%.
Submitting measurements
Raw per-round timings and environment metadata are written to out/<job>/records.jsonl alongside engine stdout/stderr logs. For security and hardware consistency, CI never ingests external records.jsonl files automatically; attach a zipped job directory along with host specifications and exact commands to an issue or pull request so maintainers can audit and schedule an official CI run.
Adding an engine
Engines are integrated declaratively through six touchpoints in pie-project/benchmarks:
| Component | Path | Purpose |
|---|---|---|
| Matrix entry | matrix/engines.yaml | Engine ID, supported OS/backends, weight formats, tiers, and version pin |
| Schema enum | src/pie_evals/schema/cell.py | Registers the engine ID in EngineName |
| Runner adapter | src/pie_evals/node/engines/<id>.py | Engine subclass registered via @register |
| Tuning recipe | src/pie_evals/node/engines/recipes/<id>.yaml | Default and throughput-tuned CLI flags with rationale |
| Known exclusions | matrix/support.yaml | Unsupported model/platform pairs and expected error signatures |
| Unit tests | tests/test_engines.py | Verifies CLI flag generation and metric parsing |
Any server exposing an OpenAI-compatible /v1/completions endpoint can subclass ServedEngine and specify only server startup and readiness checks (following the existing adapters for llama.cpp, mlx-lm, Splash, and TensorRT-LLM).
Validate the adapter locally before opening a pull request:
pie-evals check
pytest -q -m "not slow"
pie-evals jobs --tier nightly --engine <id> --platform <id> --out jobsOnce merged, maintainers trigger a tagged CI run on project hardware to populate the site.