# Reproducing

> Run the benchmark harness on your own hardware or contribute an engine adapter.

Source: https://pie-project.org/benchmarks/methodology/contribute  

The evaluation harness, engine recipes, and aggregation scripts are open source in [pie-project/benchmarks](https://github.com/pie-project/benchmarks). You can reproduce any cell locally with the same controls used in CI or add a new inference engine via pull request.

## Running the harness locally

Install Rust and `uv`, then build the runner and execute a job from the repository root:

```bash
uv venv && uv pip install -e ".[dev,orchestrate]"
pie-evals-node build --pie-commit <sha> --features cuda
pie-evals jobs --tier nightly --pie-commit <sha> --platform <id> --engine pie --engine <id> --out jobs
pie-evals-node run --job jobs/<job>.json --out out/<job> --download
pie-evals collect --tier nightly out/<job>
pie-evals report --tier nightly
```

Pass `--platform <id>` matching an entry in `matrix/platforms.yaml` (or add a new host definition there), and substitute `--features rocm` on AMD GPUs.

When running outside CI, ensure one engine has exclusive access to the accelerator, use unmodified checkpoints and KV precision matching the matrix definition, keep prefix caching disabled on throughput sweeps, and sweep batched concurrency until aggregate `tok/s` gains fall below `5%`.

## Submitting measurements

Raw per-round timings and environment metadata are written to `out/<job>/records.jsonl` alongside engine stdout/stderr logs. For security and hardware consistency, CI never ingests external `records.jsonl` files automatically; attach a zipped job directory along with host specifications and exact commands to an [issue](https://github.com/pie-project/benchmarks/issues/new) or pull request so maintainers can audit and schedule an official CI run.

## Adding an engine

Engines are integrated declaratively through six touchpoints in `pie-project/benchmarks`:

| Component        | Path                                           | Purpose                                                                  |
| ---------------- | ---------------------------------------------- | ------------------------------------------------------------------------ |
| Matrix entry     | `matrix/engines.yaml`                          | Engine ID, supported OS/backends, weight formats, tiers, and version pin |
| Schema enum      | `src/pie_evals/schema/cell.py`                 | Registers the engine ID in `EngineName`                                  |
| Runner adapter   | `src/pie_evals/node/engines/<id>.py`           | `Engine` subclass registered via `@register`                             |
| Tuning recipe    | `src/pie_evals/node/engines/recipes/<id>.yaml` | Default and throughput-tuned CLI flags with rationale                    |
| Known exclusions | `matrix/support.yaml`                          | Unsupported model/platform pairs and expected error signatures           |
| Unit tests       | `tests/test_engines.py`                        | Verifies CLI flag generation and metric parsing                          |

Any server exposing an OpenAI-compatible `/v1/completions` endpoint can subclass `ServedEngine` and specify only server startup and readiness checks (following the existing adapters for `llama.cpp`, `mlx-lm`, Splash, and TensorRT-LLM).

Validate the adapter locally before opening a pull request:

```bash
pie-evals check
pytest -q -m "not slow"
pie-evals jobs --tier nightly --engine <id> --platform <id> --out jobs
```

Once merged, maintainers trigger a tagged CI run on project hardware to populate the site.