# Scoring & metrics

> How benchmark runs are executed, aggregated, and scored across engines.

Source: https://pie-project.org/benchmarks/methodology  

Every result on this site is produced automatically by CI in [pie-project/benchmarks](https://github.com/pie-project/benchmarks) and published alongside its raw per-round logs. Engines are compared strictly within identical `(device, checkpoint, workload)` cells, with each engine configured for maximum throughput using recipes committed to the repository.

## Workloads

| Workload           | Specification                                                           | Metric                    |
| ------------------ | ----------------------------------------------------------------------- | ------------------------- |
| Single decode      | Concurrency 1 · 512 input tokens, 256 output tokens                     | Output `tok/s`            |
| Single prefill     | Concurrency 1 · 512 input tokens                                        | Input `tok/s`             |
| Batched decode     | Concurrency swept from 4 to 1,024 (lower on memory-bound Macs)          | Peak total output `tok/s` |
| Batched prefill    | Concurrency swept from 4 to 1,024 · 512 input tokens, 1 output token    | Peak total input `tok/s`  |
| Speculative decode | Single decode with matched draft model and lossless greedy verification | Output `tok/s`            |

Full parameter controls are documented in [Workloads & controls](https://pie-project.org/benchmarks/methodology/test-conditions).

## Relative speed

Performance grids default to **Relative speed**: Pie's throughput divided by the fastest non-Pie engine in the same cell, expressed as a signed percentage (`+20%` indicates `1.20×` the throughput of the strongest baseline). Differences within `±5%` (`±4.5%` unrounded) are within run-to-run variance and rendered at parity.

On the [Overview](https://pie-project.org/benchmarks), pairwise single-stream decode results on each platform are summarized into a **Bradley–Terry win probability**: we fit engine strengths `p_i` via Hunter's MM algorithm such that `Pr(i beats j) = p_i / (p_i + p_j)` with a `(+0.5, +0.5)` prior per met pair, normalized to geometric mean `1`, and report `p_i / (p_i + 1)` — the win probability against a platform-average engine.

Cells carry two status markers when a full checkpoint exceeds single-device memory:

* **`▫` Model slice:** A reduced-layer or reduced-expert checkpoint preserving exact per-layer tensor dimensions, quantization groups, and active `top-k` routing. All engines execute the identical slice, yielding a controlled relative comparison; slices are omitted from the **Throughput (tok/s)** view because absolute token rates do not reflect the full model. See [Model slices](https://pie-project.org/benchmarks/methodology/slices).
* **`⧄` Exceeds memory:** Neither the full checkpoint nor a slice was evaluated on that device.

Where Pie leads the fastest baseline by `≥25%`, maintainers audit the trace and attach a diagnostic note visible on hover (for example, missing CUDA graph capture or fallback attention kernels in a baseline release).

## Measurement & variance control

Each cell begins with 2–3 uncounted warmup iterations, followed by 3 measured rounds for single-request workloads and 2 rounds for batched sweeps. If the relative spread across rounds exceeds `2%` (`5%` for batched runs), the harness runs additional rounds up to 5 and records the median. Cells whose spread remains above threshold are flagged as noisy in the dataset; Apple Silicon hosts insert cooldown intervals between rounds to avoid thermal throttling.

## Isolation & execution environments

Before warmup, the runner acquires an exclusive host lock, terminates lingering engine processes, verifies that accelerators are idle, and records exact commit SHAs and package versions. Benchmarks track the latest upstream release of each engine; when upstream bugs block evaluation of a new architecture, minimal patches are maintained in public forks ([pie-project/vllm](https://github.com/pie-project/vllm) and [pie-project/sglang](https://github.com/pie-project/sglang)).

| Target           | CI workflow      | Host provisioning                                    |
| ---------------- | ---------------- | ---------------------------------------------------- |
| Apple Silicon    | `pie-eval.yml`   | Dedicated resident Macs, one job per host            |
| NVIDIA GPU       | `tier.yml`       | Ephemeral RunPod instances provisioned per job       |
| AMD GPU          | `tier.yml`       | Ephemeral MI300X instances provisioned per job       |
| Google TPU       | `tpu.yml`        | Dedicated Cloud TPU VM created and torn down per run |
| Browser (WebGPU) | `pod-import.yml` | Headless Chrome on dedicated GPU instances           |
| Correctness      | `quality.yml`    | MATH-500 and next-token KL verification jobs         |