Scoring & metrics

Every result on this site is produced automatically by CI in pie-project/benchmarks and published alongside its raw per-round logs. Engines are compared strictly within identical (device, checkpoint, workload) cells, with each engine configured for maximum throughput using recipes committed to the repository.

WorkloadsCopy Anchor Link

Full parameter controls are documented in Workloads & controls.

Relative speedCopy Anchor Link

Performance grids default to Relative speed: Pie's throughput divided by the fastest non-Pie engine in the same cell, expressed as a signed percentage (+20% indicates 1.20× the throughput of the strongest baseline). Differences within ±5% (±4.5% unrounded) are within run-to-run variance and rendered at parity.

On the Overview, pairwise single-stream decode results on each platform are summarized into a Bradley–Terry win probability: we fit engine strengths p_i via Hunter's MM algorithm such that Pr(i beats j) = p_i / (p_i + p_j) with a (+0.5, +0.5) prior per met pair, normalized to geometric mean 1, and report p_i / (p_i + 1) — the win probability against a platform-average engine.

Cells carry two status markers when a full checkpoint exceeds single-device memory:

• ▫ Model slice: A reduced-layer or reduced-expert checkpoint preserving exact per-layer tensor dimensions, quantization groups, and active top-k routing. All engines execute the identical slice, yielding a controlled relative comparison; slices are omitted from the Throughput (tok/s) view because absolute token rates do not reflect the full model. See Model slices.
• ⧄ Exceeds memory: Neither the full checkpoint nor a slice was evaluated on that device.

Where Pie leads the fastest baseline by ≥25%, maintainers audit the trace and attach a diagnostic note visible on hover (for example, missing CUDA graph capture or fallback attention kernels in a baseline release).

Measurement & variance controlCopy Anchor Link

Each cell begins with 2–3 uncounted warmup iterations, followed by 3 measured rounds for single-request workloads and 2 rounds for batched sweeps. If the relative spread across rounds exceeds 2% (5% for batched runs), the harness runs additional rounds up to 5 and records the median. Cells whose spread remains above threshold are flagged as noisy in the dataset; Apple Silicon hosts insert cooldown intervals between rounds to avoid thermal throttling.

Isolation & execution environmentsCopy Anchor Link

Before warmup, the runner acquires an exclusive host lock, terminates lingering engine processes, verifies that accelerators are idle, and records exact commit SHAs and package versions. Benchmarks track the latest upstream release of each engine; when upstream bugs block evaluation of a new architecture, minimal patches are maintained in public forks (pie-project/vllm and pie-project/sglang).