# Workloads & controls

> Controlled variables, engine tuning rules, and workload definitions.

Source: https://pie-project.org/benchmarks/methodology/test-conditions  

Comparisons are valid only within the same `(device, model, workload)` cell. Every run holds weights, tokenization, numerical precision, and request shapes constant while permitting each engine to use its fastest non-lossy runtime flags.

## Controlled invariants

All engines in a cell load the same checkpoint and weight quantization scheme (or the identical [model slice](https://pie-project.org/benchmarks/methodology/slices) when evaluating oversized architectures on a single accelerator). Prompts consist of seeded random vocabulary tokens so prefill never benefits from cross-request prefix caching; exact input token IDs and fixed output lengths are verified after every run.

Decoding uses greedy argmax (`temperature = 0`) with reasoning/thinking modes disabled so every engine performs the exact same number of forward passes. KV caches are held at `bf16` across engines unless a model's native release specifies otherwise. Speculative decoding is evaluated in its own workload (`spec`) using matched draft weights and lossless greedy verification, and on Single decode cells each engine is credited with the faster of its standard or speculative throughput.

Engine-specific flags (such as CUDA graph capture, chunked prefill sizing, or XLA ahead-of-time compilation on TPU for Pie, vLLM-TPU, and SGLang-JAX) are tuned per engine for peak throughput and version-controlled in `src/pie_evals/node/engines/recipes/`.

## Workload specifications

| Workload           | Request geometry                     | Concurrency                      | Reported metric               |
| ------------------ | ------------------------------------ | -------------------------------- | ----------------------------- |
| Single decode      | 512 input tokens → 256 output tokens | 1                                | Steady-state output `tok/s`   |
| Single prefill     | 512 input tokens → 256 output tokens | 1                                | Prompt processing `tok/s`     |
| Batched decode     | 512 input tokens → 256 output tokens | Swept `4 … 1,024` (lower on Mac) | Peak aggregate output `tok/s` |
| Batched prefill    | 512 input tokens → 1 output token    | Swept `4 … 1,024`                | Peak aggregate input `tok/s`  |
| Speculative decode | 512 input tokens → 256 output tokens | 1 (matched drafter)              | Verified output `tok/s`       |

For batched workloads, the harness sweeps concurrency upward until aggregate throughput plateaus (gains under `5%`) or device memory saturates, reporting each engine's peak achieved throughput on that device.