Workloads & controls
Controlled variables, engine tuning rules, and workload definitions.
Comparisons are valid only within the same (device, model, workload) cell. Every run holds weights, tokenization, numerical precision, and request shapes constant while permitting each engine to use its fastest non-lossy runtime flags.
Controlled invariants
All engines in a cell load the same checkpoint and weight quantization scheme (or the identical model slice when evaluating oversized architectures on a single accelerator). Prompts consist of seeded random vocabulary tokens so prefill never benefits from cross-request prefix caching; exact input token IDs and fixed output lengths are verified after every run.
Decoding uses greedy argmax (temperature = 0) with reasoning/thinking modes disabled so every engine performs the exact same number of forward passes. KV caches are held at bf16 across engines unless a model's native release specifies otherwise. Speculative decoding is evaluated in its own workload (spec) using matched draft weights and lossless greedy verification, and on Single decode cells each engine is credited with the faster of its standard or speculative throughput.
Engine-specific flags (such as CUDA graph capture, chunked prefill sizing, or XLA ahead-of-time compilation on TPU for Pie, vLLM-TPU, and SGLang-JAX) are tuned per engine for peak throughput and version-controlled in src/pie_evals/node/engines/recipes/.
Workload specifications
| Workload | Request geometry | Concurrency | Reported metric |
|---|---|---|---|
| Single decode | 512 input tokens → 256 output tokens | 1 | Steady-state output tok/s |
| Single prefill | 512 input tokens → 256 output tokens | 1 | Prompt processing tok/s |
| Batched decode | 512 input tokens → 256 output tokens | Swept 4 … 1,024 (lower on Mac) | Peak aggregate output tok/s |
| Batched prefill | 512 input tokens → 1 output token | Swept 4 … 1,024 | Peak aggregate input tok/s |
| Speculative decode | 512 input tokens → 256 output tokens | 1 (matched drafter) | Verified output tok/s |
For batched workloads, the harness sweeps concurrency upward until aggregate throughput plateaus (gains under 5%) or device memory saturates, reporting each engine's peak achieved throughput on that device.