Correctness

Throughput figures are meaningful only if the engine computes the intended model. The Correctness matrix verifies Pie against a reference engine (vLLM on NVIDIA/AMD/TPU, mlx-lm on Apple Silicon, or SGLang/llama.cpp where primary baselines lack model support) on matched weights and hardware using two complementary checks.

Full checkpoints: MATH-500Copy Anchor Link

Models that fit in device memory are evaluated end-to-end on MATH-500 under greedy decoding (temperature = 0, thinking disabled). Both Pie and the reference engine score the full benchmark on the same checkpoint; a cell passes when Pie's accuracy lies within the run's measured sampling tolerance of the reference score (or within ±5.0 percentage points when per-run variance is unavailable).

Model slices: next-token KL divergenceCopy Anchor Link

Because model slices remove layers or experts, they do not produce coherent multi-step reasoning traces. Instead, CI verifies forward-pass numerical equivalence directly on the next-token distribution:

1. Identical prompt token sequences are fed to Pie and the reference engine on the same slice checkpoint.
2. Both engines execute a full forward pass through every retained layer and emit first-token log-probabilities.
3. The harness computes mean forward KL divergence KL(p_ref || p_pie) and top-1 token agreement across the prompt set.
4. A cell passes when mean KL divergence is below 0.15 nats.

Threshold calibration & edge casesCopy Anchor Link

Floating-point accumulation order and fused kernel rounding between two correct implementations typically produce a mean KL near 0.001 nats, whereas structural bugs — miswired residual connections, incorrect RoPE scaling, broken MoE routing, or mismatched chat templates — push divergence well past 0.15 nats. Even on a slice, the first token exercises embedding lookup, every retained attention and MoE block, final normalization, and the LM head.

Two edge cases receive explicit handling in CI:

• Degenerate shallow slices: Retaining too few layers can leave output logits nearly uniform, masking wiring defects. Such slices are regenerated with more blocks or marked uncheckable.
• Borderline kernel drift: When aggressive low-precision kernels push KL slightly past 0.15, the reference engine is re-evaluated with unfused reference kernels; if divergence falls within 2× the reference engine's own kernel-to-kernel spread, parity is adjudicated over 40 multi-token completions.