Model slices

Frontier MoE and dense checkpoints often exceed the memory of a single GPU or Mac, yet single-accelerator kernel efficiency — attention, quantized GEMMs, MoE dispatch, and graph launch overhead — can still be isolated without multi-node confounding. A model slice (marked ▫ in the grids) trims total parameter storage to fit one device while preserving the exact per-layer compute geometry of the full architecture.

How slices are constructedCopy Anchor Link

Slicing never alters hidden dimensions, attention head counts, head dimensions, MoE intermediate widths, active experts per token (top-k), quantization group sizes, or the vocabulary projection head:

1. Expert pruning first: For Mixture-of-Experts models, total stored routed experts are reduced while keeping top-k active experts per token unchanged. Every token dispatches the same arithmetic volume and active weight footprint per layer as the full model (aside from higher L2/DRAM cache locality on expert weights, which applies equally to every engine).
2. Whole-block layer reduction second: Only if expert pruning alone still exceeds target VRAM are repeating transformer blocks removed, always in complete architectural periods (for example, preserving hybrid linear/full-attention ratios).

Every engine in a cell loads the exact same slice checkpoint, processes identical token IDs for all 256 decode steps, and undergoes next-token KL verification against a reference engine.

Scope and limitationsCopy Anchor Link

Slices are strictly a controlled relative benchmark on a single device:

• Omitted from absolute tok/s views: Because fewer layers execute per token (or fewer experts contend for capacity), raw tok/s on a slice is faster than the full model and is never reported in the Throughput (tok/s) view.
• Not a multi-GPU proxy: Multi-GPU and multi-node throughput depends on tensor-parallel all-reduce latency and expert-parallel all-to-all communication, which single-device slices do not exercise.
• Excluded from end-to-end tasks: Downstream accuracy benchmarks (MATH-500) and speculative draft acceptance rates require coherent semantics and run exclusively on unmodified checkpoints.

Representative slice configurationsCopy Anchor Link