# Pie benchmark results

> Throughput, install size and output correctness of Pie against other inference engines, measured continuously by CI on real hardware.

Source: https://pie-project.org/benchmarks  
Release measured: main-f03640150  
Snapshot taken: 2026-11-13 (the live page is newer: results are published several times a day)  
Methodology: https://pie-project.org/benchmarks/methodology.md

Coverage: 26,000+ runs, 49 days of compute, 27 models, 13 engines, 5 platforms, 120 machines.

## Head to head

For each platform, the chance (percent) that an engine decodes faster than an engine of average strength, from a Bradley-Terry fit over every machine and model both engines ran.

### Apple Silicon

| Engine | Strength | Measured on |
|---|---|---|
| Pie | 90% | 80 model–device pairs |
| Splash | 57% | 9 model–device pairs |
| vLLM Metal | 49% | 53 model–device pairs |
| MLX | 39% | 73 model–device pairs |
| llama.cpp | 11% | 66 model–device pairs |

### NVIDIA GPU

| Engine | Strength | Measured on |
|---|---|---|
| Pie | 80% | 299 model–device pairs |
| SGLang | 47% | 277 model–device pairs |
| vLLM | 39% | 305 model–device pairs |
| TensorRT-LLM | 30% | 114 model–device pairs |

### AMD GPU

| Engine | Strength | Measured on |
|---|---|---|
| Pie | 67% | 62 model–device pairs |
| SGLang | 42% | 52 model–device pairs |
| vLLM | 41% | 49 model–device pairs |

### Google TPU

| Engine | Strength | Measured on |
|---|---|---|
| Pie | 71% | 36 model–device pairs |
| vLLM | 41% | 34 model–device pairs |
| SGLang | 37% | 16 model–device pairs |

### Browser

| Engine | Strength | Measured on |
|---|---|---|
| Pie | 86% | 24 model–device pairs |
| WebLLM | 39% | 10 model–device pairs |
| llama.cpp WebGPU | 20% | 24 model–device pairs |

## Install size

Disk the runtime takes before any model weights.

### Apple Silicon

| Engine | Size | Version |
|---|---|---|
| llama.cpp | 31.3 MB | b11312 |
| LiteRT-LM | 38.7 MB | v0.19.1 |
| Pie | 103 MB | 0.5.4 |
| Splash | 171 MB | 1.1.0 |
| MLX | 359 MB | 0.31.3 |
| vLLM Metal | 1.9 GB | 0.33.0 |

### NVIDIA GPU

| Engine | Size | Version |
|---|---|---|
| Pie | 1.2 GB | 0.6.0 (bb2d6bf43) |
| TensorRT-LLM | 12.3 GB | 1.4.0rc2.post1 |
| vLLM | 17.4 GB | 0.32.0 |
| SGLang | 19.8 GB | 0.5.22 |

### AMD GPU

| Engine | Size | Version |
|---|---|---|
| Pie | 5.9 GB | 0.6.0 (5871e632) |
| vLLM | 20.1 GB | 0.32.0 |
| SGLang | 20.6 GB | 0.5.22 |

### Google TPU

| Engine | Size | Version |
|---|---|---|
| Pie | 808 MB | 0.6.0 (bb2d6bf43) |
| SGLang | 2.2 GB | sglang-jax 0.1.0rc0 |
| vLLM | 4.9 GB | vllm-tpu 0.32.0 |

### Browser

| Engine | Size | Version |
|---|---|---|
| llama.cpp WebGPU | 4.9 MB | b11312 |
| WebLLM | 14.5 MB | 0.2.87 |
| Pie | 35.2 MB | server-web 0.6.4 |

## Throughput

Tokens per second, every engine that ran the same model on the same machine, fastest first. Single decode and prefill are one request at a time (512 tokens in, 256 out); batched figures are the peak across swept concurrency.

### Apple Silicon

| Machine | Model | Setup | Engines (tok/s) |
|---|---|---|---|
| M5 Max 128 GB | Gemma-4-E2B | Single decode | Pie 205 · MLX 185 · llama.cpp 166 |
| M5 Max 128 GB | Gemma-4-E2B | Single prefill | Pie 21,112 · MLX 16,192 · llama.cpp 4,806 |
| M5 Max 128 GB | Gemma-4-E2B | Batched decode | Pie 2,020 · llama.cpp 689 · MLX 185 |
| M5 Max 128 GB | Gemma-4-E2B | Batched prefill | Pie 27,717 · MLX 17,219 · llama.cpp 7,172 |
| M5 Max 128 GB | Gemma-4-E4B | Single decode | Pie 127 · MLX 120 · llama.cpp 107 |
| M5 Max 128 GB | Gemma-4-E4B | Single prefill | Pie 7,539 · MLX 6,528 · llama.cpp 2,919 |
| M5 Max 128 GB | Gemma-4-E4B | Batched decode | Pie 1,317 · llama.cpp 459 · MLX 116 |
| M5 Max 128 GB | Gemma-4-E4B | Batched prefill | Pie 9,167 · MLX 6,833 · llama.cpp 4,172 |
| M5 Max 128 GB | Gemma-4-12B | Single decode | Pie 66.3 · MLX 61 · llama.cpp 57.4 |
| M5 Max 128 GB | Gemma-4-12B | Single prefill | Pie 1,933 · MLX 1,838 · llama.cpp 1,153 |
| M5 Max 128 GB | Gemma-4-12B | Batched decode | Pie 521 · llama.cpp 238 · MLX 54.8 |
| M5 Max 128 GB | Gemma-4-12B | Batched prefill | MLX 1,750 · Pie 1,729 · llama.cpp 1,525 |
| M5 Max 128 GB | Qwen3.6-27B | Single decode | Pie 33.4 · MLX 31.8 · llama.cpp 27 |
| M5 Max 128 GB | Qwen3.6-27B | Single prefill | Pie 900 · MLX 857 · llama.cpp 529 |
| M5 Max 128 GB | Qwen3.6-27B | Batched decode | Pie 234 · llama.cpp 97.1 · MLX 29 |
| M5 Max 128 GB | Qwen3.6-27B | Batched prefill | MLX 772 · Pie 761 · llama.cpp 575 |
| M5 Max 128 GB | GPT-OSS-20B | Single decode | Pie 184 · llama.cpp 151 · MLX 149 |
| M5 Max 128 GB | GPT-OSS-20B | Single prefill | Pie 6,092 · MLX 5,024 · llama.cpp 3,036 |
| M5 Max 128 GB | GPT-OSS-20B | Batched decode | Pie 648 · llama.cpp 532 · MLX 134 |
| M5 Max 128 GB | GPT-OSS-20B | Batched prefill | Pie 6,425 · MLX 4,939 · llama.cpp 3,726 |
| M5 Max 128 GB | GPT-OSS-120B | Single decode | Pie 122 · llama.cpp 106 · MLX 97.5 |
| M5 Max 128 GB | GPT-OSS-120B | Single prefill | MLX 2,367 · llama.cpp 1,530 · Pie 1,234 |
| M5 Max 128 GB | GPT-OSS-120B | Batched decode | llama.cpp 250 · Pie 227 · MLX 83 |
| M5 Max 128 GB | GPT-OSS-120B | Batched prefill | Pie 2,700 · llama.cpp 1,928 |
| M5 Max 128 GB | Qwen3.6-35B | Single decode | Pie 166 · MLX 130 · llama.cpp 105 |
| M5 Max 128 GB | Qwen3.6-35B | Single prefill | Pie 4,292 · MLX 4,056 · llama.cpp 2,422 |
| M5 Max 128 GB | Qwen3.6-35B | Batched decode | Pie 735 · llama.cpp 367 · MLX 119 |
| M5 Max 128 GB | Qwen3.6-35B | Batched prefill | Pie 6,587 · llama.cpp 2,800 |
| M5 Max 128 GB | Qwen3.8-Flash-Next-125B | Single decode | Pie 75 · llama.cpp 63 · MLX 60.7 |
| M5 Max 128 GB | Qwen3.8-Flash-Next-125B | Single prefill | MLX 1,221 · Pie 1,054 · llama.cpp 966 |
| M5 Max 128 GB | Qwen3.8-Flash-Next-125B | Batched decode | Pie 269 · MLX 94 · llama.cpp 77.6 |
| M5 Max 128 GB | Qwen3.8-Flash-Next-125B | Batched prefill | Pie 1,294 · MLX 1,101 |
| M5 Max 128 GB | Muse-Glimmer-30B | Single decode | Pie 32.6 · llama.cpp 29.4 |
| M5 Max 128 GB | Muse-Glimmer-30B | Single prefill | Pie 903 · llama.cpp 541 |
| M5 Max 128 GB | Muse-Glimmer-30B | Batched decode | Pie 274 · llama.cpp 120 |
| M5 Max 128 GB | Muse-Glimmer-30B | Batched prefill | Pie 769 · llama.cpp 701 |
| M5 Max 128 GB | Gemma-4-26B | Single decode | Pie 143 · MLX 119 · llama.cpp 98.1 |
| M5 Max 128 GB | Gemma-4-26B | Single prefill | Pie 4,347 · MLX 4,008 · llama.cpp 2,399 |
| M5 Max 128 GB | Gemma-4-26B | Batched decode | Pie 806 · llama.cpp 456 · MLX 113 |
| M5 Max 128 GB | Gemma-4-26B | Batched prefill | Pie 5,647 · MLX 3,986 · llama.cpp 3,479 |
| M5 Max 128 GB | Gemma-4-31B | Single decode | Pie 27 · MLX 26.5 · llama.cpp 24.6 |
| M5 Max 128 GB | Gemma-4-31B | Single prefill | Pie 718 · MLX 637 · llama.cpp 443 |
| M5 Max 128 GB | Gemma-4-31B | Batched decode | Pie 184 · llama.cpp 97 · MLX 21.2 |
| M5 Max 128 GB | Gemma-4-31B | Batched prefill | MLX 635 · Pie 587 · llama.cpp 558 |
| M5 Max 128 GB | Qwen3.8-27B | Single decode | Pie 32.8 · MLX 31.9 · llama.cpp 27.2 |
| M5 Max 128 GB | Qwen3.8-27B | Single prefill | Pie 862 · MLX 856 · llama.cpp 536 |
| M5 Max 128 GB | Qwen3.8-27B | Batched decode | Pie 228 · llama.cpp 97.4 · MLX 28.7 |
| M5 Max 128 GB | Qwen3.8-27B | Batched prefill | MLX 774 · Pie 727 · llama.cpp 572 |
| M5 Max 128 GB | Qwen3.5-4B | Single decode | Pie 167 · MLX 152 · llama.cpp 128 |
| M5 Max 128 GB | Qwen3.5-4B | Single prefill | Pie 4,932 · MLX 4,908 · llama.cpp 2,997 |
| M5 Max 128 GB | Qwen3.5-4B | Batched decode | Pie 1,166 · llama.cpp 407 · MLX 142 |
| M5 Max 128 GB | Qwen3.5-4B | Batched prefill | Pie 5,536 · MLX 4,993 · llama.cpp 3,165 |
| M5 Max 128 GB | Qwen3.5-9B | Single decode | Pie 101 · MLX 96.4 · llama.cpp 82.8 |
| M5 Max 128 GB | Qwen3.5-9B | Single prefill | Pie 2,968 · MLX 2,938 · llama.cpp 1,808 |
| M5 Max 128 GB | Qwen3.5-9B | Batched decode | Pie 800 · llama.cpp 300 · MLX 88 |
| M5 Max 128 GB | Qwen3.5-9B | Batched prefill | MLX 2,866 · Pie 2,852 · llama.cpp 1,984 |
| M5 Max 64 GB | Qwen3.5-0.8B | Single decode | vLLM Metal 266 |
| M5 Max 64 GB | Qwen3.5-0.8B | Single prefill | vLLM Metal 19,134 |
| M5 Max 64 GB | Qwen3.5-0.8B | Batched decode | vLLM Metal 3,606 |
| M5 Max 64 GB | Qwen3.5-0.8B | Batched prefill | vLLM Metal 29,554 |
| M5 Max 64 GB | Gemma-4-E2B | Single decode | Pie 201 · vLLM Metal 189 · MLX 184 · llama.cpp 167 |
| M5 Max 64 GB | Gemma-4-E2B | Single prefill | Pie 21,434 · MLX 16,147 · vLLM Metal 16,112 · llama.cpp 4,801 |
| M5 Max 64 GB | Gemma-4-E2B | Batched decode | vLLM Metal 1,968 · Pie 1,939 · llama.cpp 706 · MLX 182 |
| M5 Max 64 GB | Gemma-4-E2B | Batched prefill | Pie 31,421 · vLLM Metal 29,520 · MLX 17,139 · llama.cpp 7,217 |
| M5 Max 64 GB | Gemma-4-E4B | Single decode | Pie 129 · MLX 120 · vLLM Metal 118 · llama.cpp 105 |
| M5 Max 64 GB | Gemma-4-E4B | Single prefill | Pie 7,636 · vLLM Metal 6,660 · MLX 6,565 · llama.cpp 2,832 |
| M5 Max 64 GB | Gemma-4-E4B | Batched decode | Pie 1,272 · vLLM Metal 1,088 · llama.cpp 467 · MLX 111 |
| M5 Max 64 GB | Gemma-4-E4B | Batched prefill | vLLM Metal 9,153 · Pie 8,473 · MLX 6,676 · llama.cpp 3,901 |
| M5 Max 64 GB | Gemma-4-12B | Single decode | Pie 65.2 · MLX 57.3 · vLLM Metal 55.8 · llama.cpp 52.2 |
| M5 Max 64 GB | Gemma-4-12B | Single prefill | vLLM Metal 1,632 · MLX 1,627 · Pie 1,555 · llama.cpp 988 |
| M5 Max 64 GB | Gemma-4-12B | Batched decode | Pie 473 · vLLM Metal 373 · llama.cpp 226 · MLX 47 |
| M5 Max 64 GB | Gemma-4-12B | Batched prefill | vLLM Metal 1,800 · MLX 1,596 · Pie 1,564 · llama.cpp 1,419 |
| M5 Max 64 GB | Qwen3.6-27B | Single decode | Pie 33.6 · vLLM Metal 33 · MLX 32 · llama.cpp 25.3 |
| M5 Max 64 GB | Qwen3.6-27B | Single prefill | Pie 849 · vLLM Metal 823 · MLX 781 · llama.cpp 454 |
| M5 Max 64 GB | Qwen3.6-27B | Batched decode | Pie 220 · vLLM Metal 169 · llama.cpp 92.4 · MLX 24.4 |
| M5 Max 64 GB | Qwen3.6-27B | Batched prefill | vLLM Metal 807 · MLX 714 · Pie 689 · llama.cpp 538 |
| M5 Max 64 GB | GPT-OSS-20B | Single decode | Pie 184 · vLLM Metal 158 · llama.cpp 148 · MLX 148 |
| M5 Max 64 GB | GPT-OSS-20B | Single prefill | Pie 6,052 · MLX 4,911 · vLLM Metal 4,714 · llama.cpp 2,909 |
| M5 Max 64 GB | GPT-OSS-20B | Batched decode | vLLM Metal 736 · Pie 597 · llama.cpp 497 · MLX 129 |
| M5 Max 64 GB | GPT-OSS-20B | Batched prefill | Pie 6,116 · vLLM Metal 6,008 · MLX 4,538 · llama.cpp 3,429 |
| M5 Max 64 GB | Qwen3.6-35B | Single decode | Pie 168 · MLX 130 · vLLM Metal 122 · llama.cpp 104 |
| M5 Max 64 GB | Qwen3.6-35B | Single prefill | Pie 4,535 · MLX 4,072 · vLLM Metal 3,933 · llama.cpp 2,333 |
| M5 Max 64 GB | Qwen3.6-35B | Batched decode | Pie 688 · vLLM Metal 592 · llama.cpp 340 · MLX 121 |
| M5 Max 64 GB | Qwen3.6-35B | Batched prefill | Pie 6,189 · vLLM Metal 5,678 · MLX 3,880 · llama.cpp 2,591 |
| M5 Max 64 GB | Muse-Glimmer-30B | Single decode | Pie 33.2 · vLLM Metal 32.4 · llama.cpp 28 |
| M5 Max 64 GB | Muse-Glimmer-30B | Single prefill | Pie 911 · vLLM Metal 759 · llama.cpp 465 |
| M5 Max 64 GB | Muse-Glimmer-30B | Batched decode | Pie 249 · vLLM Metal 187 · llama.cpp 115 |
| M5 Max 64 GB | Muse-Glimmer-30B | Batched prefill | vLLM Metal 811 · Pie 688 · llama.cpp 653 |
| M5 Max 64 GB | Gemma-4-26B | Single decode | Pie 144 · vLLM Metal 119 · MLX 118 · llama.cpp 98 |
| M5 Max 64 GB | Gemma-4-26B | Single prefill | Pie 4,329 · vLLM Metal 4,114 · MLX 3,972 · llama.cpp 2,343 |
| M5 Max 64 GB | Gemma-4-26B | Batched decode | Pie 753 · vLLM Metal 482 · llama.cpp 417 · MLX 108 |
| M5 Max 64 GB | Gemma-4-26B | Batched prefill | vLLM Metal 5,357 · Pie 5,304 · MLX 3,717 · llama.cpp 3,163 |
| M5 Max 64 GB | Gemma-4-31B | Single decode | Pie 27.3 · MLX 24.8 · vLLM Metal 24.6 · llama.cpp 23 |
| M5 Max 64 GB | Gemma-4-31B | Single prefill | vLLM Metal 513 · MLX 510 · Pie 447 · llama.cpp 381 |
| M5 Max 64 GB | Gemma-4-31B | Batched decode | Pie 183 · vLLM Metal 140 · llama.cpp 67 · MLX 19.4 |
| M5 Max 64 GB | Gemma-4-31B | Batched prefill | MLX 617 · Pie 570 · vLLM Metal 528 |
| M5 Max 64 GB | Qwen3.8-27B | Single decode | Pie 33.6 · vLLM Metal 33 · MLX 32 · llama.cpp 25.7 |
| M5 Max 64 GB | Qwen3.8-27B | Single prefill | Pie 850 · vLLM Metal 779 · MLX 690 · llama.cpp 457 |
| M5 Max 64 GB | Qwen3.8-27B | Batched decode | Pie 220 · vLLM Metal 169 · llama.cpp 92.4 · MLX 24.6 |
| M5 Max 64 GB | Qwen3.8-27B | Batched prefill | vLLM Metal 807 · MLX 713 · Pie 687 · llama.cpp 538 |
| M5 Max 64 GB | Qwen3.5-4B | Single decode | Pie 166 · vLLM Metal 160 · MLX 152 · llama.cpp 121 |
| M5 Max 64 GB | Qwen3.5-4B | Single prefill | Pie 5,092 · MLX 4,920 · vLLM Metal 4,834 · llama.cpp 2,765 |
| M5 Max 64 GB | Qwen3.5-4B | Batched decode | Pie 1,145 · vLLM Metal 923 · llama.cpp 394 · MLX 137 |
| M5 Max 64 GB | Qwen3.5-4B | Batched prefill | vLLM Metal 5,583 · Pie 5,068 · MLX 4,704 · llama.cpp 2,982 |
| M5 Max 64 GB | Qwen3.5-9B | Single decode | vLLM Metal 102 · Pie 102 · MLX 96.2 · llama.cpp 77.3 |
| M5 Max 64 GB | Qwen3.5-9B | Single prefill | Pie 3,034 · MLX 2,933 · vLLM Metal 2,913 · llama.cpp 1,583 |
| M5 Max 64 GB | Qwen3.5-9B | Batched decode | Pie 732 · vLLM Metal 568 · llama.cpp 282 · MLX 77.3 |
| M5 Max 64 GB | Qwen3.5-9B | Batched prefill | vLLM Metal 2,994 · MLX 2,645 · Pie 2,541 · llama.cpp 1,821 |
| M5 16 GB | Gemma-4-E2B | Batched decode | llama.cpp 202 · MLX 48.9 |
| M5 16 GB | Gemma-4-E4B | Batched decode | llama.cpp 92.8 · MLX 25.5 |
| M5 16 GB | Gemma-4-12B | Batched decode | MLX 9.6 |
| M5 16 GB | Qwen3.5-4B | Batched decode | llama.cpp 70.6 · MLX 34.5 |
| M5 16 GB | Qwen3.5-9B | Batched decode | llama.cpp 31.9 |
| M4 Pro 48 GB | Qwen3.5-0.8B | Single decode | vLLM Metal 136 |
| M4 Pro 48 GB | Qwen3.5-0.8B | Single prefill | vLLM Metal 4,718 |
| M4 Pro 48 GB | Qwen3.5-0.8B | Batched decode | vLLM Metal 970 |
| M4 Pro 48 GB | Qwen3.5-0.8B | Batched prefill | vLLM Metal 5,512 |
| M4 Pro 48 GB | Gemma-4-E2B | Single decode | Pie 126 · vLLM Metal 114 · MLX 111 · llama.cpp 103 |
| M4 Pro 48 GB | Gemma-4-E2B | Single prefill | Pie 4,557 · MLX 4,258 · vLLM Metal 3,710 · llama.cpp 1,322 |
| M4 Pro 48 GB | Gemma-4-E2B | Batched decode | vLLM Metal 731 · Pie 723 · llama.cpp 393 · MLX 108 |
| M4 Pro 48 GB | Gemma-4-E2B | Batched prefill | Pie 5,098 · vLLM Metal 4,416 · MLX 4,318 · llama.cpp 1,517 |
| M4 Pro 48 GB | Gemma-4-E4B | Single decode | Pie 71.3 · vLLM Metal 65.4 · MLX 64.9 · llama.cpp 59.3 |
| M4 Pro 48 GB | Gemma-4-E4B | Single prefill | Pie 1,276 · MLX 1,237 · vLLM Metal 1,160 · llama.cpp 677 |
| M4 Pro 48 GB | Gemma-4-E4B | Batched decode | Pie 350 · vLLM Metal 345 · llama.cpp 225 · MLX 60.7 |
| M4 Pro 48 GB | Gemma-4-E4B | Batched prefill | Pie 1,352 · vLLM Metal 1,287 · MLX 1,244 · llama.cpp 759 |
| M4 Pro 48 GB | Gemma-4-12B | Single decode | Pie 33.4 · vLLM Metal 31.4 · MLX 31.1 · llama.cpp 29.2 |
| M4 Pro 48 GB | Gemma-4-12B | Single prefill | Pie 286 · MLX 280 · vLLM Metal 267 · llama.cpp 259 |
| M4 Pro 48 GB | Gemma-4-12B | Batched decode | Pie 110 · vLLM Metal 110 · llama.cpp 95.4 · MLX 27.2 |
| M4 Pro 48 GB | Gemma-4-12B | Batched prefill | Pie 298 · llama.cpp 282 · MLX 280 |
| M4 Pro 48 GB | Qwen3.6-27B | Single decode | Pie 15.8 · vLLM Metal 15.6 · MLX 15.1 · llama.cpp 13.2 |
| M4 Pro 48 GB | Qwen3.6-27B | Single prefill | Pie 214 · MLX 126 · vLLM Metal 125 · llama.cpp 118 |
| M4 Pro 48 GB | Qwen3.6-27B | Batched decode | Pie 51.6 · llama.cpp 32.9 |
| M4 Pro 48 GB | Qwen3.6-27B | Batched prefill | Pie 132 · MLX 126 |
| M4 Pro 48 GB | GPT-OSS-20B | Single decode | Pie 97.9 · vLLM Metal 90 · MLX 85 · llama.cpp 80.7 |
| M4 Pro 48 GB | GPT-OSS-20B | Single prefill | Pie 1,028 · MLX 893 · vLLM Metal 862 · llama.cpp 810 |
| M4 Pro 48 GB | GPT-OSS-20B | Batched decode | Pie 250 · vLLM Metal 219 · llama.cpp 193 · MLX 71.7 |
| M4 Pro 48 GB | GPT-OSS-20B | Batched prefill | Pie 1,113 · vLLM Metal 944 · llama.cpp 912 · MLX 897 |
| M4 Pro 48 GB | Qwen3.6-35B | Single decode | Pie 98.5 · vLLM Metal 86 · MLX 79.8 · llama.cpp 54.4 |
| M4 Pro 48 GB | Qwen3.6-35B | Single prefill | Pie 989 · MLX 946 · vLLM Metal 890 · llama.cpp 745 |
| M4 Pro 48 GB | Qwen3.6-35B | Batched decode | Pie 245 · vLLM Metal 227 · llama.cpp 143 · MLX 69.4 |
| M4 Pro 48 GB | Qwen3.6-35B | Batched prefill | Pie 1,171 · vLLM Metal 1,041 · MLX 948 · llama.cpp 820 |
| M4 Pro 48 GB | Muse-Glimmer-30B | Single decode | Pie 15.7 · vLLM Metal 15.6 · llama.cpp 14.5 |
| M4 Pro 48 GB | Muse-Glimmer-30B | Single prefill | Pie 135 · vLLM Metal 124 · llama.cpp 120 |
| M4 Pro 48 GB | Muse-Glimmer-30B | Batched decode | Pie 57 · vLLM Metal 51.9 · llama.cpp 46.6 |
| M4 Pro 48 GB | Muse-Glimmer-30B | Batched prefill | Pie 131 · llama.cpp 128 |
| M4 Pro 48 GB | Gemma-4-26B | Single decode | Pie 83.2 · vLLM Metal 73.8 · MLX 70.4 · llama.cpp 53.9 |
| M4 Pro 48 GB | Gemma-4-26B | Single prefill | Pie 874 · MLX 830 · vLLM Metal 759 · llama.cpp 704 |
| M4 Pro 48 GB | Gemma-4-26B | Batched decode | Pie 232 · vLLM Metal 192 · llama.cpp 175 · MLX 63.4 |
| M4 Pro 48 GB | Gemma-4-26B | Batched prefill | Pie 984 · vLLM Metal 846 · MLX 834 · llama.cpp 792 |
| M4 Pro 48 GB | Gemma-4-31B | Single decode | Pie 13.3 · MLX 12.9 · vLLM Metal 12.8 |
| M4 Pro 48 GB | Gemma-4-31B | Single prefill | Pie 106 · MLX 105 · vLLM Metal 101 |
| M4 Pro 48 GB | Gemma-4-31B | Batched decode | Pie 42 · vLLM Metal 41.8 · MLX 11.3 |
| M4 Pro 48 GB | Gemma-4-31B | Batched prefill | Pie 109 · MLX 105 |
| M4 Pro 48 GB | Qwen3.8-27B | Single decode | Pie 15.8 · vLLM Metal 15.6 · MLX 15.1 · llama.cpp 12.8 |
| M4 Pro 48 GB | Qwen3.8-27B | Single prefill | Pie 214 · MLX 127 · vLLM Metal 125 · llama.cpp 116 |
| M4 Pro 48 GB | Qwen3.8-27B | Batched decode | Pie 51.6 · llama.cpp 33.8 · MLX 13 |
| M4 Pro 48 GB | Qwen3.8-27B | Batched prefill | Pie 132 · MLX 126 |
| M4 Pro 48 GB | Qwen3.5-4B | Single decode | Pie 87.3 · vLLM Metal 84.2 · MLX 78 · llama.cpp 61.7 |
| M4 Pro 48 GB | Qwen3.5-4B | Single prefill | Pie 1,212 · MLX 825 · vLLM Metal 788 · llama.cpp 711 |
| M4 Pro 48 GB | Qwen3.5-4B | Batched decode | Pie 288 · vLLM Metal 282 · llama.cpp 169 · MLX 68.3 |
| M4 Pro 48 GB | Qwen3.5-4B | Batched prefill | Pie 849 · vLLM Metal 845 · MLX 825 · llama.cpp 726 |
| M4 Pro 48 GB | Qwen3.5-9B | Single decode | Pie 51.7 · vLLM Metal 50.5 · MLX 47.9 · llama.cpp 38.9 |
| M4 Pro 48 GB | Qwen3.5-9B | Single prefill | Pie 723 · MLX 449 · vLLM Metal 434 · llama.cpp 397 |
| M4 Pro 48 GB | Qwen3.5-9B | Batched decode | Pie 173 · vLLM Metal 167 · llama.cpp 116 · MLX 41.3 |
| M4 Pro 48 GB | Qwen3.5-9B | Batched prefill | Pie 472 · vLLM Metal 463 · MLX 448 · llama.cpp 411 |
| M3 16 GB | Qwen3.5-0.8B | Single decode | vLLM Metal 55.7 |
| M3 16 GB | Qwen3.5-0.8B | Single prefill | vLLM Metal 2,146 |
| M3 16 GB | Qwen3.5-0.8B | Batched decode | vLLM Metal 441 |
| M3 16 GB | Qwen3.5-0.8B | Batched prefill | vLLM Metal 2,319 |
| M3 16 GB | Gemma-4-E2B | Single decode | Pie 60.2 · vLLM Metal 57.7 · MLX 55.4 · llama.cpp 50.8 |
| M3 16 GB | Gemma-4-E2B | Single prefill | Pie 2,408 · MLX 1,842 · vLLM Metal 1,593 · llama.cpp 591 |
| M3 16 GB | Gemma-4-E2B | Batched decode | Pie 321 · vLLM Metal 290 · llama.cpp 193 · MLX 47 |
| M3 16 GB | Gemma-4-E2B | Batched prefill | Pie 2,616 · MLX 1,896 · vLLM Metal 1,815 |
| M3 16 GB | Gemma-4-E4B | Single decode | Pie 31.9 · vLLM Metal 30.8 · MLX 26.9 · llama.cpp 25.4 |
| M3 16 GB | Gemma-4-E4B | Single prefill | Pie 687 · MLX 524 · vLLM Metal 493 · llama.cpp 274 |
| M3 16 GB | Gemma-4-E4B | Batched decode | Pie 122 · vLLM Metal 111 · llama.cpp 59.4 · MLX 24.6 |
| M3 16 GB | Gemma-4-E4B | Batched prefill | Pie 749 · MLX 526 · vLLM Metal 435 |
| M3 16 GB | Gemma-4-12B | Single decode | Pie 12.6 · MLX 10.3 · vLLM Metal 8.2 |
| M3 16 GB | Gemma-4-12B | Single prefill | Pie 154 · MLX 102 · vLLM Metal 93.3 |
| M3 16 GB | Gemma-4-12B | Batched decode | Pie 33.4 · vLLM Metal 29.4 · MLX 7.5 |
| M3 16 GB | Gemma-4-12B | Batched prefill | Pie 111 · MLX 76.3 |
| M3 16 GB | Qwen3.5-4B | Single decode | Pie 35.1 · vLLM Metal 34.7 · MLX 33.6 · llama.cpp 26.2 |
| M3 16 GB | Qwen3.5-4B | Single prefill | Pie 515 · MLX 346 · vLLM Metal 334 · llama.cpp 276 |
| M3 16 GB | Qwen3.5-4B | Batched decode | Pie 98.5 · vLLM Metal 86.9 · llama.cpp 40.6 · MLX 24.9 |
| M3 16 GB | Qwen3.5-4B | Batched prefill | Pie 538 · MLX 335 |
| M3 16 GB | Qwen3.5-9B | Single decode | Pie 19.6 · vLLM Metal 19.2 · MLX 19 · llama.cpp 12.3 |
| M3 16 GB | Qwen3.5-9B | Single prefill | Pie 283 · MLX 177 · vLLM Metal 171 · llama.cpp 140 |
| M3 16 GB | Qwen3.5-9B | Batched decode | Pie 68.3 · llama.cpp 31.4 · MLX 11.7 |
| M3 16 GB | Qwen3.5-9B | Batched prefill | Pie 306 · MLX 131 |
| M2 16 GB | Qwen3.5-0.8B | Single decode | vLLM Metal 48.5 |
| M2 16 GB | Qwen3.5-0.8B | Single prefill | vLLM Metal 1,529 |
| M2 16 GB | Qwen3.5-0.8B | Batched decode | vLLM Metal 362 |
| M2 16 GB | Qwen3.5-0.8B | Batched prefill | vLLM Metal 1,304 |
| M2 16 GB | Gemma-4-E2B | Single decode | Pie 56.2 · MLX 46.9 · llama.cpp 46.7 · vLLM Metal 42.5 |
| M2 16 GB | Gemma-4-E2B | Single prefill | Pie 1,790 · MLX 1,343 · vLLM Metal 983 · llama.cpp 524 |
| M2 16 GB | Gemma-4-E2B | Batched decode | Pie 281 · vLLM Metal 194 · llama.cpp 179 · MLX 42.2 |
| M2 16 GB | Gemma-4-E2B | Batched prefill | Pie 2,011 · MLX 1,369 · vLLM Metal 1,162 · llama.cpp 510 |
| M2 16 GB | Gemma-4-E4B | Single decode | Pie 30.1 · llama.cpp 25.9 · MLX 24.5 · vLLM Metal 23.1 |
| M2 16 GB | Gemma-4-E4B | Single prefill | Pie 525 · MLX 374 · vLLM Metal 284 · llama.cpp 268 |
| M2 16 GB | Gemma-4-E4B | Batched decode | Pie 110 · vLLM Metal 76.9 · llama.cpp 69.1 · MLX 21.4 |
| M2 16 GB | Gemma-4-E4B | Batched prefill | Pie 586 · MLX 378 |
| M2 16 GB | Gemma-4-12B | Single decode | MLX 10.3 · Pie 9.4 · vLLM Metal 9 |
| M2 16 GB | Gemma-4-12B | Single prefill | Pie 114 · MLX 82.6 · vLLM Metal 57 |
| M2 16 GB | Gemma-4-12B | Batched decode | Pie 27.8 · vLLM Metal 22.5 · MLX 8.1 |
| M2 16 GB | Gemma-4-12B | Batched prefill | Pie 105 · MLX 66 |
| M2 16 GB | Qwen3.5-4B | Single decode | Pie 33.8 · MLX 27.6 · vLLM Metal 26.7 · llama.cpp 26 |
| M2 16 GB | Qwen3.5-4B | Single prefill | Pie 331 · llama.cpp 277 · MLX 242 · vLLM Metal 139 |
| M2 16 GB | Qwen3.5-4B | Batched decode | Pie 47.9 · llama.cpp 43.2 · MLX 23.2 |
| M2 16 GB | Qwen3.5-4B | Batched prefill | Pie 372 · MLX 245 |
| M2 16 GB | Qwen3.5-9B | Single decode | MLX 16.7 · Pie 16.2 · vLLM Metal 16 · llama.cpp 12.9 |
| M2 16 GB | Qwen3.5-9B | Single prefill | Pie 136 · llama.cpp 135 · MLX 131 · vLLM Metal 82.1 |
| M2 16 GB | Qwen3.5-9B | Batched decode | Pie 31 · llama.cpp 25.4 · MLX 12.8 |
| M2 16 GB | Qwen3.5-9B | Batched prefill | Pie 212 · MLX 117 |

### NVIDIA GPU

| Machine | Model | Setup | Engines (tok/s) |
|---|---|---|---|
| B200 180 GB | Qwen3.5-0.8B | Single decode | Pie 675 · SGLang 558 · vLLM 397 · TensorRT-LLM 275 |
| B200 180 GB | Qwen3.5-0.8B | Single prefill | Pie 45,816 · SGLang 17,750 · vLLM 10,914 · TensorRT-LLM 4,764 |
| B200 180 GB | Qwen3.5-0.8B | Batched decode | Pie 84,398 · SGLang 55,409 · vLLM 39,249 · TensorRT-LLM 6,454 |
| B200 180 GB | Qwen3.5-0.8B | Batched prefill | Pie 470,170 · SGLang 439,719 · vLLM 254,530 · TensorRT-LLM 146,073 |
| B200 180 GB | Qwen3-0.6B | Single decode | Pie 969 · SGLang 637 · vLLM 443 · TensorRT-LLM 297 |
| B200 180 GB | Qwen3-0.6B | Single prefill | Pie 59,499 · SGLang 32,206 · vLLM 12,628 · TensorRT-LLM 7,172 |
| B200 180 GB | Qwen3-0.6B | Batched decode | Pie 90,301 · SGLang 56,847 · vLLM 43,917 · TensorRT-LLM 9,524 |
| B200 180 GB | Qwen3-0.6B | Batched prefill | Pie 662,963 · SGLang 377,453 · vLLM 211,925 · TensorRT-LLM 72,704 |
| B200 180 GB | Gemma-4-E4B | Single decode | Pie 290 · vLLM 252 · SGLang 237 · TensorRT-LLM 229 |
| B200 180 GB | Gemma-4-E4B | Single prefill | Pie 36,744 · SGLang 9,256 · vLLM 8,622 · TensorRT-LLM 4,258 |
| B200 180 GB | Gemma-4-E4B | Batched decode | SGLang 40,383 · Pie 37,249 · vLLM 33,247 · TensorRT-LLM 8,754 |
| B200 180 GB | Gemma-4-E4B | Batched prefill | vLLM 126,822 · SGLang 121,701 · Pie 116,759 · TensorRT-LLM 100,267 |
| B200 180 GB | Qwen3.6-27B | Single decode | TensorRT-LLM 103 · Pie 102 · SGLang 102 · vLLM 95.2 |
| B200 180 GB | Qwen3.6-27B | Single prefill | Pie 10,947 · SGLang 8,752 · vLLM 4,872 · TensorRT-LLM 3,601 |
| B200 180 GB | Qwen3.6-27B | Batched decode | Pie 7,765 · vLLM 7,538 · SGLang 6,519 · TensorRT-LLM 6,171 |
| B200 180 GB | Qwen3.6-27B | Batched prefill | vLLM 23,790 · TensorRT-LLM 23,567 · SGLang 21,905 · Pie 21,249 |
| B200 180 GB | GPT-OSS-20B | Single decode | SGLang 569 · Pie 550 · vLLM 456 · TensorRT-LLM 285 |
| B200 180 GB | GPT-OSS-20B | Single prefill | Pie 36,203 · SGLang 30,644 · vLLM 27,768 · TensorRT-LLM 7,475 |
| B200 180 GB | GPT-OSS-20B | Batched decode | vLLM 43,895 · SGLang 38,625 · Pie 35,733 · TensorRT-LLM 11,287 |
| B200 180 GB | GPT-OSS-20B | Batched prefill | vLLM 138,139 · Pie 115,227 · SGLang 94,130 · TensorRT-LLM 79,131 |
| B200 180 GB | GPT-OSS-120B | Single decode | TensorRT-LLM 428 · SGLang 418 · vLLM 397 · Pie 378 |
| B200 180 GB | GPT-OSS-120B | Single prefill | Pie 19,318 · vLLM 18,538 · SGLang 18,177 · TensorRT-LLM 13,497 |
| B200 180 GB | GPT-OSS-120B | Batched decode | vLLM 28,699 · SGLang 24,257 · Pie 20,408 · TensorRT-LLM 11,495 |
| B200 180 GB | GPT-OSS-120B | Batched prefill | vLLM 90,339 · Pie 72,754 · SGLang 62,902 · TensorRT-LLM 62,034 |
| B200 180 GB | Qwen3.6-35B | Single decode | SGLang 396 · Pie 367 · vLLM 325 · TensorRT-LLM 279 |
| B200 180 GB | Qwen3.6-35B | Single prefill | Pie 24,922 · SGLang 13,101 · vLLM 3,924 · TensorRT-LLM 3,257 |
| B200 180 GB | Qwen3.6-35B | Batched decode | vLLM 22,721 · Pie 22,450 · SGLang 18,545 · TensorRT-LLM 10,779 |
| B200 180 GB | Qwen3.6-35B | Batched prefill | vLLM 126,246 · SGLang 112,748 · Pie 107,876 · TensorRT-LLM 81,207 |
| B200 180 GB | Muse-Glimmer-30B | Single decode | Pie 102 · SGLang 95.8 · vLLM 87.9 |
| B200 180 GB | Muse-Glimmer-30B | Single prefill | Pie 14,663 · SGLang 8,779 · vLLM 7,309 |
| B200 180 GB | Muse-Glimmer-30B | Batched decode | Pie 12,979 · vLLM 12,602 · SGLang 7,343 |
| B200 180 GB | Muse-Glimmer-30B | Batched prefill | SGLang 24,352 · vLLM 23,581 · Pie 22,238 |
| B200 180 GB | Gemma-4-26B | Single decode | Pie 341 · SGLang 291 · vLLM 256 · TensorRT-LLM 217 |
| B200 180 GB | Gemma-4-26B | Single prefill | Pie 28,266 · SGLang 7,926 · vLLM 6,521 · TensorRT-LLM 3,699 |
| B200 180 GB | Gemma-4-26B | Batched decode | vLLM 31,100 · Pie 27,712 · SGLang 27,511 · TensorRT-LLM 9,763 |
| B200 180 GB | Gemma-4-26B | Batched prefill | vLLM 114,958 · Pie 114,220 · SGLang 104,786 · TensorRT-LLM 49,721 |
| B200 180 GB | Gemma-4-31B | Single decode | SGLang 82.2 · Pie 81.5 · TensorRT-LLM 80.9 · vLLM 76.9 |
| B200 180 GB | Gemma-4-31B | Single prefill | Pie 12,555 · SGLang 8,174 · vLLM 7,335 · TensorRT-LLM 3,534 |
| B200 180 GB | Gemma-4-31B | Batched decode | vLLM 6,493 · Pie 6,134 · SGLang 6,021 · TensorRT-LLM 4,930 |
| B200 180 GB | Gemma-4-31B | Batched prefill | SGLang 19,932 · vLLM 19,717 · TensorRT-LLM 18,880 · Pie 18,509 |
| B200 180 GB | Qwen3.8-27B | Single decode | TensorRT-LLM 103 · SGLang 102 · Pie 102 · vLLM 94.5 |
| B200 180 GB | Qwen3.8-27B | Single prefill | Pie 11,245 · SGLang 8,779 · vLLM 4,909 · TensorRT-LLM 3,839 |
| B200 180 GB | Qwen3.8-27B | Batched decode | vLLM 7,539 · Pie 6,802 · SGLang 6,540 · TensorRT-LLM 5,549 |
| B200 180 GB | Qwen3.8-27B | Batched prefill | TensorRT-LLM 22,873 · SGLang 22,350 · vLLM 21,806 · Pie 21,218 |
| B200 180 GB | Qwen3.5-9B | Single decode | Pie 284 · SGLang 280 · TensorRT-LLM 275 · vLLM 248 |
| B200 180 GB | Qwen3.5-9B | Single prefill | Pie 26,681 · SGLang 20,861 · TensorRT-LLM 11,563 · vLLM 6,042 |
| B200 180 GB | Qwen3.5-9B | Batched decode | Pie 23,777 · vLLM 22,510 · SGLang 20,434 · TensorRT-LLM 10,027 |
| B200 180 GB | Qwen3.5-9B | Batched prefill | SGLang 73,903 · vLLM 72,879 · Pie 71,320 · TensorRT-LLM 70,860 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Single decode | TensorRT-LLM 573 · SGLang 567 · Pie 562 · vLLM 385 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Single prefill | Pie 63,884 · SGLang 29,290 · vLLM 14,907 · TensorRT-LLM 11,588 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Batched decode | TensorRT-LLM 21,984 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Batched prefill | Pie 217,989 · vLLM 191,574 · SGLang 186,448 · TensorRT-LLM 175,675 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Single decode | Pie 681 · SGLang 636 · vLLM 626 · TensorRT-LLM 580 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Single prefill | Pie 85,375 · SGLang 73,009 · vLLM 68,556 · TensorRT-LLM 16,932 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Batched decode | Pie 31,859 · vLLM 29,507 · SGLang 29,228 · TensorRT-LLM 23,224 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Batched prefill | Pie 290,442 · vLLM 261,470 · SGLang 248,360 · TensorRT-LLM 198,607 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Single decode | Pie 133 · vLLM 117 · SGLang 114 · TensorRT-LLM 105 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Single prefill | Pie 22,255 · vLLM 14,936 · SGLang 11,302 · TensorRT-LLM 7,373 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Batched decode | Pie 14,827 · vLLM 14,451 · SGLang 13,615 · TensorRT-LLM 11,876 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Batched prefill | vLLM 40,618 · Pie 38,977 · TensorRT-LLM 37,114 · SGLang 35,100 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Single decode | Pie 27.7 · TensorRT-LLM 27.2 · SGLang 27.1 · vLLM 26.7 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Single prefill | Pie 5,341 · SGLang 5,210 · TensorRT-LLM 4,707 · vLLM 4,676 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Batched decode | Pie 1,834 · TensorRT-LLM 1,814 · vLLM 1,560 · SGLang 1,260 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Batched prefill | TensorRT-LLM 7,025 · vLLM 6,945 · Pie 6,865 · SGLang 6,731 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Single decode | Pie 280 · vLLM 260 · SGLang 245 · TensorRT-LLM 227 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Single prefill | Pie 24,159 · SGLang 19,338 · vLLM 18,268 · TensorRT-LLM 13,991 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Batched decode | TensorRT-LLM 23,048 · Pie 19,240 · vLLM 17,141 · SGLang 13,874 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Batched prefill | TensorRT-LLM 65,736 · Pie 49,242 · vLLM 41,689 · SGLang 31,920 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Single decode | Pie 206 · vLLM 188 · SGLang 178 · TensorRT-LLM 156 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Single prefill | Pie 12,849 · vLLM 11,919 · TensorRT-LLM 11,822 · SGLang 10,070 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Batched decode | TensorRT-LLM 11,517 · vLLM 8,470 · Pie 7,998 · SGLang 3,673 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Batched prefill | TensorRT-LLM 42,843 · Pie 30,372 · vLLM 27,725 · SGLang 21,200 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Single decode | SGLang 192 · Pie 186 · TensorRT-LLM 179 · vLLM 167 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Single prefill | Pie 12,824 · SGLang 12,686 · vLLM 11,143 · TensorRT-LLM 5,868 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Batched decode | TensorRT-LLM 4,831 · Pie 4,747 · vLLM 3,522 · SGLang 2,701 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Batched prefill | SGLang 41,034 · Pie 40,099 · TensorRT-LLM 39,497 · vLLM 38,140 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Single decode | Pie 27.1 · SGLang 26 · vLLM 25.9 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Single prefill | Pie 6,293 · SGLang 5,484 · vLLM 5,454 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Batched decode | Pie 3,483 · vLLM 3,244 · SGLang 3,158 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Batched prefill | Pie 7,499 · SGLang 6,966 · vLLM 6,958 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Single decode | Pie 154 · SGLang 139 · vLLM 133 · TensorRT-LLM 118 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Single prefill | Pie 14,030 · vLLM 11,111 · SGLang 9,599 · TensorRT-LLM 6,621 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Batched decode | Pie 6,892 · vLLM 6,730 · SGLang 4,927 · TensorRT-LLM 4,730 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Batched prefill | vLLM 38,562 · Pie 37,461 · TensorRT-LLM 35,825 · SGLang 32,018 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Single decode | Pie 23.4 · SGLang 22.7 · vLLM 22.5 · TensorRT-LLM 22.5 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Single prefill | Pie 5,304 · TensorRT-LLM 4,209 · SGLang 4,201 · vLLM 4,139 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Batched decode | Pie 1,088 · vLLM 1,047 · SGLang 889 · TensorRT-LLM 729 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Batched prefill | vLLM 6,088 · Pie 5,965 · TensorRT-LLM 5,939 · SGLang 5,437 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Single decode | Pie 27.9 · SGLang 27.2 · TensorRT-LLM 27.2 · vLLM 26.6 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Single prefill | Pie 6,061 · SGLang 5,553 · vLLM 5,073 · TensorRT-LLM 4,801 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Batched decode | Pie 1,833 · TensorRT-LLM 1,811 · vLLM 1,561 · SGLang 1,264 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Batched prefill | TensorRT-LLM 7,125 · vLLM 6,948 · Pie 6,869 · SGLang 6,805 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Single decode | Pie 88.6 · TensorRT-LLM 86.3 · SGLang 86.2 · vLLM 83.8 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Single prefill | Pie 18,324 · SGLang 14,774 · TensorRT-LLM 12,756 · vLLM 9,927 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Batched decode | Pie 8,394 · TensorRT-LLM 7,854 · vLLM 6,422 · SGLang 6,072 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Batched prefill | vLLM 24,245 · Pie 24,141 · TensorRT-LLM 23,713 · SGLang 22,828 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Single decode | TensorRT-LLM 730 · Pie 724 · SGLang 519 · vLLM 465 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Single prefill | Pie 65,629 · SGLang 19,275 · TensorRT-LLM 15,961 · vLLM 15,449 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Batched decode | Pie 30,630 · TensorRT-LLM 25,679 · SGLang 21,929 · vLLM 21,112 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Batched prefill | Pie 152,028 · TensorRT-LLM 142,701 · SGLang 127,016 · vLLM 98,301 |
| RTX 5090 32 GB | Qwen3-0.6B | Single decode | Pie 769 · TensorRT-LLM 710 · SGLang 708 · vLLM 692 |
| RTX 5090 32 GB | Qwen3-0.6B | Single prefill | Pie 71,535 · SGLang 53,548 · vLLM 29,465 · TensorRT-LLM 21,992 |
| RTX 5090 32 GB | Qwen3-0.6B | Batched decode | Pie 29,583 · SGLang 26,623 · TensorRT-LLM 25,638 · vLLM 25,440 |
| RTX 5090 32 GB | Qwen3-0.6B | Batched prefill | TensorRT-LLM 181,100 · Pie 179,831 · vLLM 140,466 · SGLang 135,364 |
| RTX 5090 32 GB | Gemma-4-E4B | Single decode | Pie 146 · vLLM 140 · SGLang 138 · TensorRT-LLM 132 |
| RTX 5090 32 GB | Gemma-4-E4B | Single prefill | Pie 16,612 · vLLM 13,576 · SGLang 11,174 · TensorRT-LLM 9,338 |
| RTX 5090 32 GB | Gemma-4-E4B | Batched decode | vLLM 9,748 · Pie 9,429 · TensorRT-LLM 4,443 · SGLang 1,453 |
| RTX 5090 32 GB | Gemma-4-E4B | Batched prefill | Pie 24,563 · vLLM 24,181 · TensorRT-LLM 23,879 · SGLang 20,770 |
| RTX 5090 32 GB | GPT-OSS-20B | Single decode | Pie 318 · vLLM 293 · SGLang 279 · TensorRT-LLM 233 |
| RTX 5090 32 GB | GPT-OSS-20B | Single prefill | Pie 19,891 · vLLM 16,812 · SGLang 16,244 · TensorRT-LLM 15,240 |
| RTX 5090 32 GB | GPT-OSS-20B | Batched decode | TensorRT-LLM 16,098 · Pie 11,771 · vLLM 10,893 · SGLang 9,979 |
| RTX 5090 32 GB | GPT-OSS-20B | Batched prefill | TensorRT-LLM 52,472 · Pie 32,873 · vLLM 27,370 · SGLang 23,606 |
| RTX 5090 32 GB | Qwen3.5-9B | Single decode | TensorRT-LLM 98.6 · Pie 98 · SGLang 98 · vLLM 95.8 |
| RTX 5090 32 GB | Qwen3.5-9B | Single prefill | Pie 12,248 · SGLang 11,026 · TensorRT-LLM 10,457 · vLLM 9,412 |
| RTX 5090 32 GB | Qwen3.5-9B | Batched decode | Pie 5,597 · TensorRT-LLM 5,448 · vLLM 4,249 |
| RTX 5090 32 GB | Qwen3.5-9B | Batched prefill | Pie 15,093 · vLLM 14,320 · TensorRT-LLM 14,224 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Single decode | SGLang 444 · TensorRT-LLM 437 · Pie 430 · vLLM 418 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Single prefill | Pie 52,836 · SGLang 22,415 · vLLM 19,021 · TensorRT-LLM 16,945 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Batched decode | Pie 17,703 · TensorRT-LLM 17,227 · vLLM 12,462 · SGLang 11,719 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Batched prefill | Pie 93,699 · TensorRT-LLM 89,324 · SGLang 73,789 · vLLM 60,105 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Single decode | SGLang 482 · Pie 482 · vLLM 473 · TensorRT-LLM 470 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Single prefill | Pie 73,254 · SGLang 42,978 · vLLM 27,066 · TensorRT-LLM 22,853 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Batched decode | Pie 16,795 · TensorRT-LLM 15,826 · vLLM 15,682 · SGLang 15,279 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Batched prefill | Pie 119,505 · TensorRT-LLM 109,821 · SGLang 105,519 · vLLM 104,344 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Single decode | Pie 79.7 · vLLM 76.8 · SGLang 75.7 · TensorRT-LLM 73.9 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Single prefill | Pie 13,530 · vLLM 10,489 · SGLang 10,066 · TensorRT-LLM 9,639 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Batched decode | Pie 6,424 · vLLM 6,100 · TensorRT-LLM 2,948 · SGLang 2,223 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Batched prefill | Pie 16,795 · vLLM 15,887 · TensorRT-LLM 14,882 · SGLang 14,107 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Single decode | Pie 183 · vLLM 174 · SGLang 170 · TensorRT-LLM 140 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Single prefill | TensorRT-LLM 14,148 · Pie 13,893 · vLLM 11,703 · SGLang 10,567 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Batched decode | TensorRT-LLM 8,977 · Pie 7,652 · vLLM 6,851 · SGLang 5,833 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Batched prefill | TensorRT-LLM 27,230 · Pie 20,379 · vLLM 17,659 · SGLang 13,804 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Single decode | TensorRT-LLM 50 · Pie 49.8 · SGLang 49.7 · vLLM 49.4 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Single prefill | Pie 9,139 · SGLang 7,069 · TensorRT-LLM 6,983 · vLLM 6,660 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Batched decode | Pie 3,018 · TensorRT-LLM 2,889 · vLLM 2,267 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Batched prefill | Pie 10,052 · TensorRT-LLM 8,398 · vLLM 8,365 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Single decode | Pie 929 · TensorRT-LLM 713 · SGLang 643 · vLLM 553 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Single prefill | Pie 69,360 · SGLang 25,328 · vLLM 20,707 · TensorRT-LLM 15,841 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Batched decode | Pie 60,127 · SGLang 41,846 · vLLM 32,182 · TensorRT-LLM 27,648 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Batched prefill | vLLM 340,786 · Pie 305,796 · SGLang 210,282 · TensorRT-LLM 185,697 |
| H200 SXM 141 GB | Qwen3-0.6B | Single decode | Pie 888 · SGLang 664 · vLLM 611 · TensorRT-LLM 582 |
| H200 SXM 141 GB | Qwen3-0.6B | Single prefill | Pie 104,679 · SGLang 56,488 · vLLM 40,553 · TensorRT-LLM 21,679 |
| H200 SXM 141 GB | Qwen3-0.6B | Batched decode | Pie 65,059 · SGLang 54,795 · vLLM 53,991 · TensorRT-LLM 31,471 |
| H200 SXM 141 GB | Qwen3-0.6B | Batched prefill | SGLang 434,754 · vLLM 412,407 · Pie 400,528 · TensorRT-LLM 252,916 |
| H200 SXM 141 GB | Gemma-4-E4B | Single decode | Pie 228 · SGLang 191 · vLLM 189 · TensorRT-LLM 162 |
| H200 SXM 141 GB | Gemma-4-E4B | Single prefill | Pie 37,639 · vLLM 25,378 · SGLang 13,493 · TensorRT-LLM 8,769 |
| H200 SXM 141 GB | Gemma-4-E4B | Batched decode | SGLang 24,229 · vLLM 23,987 · Pie 23,721 · TensorRT-LLM 21,491 |
| H200 SXM 141 GB | Gemma-4-E4B | Batched prefill | vLLM 69,777 · Pie 65,371 · SGLang 61,185 · TensorRT-LLM 59,231 |
| H200 SXM 141 GB | Qwen3.6-27B | Single decode | Pie 73 · SGLang 68.5 · TensorRT-LLM 67.9 · vLLM 65.6 |
| H200 SXM 141 GB | Qwen3.6-27B | Single prefill | SGLang 9,642 · Pie 9,342 · vLLM 8,851 · TensorRT-LLM 6,764 |
| H200 SXM 141 GB | Qwen3.6-27B | Batched decode | vLLM 4,369 · Pie 4,368 · TensorRT-LLM 4,104 · SGLang 3,906 |
| H200 SXM 141 GB | Qwen3.6-27B | Batched prefill | vLLM 12,330 · SGLang 12,215 · TensorRT-LLM 11,450 · Pie 11,441 |
| H200 SXM 141 GB | GPT-OSS-20B | Single decode | Pie 438 · vLLM 328 · TensorRT-LLM 324 · SGLang 317 |
| H200 SXM 141 GB | GPT-OSS-20B | Single prefill | vLLM 29,595 · Pie 28,862 · SGLang 25,162 · TensorRT-LLM 10,227 |
| H200 SXM 141 GB | GPT-OSS-20B | Batched decode | Pie 24,792 · SGLang 22,032 · vLLM 22,012 · TensorRT-LLM 20,447 |
| H200 SXM 141 GB | GPT-OSS-20B | Batched prefill | SGLang 66,327 · Pie 60,913 · TensorRT-LLM 60,737 · vLLM 58,424 |
| H200 SXM 141 GB | GPT-OSS-120B | Single decode | Pie 290 · vLLM 225 · TensorRT-LLM 224 · SGLang 216 |
| H200 SXM 141 GB | GPT-OSS-120B | Single prefill | SGLang 13,915 · vLLM 13,488 · Pie 9,874 · TensorRT-LLM 7,125 |
| H200 SXM 141 GB | GPT-OSS-120B | Batched decode | SGLang 12,529 · TensorRT-LLM 11,346 · vLLM 10,981 · Pie 9,688 |
| H200 SXM 141 GB | GPT-OSS-120B | Batched prefill | vLLM 41,652 · SGLang 36,554 · TensorRT-LLM 33,697 · Pie 31,120 |
| H200 SXM 141 GB | Qwen3.6-35B | Single decode | Pie 302 · SGLang 274 · TensorRT-LLM 248 · vLLM 233 |
| H200 SXM 141 GB | Qwen3.6-35B | Single prefill | Pie 21,639 · SGLang 19,159 · vLLM 8,030 · TensorRT-LLM 7,553 |
| H200 SXM 141 GB | Qwen3.6-35B | Batched decode | TensorRT-LLM 13,645 · Pie 12,512 · vLLM 12,114 · SGLang 9,941 |
| H200 SXM 141 GB | Qwen3.6-35B | Batched prefill | SGLang 63,236 · TensorRT-LLM 62,843 · Pie 55,118 · vLLM 55,085 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Single decode | Pie 73.1 · SGLang 65.8 · vLLM 63.6 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Single prefill | SGLang 11,588 · Pie 11,277 · vLLM 11,114 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Batched decode | SGLang 6,810 · vLLM 6,573 · Pie 6,356 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Batched prefill | SGLang 12,971 · Pie 12,922 · vLLM 12,324 |
| H200 SXM 141 GB | Gemma-4-26B | Single decode | Pie 293 · SGLang 234 · vLLM 212 · TensorRT-LLM 165 |
| H200 SXM 141 GB | Gemma-4-26B | Single prefill | Pie 24,493 · vLLM 17,341 · SGLang 11,971 · TensorRT-LLM 8,015 |
| H200 SXM 141 GB | Gemma-4-26B | Batched decode | Pie 15,430 · vLLM 14,330 · SGLang 12,808 · TensorRT-LLM 10,657 |
| H200 SXM 141 GB | Gemma-4-26B | Batched prefill | vLLM 64,793 · TensorRT-LLM 56,762 · SGLang 55,088 · Pie 54,256 |
| H200 SXM 141 GB | Gemma-4-31B | Single decode | Pie 60.6 · SGLang 57.7 · vLLM 54 |
| H200 SXM 141 GB | Gemma-4-31B | Single prefill | vLLM 9,503 · Pie 9,090 · SGLang 8,724 |
| H200 SXM 141 GB | Gemma-4-31B | Batched decode | Pie 3,317 · SGLang 3,272 · vLLM 3,053 |
| H200 SXM 141 GB | Gemma-4-31B | Batched prefill | vLLM 10,398 · Pie 9,902 · SGLang 9,646 |
| H200 SXM 141 GB | Qwen3.8-27B | Single decode | Pie 72.9 · SGLang 70.6 · TensorRT-LLM 67.6 · vLLM 67.3 |
| H200 SXM 141 GB | Qwen3.8-27B | Single prefill | SGLang 10,072 · vLLM 9,919 · Pie 9,222 · TensorRT-LLM 6,502 |
| H200 SXM 141 GB | Qwen3.8-27B | Batched decode | Pie 4,366 · vLLM 4,363 · TensorRT-LLM 4,102 · SGLang 3,905 |
| H200 SXM 141 GB | Qwen3.8-27B | Batched prefill | SGLang 12,287 · vLLM 12,277 · TensorRT-LLM 11,585 · Pie 11,406 |
| H200 SXM 141 GB | Qwen3.5-9B | Single decode | Pie 211 · SGLang 201 · TensorRT-LLM 196 · vLLM 190 |
| H200 SXM 141 GB | Qwen3.5-9B | Single prefill | Pie 26,944 · SGLang 23,042 · vLLM 16,979 · TensorRT-LLM 9,612 |
| H200 SXM 141 GB | Qwen3.5-9B | Batched decode | TensorRT-LLM 15,246 · Pie 14,401 · vLLM 13,785 · SGLang 12,969 |
| H200 SXM 141 GB | Qwen3.5-9B | Batched prefill | vLLM 40,827 · SGLang 40,086 · Pie 38,957 · TensorRT-LLM 37,158 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Single decode | Pie 841 · SGLang 742 · vLLM 655 · TensorRT-LLM 622 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Single prefill | Pie 75,991 · SGLang 28,921 · vLLM 22,931 · TensorRT-LLM 12,971 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Batched decode | Pie 51,807 · SGLang 37,149 · vLLM 34,065 · TensorRT-LLM 11,627 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Batched prefill | Pie 269,828 · vLLM 252,214 · SGLang 219,193 · TensorRT-LLM 143,986 |
| H100 SXM 80 GB | Qwen3-0.6B | Single decode | Pie 802 · SGLang 695 · vLLM 645 · TensorRT-LLM 613 |
| H100 SXM 80 GB | Qwen3-0.6B | Single prefill | Pie 70,306 · SGLang 34,297 · vLLM 27,078 · TensorRT-LLM 18,154 |
| H100 SXM 80 GB | Qwen3-0.6B | Batched decode | SGLang 40,040 · vLLM 37,141 · Pie 34,322 · TensorRT-LLM 11,661 |
| H100 SXM 80 GB | Qwen3-0.6B | Batched prefill | Pie 390,411 · SGLang 375,806 · vLLM 178,333 · TensorRT-LLM 173,238 |
| H100 SXM 80 GB | Gemma-4-E4B | Single decode | Pie 195 · SGLang 166 · vLLM 163 · TensorRT-LLM 146 |
| H100 SXM 80 GB | Gemma-4-E4B | Single prefill | Pie 29,796 · SGLang 12,950 · vLLM 11,085 · TensorRT-LLM 8,045 |
| H100 SXM 80 GB | Gemma-4-E4B | Batched decode | vLLM 19,351 · SGLang 18,527 · Pie 17,349 · TensorRT-LLM 10,616 |
| H100 SXM 80 GB | Gemma-4-E4B | Batched prefill | vLLM 64,417 · Pie 61,472 · SGLang 59,471 · TensorRT-LLM 57,388 |
| H100 SXM 80 GB | Qwen3.6-27B | Single decode | Pie 53.8 · SGLang 52.2 · TensorRT-LLM 51.4 · vLLM 50.6 |
| H100 SXM 80 GB | Qwen3.6-27B | Single prefill | Pie 9,945 · SGLang 9,423 · vLLM 6,686 · TensorRT-LLM 6,486 |
| H100 SXM 80 GB | Qwen3.6-27B | Batched decode | TensorRT-LLM 3,289 · Pie 3,244 · vLLM 2,389 · SGLang 1,598 |
| H100 SXM 80 GB | Qwen3.6-27B | Batched prefill | vLLM 11,837 · SGLang 11,829 · Pie 11,302 · TensorRT-LLM 10,988 |
| H100 SXM 80 GB | GPT-OSS-20B | Single decode | Pie 385 · vLLM 305 · TensorRT-LLM 299 · SGLang 298 |
| H100 SXM 80 GB | GPT-OSS-20B | Single prefill | Pie 22,779 · vLLM 22,379 · SGLang 20,203 · TensorRT-LLM 10,169 |
| H100 SXM 80 GB | GPT-OSS-20B | Batched decode | SGLang 21,360 · vLLM 18,398 · Pie 15,291 · TensorRT-LLM 10,500 |
| H100 SXM 80 GB | GPT-OSS-20B | Batched prefill | vLLM 66,370 · SGLang 63,978 · Pie 59,779 · TensorRT-LLM 58,560 |
| H100 SXM 80 GB | GPT-OSS-120B | Single decode | Pie 263 · vLLM 214 · SGLang 204 · TensorRT-LLM 203 |
| H100 SXM 80 GB | GPT-OSS-120B | Single prefill | SGLang 12,998 · vLLM 11,811 · Pie 10,177 · TensorRT-LLM 6,216 |
| H100 SXM 80 GB | GPT-OSS-120B | Batched decode | TensorRT-LLM 7,349 · vLLM 6,290 · Pie 5,225 · SGLang 3,683 |
| H100 SXM 80 GB | GPT-OSS-120B | Batched prefill | SGLang 37,544 · TensorRT-LLM 34,291 · Pie 31,886 · vLLM 25,417 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Single decode | Pie 51.6 · SGLang 49.4 · vLLM 48.5 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Single prefill | Pie 10,495 · SGLang 10,008 · vLLM 8,002 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Batched decode | SGLang 5,894 · vLLM 5,811 · Pie 5,611 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Batched prefill | SGLang 12,313 · Pie 11,990 · vLLM 11,863 |
| H100 SXM 80 GB | Gemma-4-26B | Single decode | Pie 226 · SGLang 205 · vLLM 190 · TensorRT-LLM 151 |
| H100 SXM 80 GB | Gemma-4-26B | Single prefill | Pie 19,908 · SGLang 11,625 · vLLM 8,991 · TensorRT-LLM 4,848 |
| H100 SXM 80 GB | Gemma-4-26B | Batched decode | vLLM 9,151 · Pie 8,358 · SGLang 7,489 · TensorRT-LLM 5,085 |
| H100 SXM 80 GB | Gemma-4-26B | Batched prefill | vLLM 60,782 · TensorRT-LLM 52,397 · Pie 50,121 · SGLang 47,118 |
| H100 SXM 80 GB | Gemma-4-31B | Single decode | Pie 43.2 · SGLang 42.3 · vLLM 41.5 |
| H100 SXM 80 GB | Gemma-4-31B | Single prefill | Pie 8,619 · SGLang 7,961 · vLLM 6,872 |
| H100 SXM 80 GB | Gemma-4-31B | Batched decode | vLLM 1,317 · SGLang 787 · Pie 695 |
| H100 SXM 80 GB | Gemma-4-31B | Batched prefill | vLLM 10,000 · Pie 9,555 · SGLang 9,461 |
| H100 SXM 80 GB | Qwen3.8-27B | Single decode | Pie 52.6 · SGLang 51.9 · TensorRT-LLM 51.4 · vLLM 48.5 |
| H100 SXM 80 GB | Qwen3.8-27B | Single prefill | Pie 9,556 · SGLang 9,066 · TensorRT-LLM 6,706 · vLLM 6,343 |
| H100 SXM 80 GB | Qwen3.8-27B | Batched decode | TensorRT-LLM 3,286 · Pie 3,011 · vLLM 2,395 · SGLang 1,704 |
| H100 SXM 80 GB | Qwen3.8-27B | Batched prefill | vLLM 12,067 · TensorRT-LLM 11,768 · SGLang 11,371 · Pie 11,150 |
| H100 SXM 80 GB | Qwen3.5-9B | Single decode | Pie 165 · SGLang 157 · TensorRT-LLM 155 · vLLM 150 |
| H100 SXM 80 GB | Qwen3.5-9B | Single prefill | Pie 22,342 · SGLang 19,806 · TensorRT-LLM 11,078 · vLLM 10,052 |
| H100 SXM 80 GB | Qwen3.5-9B | Batched decode | Pie 13,843 · vLLM 12,182 · SGLang 11,934 · TensorRT-LLM 10,885 |
| H100 SXM 80 GB | Qwen3.5-9B | Batched prefill | vLLM 39,385 · SGLang 39,172 · TensorRT-LLM 38,538 · Pie 38,040 |
| L40S 46 GB | Qwen3.5-0.8B | Single decode | Pie 381 · SGLang 380 · TensorRT-LLM 378 · vLLM 357 |
| L40S 46 GB | Qwen3.5-0.8B | Single prefill | Pie 54,756 · SGLang 27,600 · vLLM 17,299 · TensorRT-LLM 10,602 |
| L40S 46 GB | Qwen3.5-0.8B | Batched decode | TensorRT-LLM 11,314 |
| L40S 46 GB | Qwen3.5-0.8B | Batched prefill | Pie 107,529 · SGLang 102,654 · TensorRT-LLM 83,972 · vLLM 50,178 |
| L40S 46 GB | Qwen3-0.6B | Single decode | Pie 418 · SGLang 415 · vLLM 405 · TensorRT-LLM 398 |
| L40S 46 GB | Qwen3-0.6B | Single prefill | Pie 71,065 · SGLang 55,591 · vLLM 34,553 · TensorRT-LLM 15,391 |
| L40S 46 GB | Qwen3-0.6B | Batched decode | Pie 15,940 · vLLM 15,785 · SGLang 15,098 · TensorRT-LLM 12,600 |
| L40S 46 GB | Qwen3-0.6B | Batched prefill | Pie 152,452 · vLLM 152,450 · SGLang 141,947 · TensorRT-LLM 135,406 |
| L40S 46 GB | Gemma-4-E4B | Single decode | Pie 70 · vLLM 69.7 · SGLang 67.9 · TensorRT-LLM 66 |
| L40S 46 GB | Gemma-4-E4B | Single prefill | Pie 15,492 · vLLM 14,639 · SGLang 13,777 · TensorRT-LLM 6,631 |
| L40S 46 GB | Gemma-4-E4B | Batched decode | vLLM 6,379 · Pie 6,337 · SGLang 6,034 · TensorRT-LLM 5,506 |
| L40S 46 GB | Gemma-4-E4B | Batched prefill | vLLM 20,657 · Pie 19,739 · TensorRT-LLM 19,553 · SGLang 18,160 |
| L40S 46 GB | GPT-OSS-20B | Single decode | Pie 162 · vLLM 158 · SGLang 137 |
| L40S 46 GB | GPT-OSS-20B | Single prefill | Pie 18,678 · vLLM 16,734 · SGLang 639 |
| L40S 46 GB | GPT-OSS-20B | Batched prefill | Pie 27,648 · vLLM 23,099 · SGLang 1,066 |
| L40S 46 GB | Qwen3.5-9B | Single decode | Pie 45.3 · TensorRT-LLM 45 · SGLang 44.9 · vLLM 44.5 |
| L40S 46 GB | Qwen3.5-9B | Single prefill | SGLang 9,305 · vLLM 9,091 · Pie 8,933 · TensorRT-LLM 7,641 |
| L40S 46 GB | Qwen3.5-9B | Batched decode | Pie 3,571 · TensorRT-LLM 3,521 · vLLM 2,933 · SGLang 2,620 |
| L40S 46 GB | Qwen3.5-9B | Batched prefill | vLLM 12,587 · SGLang 12,001 · TensorRT-LLM 11,554 · Pie 10,951 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Single decode | SGLang 490 · Pie 470 · vLLM 462 · TensorRT-LLM 459 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Single prefill | Pie 56,365 · SGLang 25,522 · vLLM 15,752 · TensorRT-LLM 10,261 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Batched decode | TensorRT-LLM 13,359 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Batched prefill | Pie 94,322 · TensorRT-LLM 86,157 · SGLang 67,051 · vLLM 27,981 |
| RTX 4090 24 GB | Qwen3-0.6B | Single decode | Pie 482 · TensorRT-LLM 472 · SGLang 464 · vLLM 441 |
| RTX 4090 24 GB | Qwen3-0.6B | Single prefill | Pie 61,808 · SGLang 39,689 · vLLM 24,408 · TensorRT-LLM 13,446 |
| RTX 4090 24 GB | Qwen3-0.6B | Batched decode | vLLM 18,108 · SGLang 17,243 · Pie 16,473 · TensorRT-LLM 15,168 |
| RTX 4090 24 GB | Qwen3-0.6B | Batched prefill | vLLM 125,628 · TensorRT-LLM 121,096 · Pie 114,619 · SGLang 114,218 |
| RTX 4090 24 GB | Gemma-4-E4B | Single decode | Pie 78.8 · vLLM 78.6 · SGLang 76.7 |
| RTX 4090 24 GB | Gemma-4-E4B | Single prefill | Pie 12,922 · vLLM 11,880 · SGLang 9,562 |
| RTX 4090 24 GB | Gemma-4-E4B | Batched decode | vLLM 5,841 · Pie 4,566 · SGLang 3,675 |
| RTX 4090 24 GB | Gemma-4-E4B | Batched prefill | vLLM 17,175 · Pie 16,776 · SGLang 15,330 |
| RTX 4090 24 GB | GPT-OSS-20B | Single decode | Pie 182 · vLLM 176 · SGLang 150 |
| RTX 4090 24 GB | GPT-OSS-20B | Single prefill | Pie 14,393 · vLLM 12,569 · SGLang 806 |
| RTX 4090 24 GB | GPT-OSS-20B | Batched prefill | Pie 20,423 · vLLM 19,483 · SGLang 1,529 |
| RTX 4090 24 GB | Qwen3.5-9B | Single decode | SGLang 57.1 · vLLM 56.6 · Pie 54.9 |
| RTX 4090 24 GB | Qwen3.5-9B | Single prefill | SGLang 7,932 · vLLM 7,291 · Pie 6,641 |
| RTX 4090 24 GB | Qwen3.5-9B | Batched decode | vLLM 810 · SGLang 727 · Pie 465 |
| RTX 4090 24 GB | Qwen3.5-9B | Batched prefill | vLLM 10,124 · SGLang 9,789 · Pie 7,952 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Single decode | Pie 527 · SGLang 517 · TensorRT-LLM 482 · vLLM 457 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Single prefill | Pie 41,156 · SGLang 20,161 · vLLM 14,213 · TensorRT-LLM 12,735 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Batched decode | Pie 28,011 · SGLang 20,756 · TensorRT-LLM 18,708 · vLLM 16,682 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Batched prefill | Pie 121,257 · TensorRT-LLM 110,830 · SGLang 93,624 · vLLM 50,468 |
| A100 SXM 80 GB | Qwen3-0.6B | Single decode | SGLang 518 · Pie 515 · vLLM 482 · TensorRT-LLM 441 |
| A100 SXM 80 GB | Qwen3-0.6B | Single prefill | Pie 60,557 · SGLang 45,587 · vLLM 28,642 · TensorRT-LLM 19,416 |
| A100 SXM 80 GB | Qwen3-0.6B | Batched decode | Pie 26,067 · SGLang 24,483 · vLLM 24,243 · TensorRT-LLM 22,263 |
| A100 SXM 80 GB | Qwen3-0.6B | Batched prefill | Pie 163,016 · TensorRT-LLM 154,723 · SGLang 146,912 · vLLM 133,029 |
| A100 SXM 80 GB | Gemma-4-E4B | Single decode | Pie 128 · vLLM 112 · SGLang 107 · TensorRT-LLM 94.6 |
| A100 SXM 80 GB | Gemma-4-E4B | Single prefill | Pie 17,592 · vLLM 14,396 · SGLang 9,281 · TensorRT-LLM 7,470 |
| A100 SXM 80 GB | Gemma-4-E4B | Batched decode | vLLM 10,539 · Pie 9,494 · SGLang 9,162 · TensorRT-LLM 8,998 |
| A100 SXM 80 GB | Gemma-4-E4B | Batched prefill | vLLM 24,724 · Pie 23,997 · TensorRT-LLM 23,781 · SGLang 14,922 |
| A100 SXM 80 GB | Qwen3.6-27B | Single decode | Pie 31.2 · SGLang 29.4 · TensorRT-LLM 29.2 · vLLM 28.8 |
| A100 SXM 80 GB | Qwen3.6-27B | Single prefill | Pie 3,763 · SGLang 3,601 · TensorRT-LLM 3,469 · vLLM 3,364 |
| A100 SXM 80 GB | Qwen3.6-27B | Batched decode | TensorRT-LLM 1,570 · Pie 1,544 · vLLM 1,157 · SGLang 828 |
| A100 SXM 80 GB | Qwen3.6-27B | Batched prefill | SGLang 4,470 · TensorRT-LLM 4,415 · Pie 4,364 · vLLM 4,256 |
| A100 SXM 80 GB | GPT-OSS-20B | Single decode | Pie 257 · vLLM 238 · SGLang 217 |
| A100 SXM 80 GB | GPT-OSS-20B | Single prefill | Pie 15,977 · vLLM 13,818 · SGLang 11,657 |
| A100 SXM 80 GB | GPT-OSS-20B | Batched decode | Pie 9,277 · vLLM 8,807 · SGLang 7,450 |
| A100 SXM 80 GB | GPT-OSS-20B | Batched prefill | Pie 24,086 · vLLM 20,222 · SGLang 16,265 |
| A100 SXM 80 GB | GPT-OSS-120B | Single decode | Pie 182 · vLLM 166 · SGLang 150 |
| A100 SXM 80 GB | GPT-OSS-120B | Single prefill | Pie 6,972 · vLLM 6,930 · SGLang 6,104 |
| A100 SXM 80 GB | GPT-OSS-120B | Batched decode | vLLM 2,598 · Pie 2,177 · SGLang 1,117 |
| A100 SXM 80 GB | GPT-OSS-120B | Batched prefill | Pie 12,262 · vLLM 10,017 · SGLang 9,320 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Single decode | Pie 31.1 · SGLang 28.7 · vLLM 28.7 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Single prefill | Pie 3,538 · SGLang 3,470 · vLLM 3,299 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Batched decode | Pie 2,490 · SGLang 2,354 · vLLM 2,345 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Batched prefill | SGLang 4,500 · vLLM 4,439 · Pie 4,287 |
| A100 SXM 80 GB | Gemma-4-26B | Single decode | Pie 148 · SGLang 131 · vLLM 130 · TensorRT-LLM 104 |
| A100 SXM 80 GB | Gemma-4-26B | Single prefill | Pie 13,410 · vLLM 9,836 · SGLang 7,577 · TensorRT-LLM 6,782 |
| A100 SXM 80 GB | Gemma-4-26B | Batched decode | vLLM 4,701 · Pie 4,492 · SGLang 3,791 · TensorRT-LLM 3,239 |
| A100 SXM 80 GB | Gemma-4-26B | Batched prefill | TensorRT-LLM 22,818 · Pie 21,647 · vLLM 20,449 · SGLang 11,783 |
| A100 SXM 80 GB | Gemma-4-31B | Single decode | Pie 25.8 · SGLang 24.4 · vLLM 23.9 |
| A100 SXM 80 GB | Gemma-4-31B | Single prefill | Pie 3,189 · vLLM 3,169 · SGLang 2,096 |
| A100 SXM 80 GB | Gemma-4-31B | Batched decode | vLLM 638 · Pie 466 · SGLang 407 |
| A100 SXM 80 GB | Gemma-4-31B | Batched prefill | Pie 3,620 · vLLM 3,585 · SGLang 2,363 |
| A100 SXM 80 GB | Qwen3.8-27B | Single decode | Pie 31.3 · SGLang 29.5 · TensorRT-LLM 29.2 · vLLM 29 |
| A100 SXM 80 GB | Qwen3.8-27B | Single prefill | Pie 3,606 · SGLang 3,510 · vLLM 3,485 · TensorRT-LLM 3,436 |
| A100 SXM 80 GB | Qwen3.8-27B | Batched decode | TensorRT-LLM 1,573 · Pie 1,542 · vLLM 1,156 · SGLang 825 |
| A100 SXM 80 GB | Qwen3.8-27B | Batched prefill | TensorRT-LLM 4,408 · SGLang 4,118 · Pie 4,083 · vLLM 3,977 |
| A100 SXM 80 GB | Qwen3.5-9B | Single decode | Pie 95 · SGLang 89.9 · TensorRT-LLM 88.9 · vLLM 88.9 |
| A100 SXM 80 GB | Qwen3.5-9B | Single prefill | Pie 11,199 · SGLang 10,823 · vLLM 10,094 · TensorRT-LLM 9,285 |
| A100 SXM 80 GB | Qwen3.5-9B | Batched decode | TensorRT-LLM 5,937 · Pie 5,544 · vLLM 5,219 · SGLang 4,944 |
| A100 SXM 80 GB | Qwen3.5-9B | Batched prefill | SGLang 15,182 · vLLM 14,682 · TensorRT-LLM 14,515 · Pie 14,345 |
| A40 48 GB | Qwen3.5-0.8B | Single decode | SGLang 292 · Pie 285 · TensorRT-LLM 282 · vLLM 280 |
| A40 48 GB | Qwen3.5-0.8B | Single prefill | Pie 24,869 · SGLang 19,181 · vLLM 10,666 · TensorRT-LLM 9,341 |
| A40 48 GB | Qwen3.5-0.8B | Batched decode | Pie 11,295 · vLLM 8,557 · SGLang 8,277 · TensorRT-LLM 7,934 |
| A40 48 GB | Qwen3.5-0.8B | Batched prefill | Pie 57,102 · TensorRT-LLM 52,794 · SGLang 50,291 · vLLM 48,465 |
| A40 48 GB | Qwen3-0.6B | Single decode | Pie 340 · SGLang 325 · vLLM 316 · TensorRT-LLM 291 |
| A40 48 GB | Qwen3-0.6B | Single prefill | Pie 33,643 · SGLang 26,093 · vLLM 18,365 · TensorRT-LLM 13,701 |
| A40 48 GB | Qwen3-0.6B | Batched decode | Pie 11,383 · vLLM 10,604 · SGLang 10,350 · TensorRT-LLM 8,483 |
| A40 48 GB | Qwen3-0.6B | Batched prefill | Pie 77,990 · TensorRT-LLM 73,093 · vLLM 69,917 · SGLang 61,985 |
| A40 48 GB | Gemma-4-E4B | Single decode | Pie 57.9 · vLLM 51.5 · SGLang 50.2 · TensorRT-LLM 48.2 |
| A40 48 GB | Gemma-4-E4B | Single prefill | Pie 8,066 · SGLang 6,717 · vLLM 6,536 · TensorRT-LLM 6,290 |
| A40 48 GB | Gemma-4-E4B | Batched decode | Pie 4,462 · vLLM 4,367 · SGLang 4,039 · TensorRT-LLM 3,941 |
| A40 48 GB | Gemma-4-E4B | Batched prefill | vLLM 11,584 · Pie 11,180 · TensorRT-LLM 10,631 · SGLang 8,954 |
| A40 48 GB | GPT-OSS-20B | Single decode | Pie 129 · vLLM 123 · SGLang 120 |
| A40 48 GB | GPT-OSS-20B | Single prefill | Pie 7,941 · vLLM 7,625 · SGLang 6,390 |
| A40 48 GB | GPT-OSS-20B | Batched decode | vLLM 4,477 · Pie 3,761 · SGLang 3,330 |
| A40 48 GB | GPT-OSS-20B | Batched prefill | Pie 11,893 · vLLM 11,573 · SGLang 7,937 |
| A40 48 GB | Qwen3.5-9B | Single decode | Pie 36.5 · SGLang 33.2 · TensorRT-LLM 33.1 · vLLM 32.9 |
| A40 48 GB | Qwen3.5-9B | Single prefill | Pie 5,596 · SGLang 5,152 · TensorRT-LLM 4,928 · vLLM 4,334 |
| A40 48 GB | Qwen3.5-9B | Batched decode | Pie 2,522 · TensorRT-LLM 2,338 · vLLM 1,973 · SGLang 1,827 |
| A40 48 GB | Qwen3.5-9B | Batched prefill | SGLang 6,534 · Pie 6,492 · vLLM 6,486 · TensorRT-LLM 6,315 |

### AMD GPU

| Machine | Model | Setup | Engines (tok/s) |
|---|---|---|---|
| MI355X 288 GB | Qwen3.5-0.8B | Single decode | Pie 751 · SGLang 718 · vLLM 546 |
| MI355X 288 GB | Qwen3.5-0.8B | Single prefill | Pie 80,944 · vLLM 24,473 · SGLang 20,517 |
| MI355X 288 GB | Qwen3.5-0.8B | Batched decode | SGLang 58,959 · Pie 41,995 · vLLM 32,585 |
| MI355X 288 GB | Qwen3.5-0.8B | Batched prefill | Pie 323,319 · SGLang 308,832 · vLLM 88,270 |
| MI355X 288 GB | Qwen3-0.6B | Single decode | SGLang 665 · vLLM 631 · Pie 558 |
| MI355X 288 GB | Qwen3-0.6B | Single prefill | Pie 91,340 · vLLM 35,877 · SGLang 34,159 |
| MI355X 288 GB | Qwen3-0.6B | Batched decode | SGLang 64,176 · Pie 59,280 · vLLM 50,204 |
| MI355X 288 GB | Qwen3-0.6B | Batched prefill | SGLang 495,664 · Pie 450,305 · vLLM 146,985 |
| MI355X 288 GB | Gemma-4-E4B | Single decode | Pie 235 · vLLM 210 · SGLang 176 |
| MI355X 288 GB | Gemma-4-E4B | Single prefill | Pie 36,428 · vLLM 21,190 · SGLang 15,262 |
| MI355X 288 GB | Gemma-4-E4B | Batched decode | Pie 33,242 · vLLM 27,777 · SGLang 27,269 |
| MI355X 288 GB | Gemma-4-E4B | Batched prefill | Pie 106,223 · SGLang 101,060 · vLLM 81,764 |
| MI355X 288 GB | Qwen3.6-27B | Single decode | Pie 93 · vLLM 84.3 · SGLang 82.7 |
| MI355X 288 GB | Qwen3.6-27B | Single prefill | Pie 11,244 · vLLM 10,554 · SGLang 9,606 |
| MI355X 288 GB | Qwen3.6-27B | Batched decode | vLLM 6,267 · SGLang 6,204 · Pie 5,706 |
| MI355X 288 GB | Qwen3.6-27B | Batched prefill | SGLang 23,034 · Pie 20,333 · vLLM 20,078 |
| MI355X 288 GB | GPT-OSS-20B | Single decode | SGLang 444 · vLLM 366 · Pie 358 |
| MI355X 288 GB | GPT-OSS-20B | Single prefill | vLLM 51,812 · Pie 30,269 · SGLang 18,973 |
| MI355X 288 GB | GPT-OSS-20B | Batched decode | SGLang 43,649 · vLLM 30,460 · Pie 22,617 |
| MI355X 288 GB | GPT-OSS-20B | Batched prefill | SGLang 133,334 · vLLM 80,988 · Pie 68,276 |
| MI355X 288 GB | GPT-OSS-120B | Single decode | SGLang 321 · vLLM 247 · Pie 245 |
| MI355X 288 GB | GPT-OSS-120B | Single prefill | vLLM 30,518 · Pie 17,185 · SGLang 11,863 |
| MI355X 288 GB | GPT-OSS-120B | Batched decode | SGLang 30,132 · vLLM 21,092 · Pie 13,619 |
| MI355X 288 GB | GPT-OSS-120B | Batched prefill | SGLang 116,011 · vLLM 53,707 · Pie 42,272 |
| MI355X 288 GB | Qwen3.6-35B | Single decode | SGLang 276 · Pie 273 · vLLM 169 |
| MI355X 288 GB | Qwen3.6-35B | Single prefill | Pie 22,703 · SGLang 9,868 · vLLM 8,539 |
| MI355X 288 GB | Qwen3.6-35B | Batched decode | SGLang 17,564 · vLLM 13,895 · Pie 11,278 |
| MI355X 288 GB | Qwen3.6-35B | Batched prefill | SGLang 90,253 · Pie 72,563 · vLLM 33,737 |
| MI355X 288 GB | Muse-Glimmer-30B | Single decode | Pie 85 · vLLM 82.5 · SGLang 80.3 |
| MI355X 288 GB | Muse-Glimmer-30B | Single prefill | vLLM 14,062 · Pie 13,478 · SGLang 12,736 |
| MI355X 288 GB | Muse-Glimmer-30B | Batched decode | Pie 11,671 · SGLang 11,665 · vLLM 11,650 |
| MI355X 288 GB | Muse-Glimmer-30B | Batched prefill | Pie 24,322 · SGLang 24,014 · vLLM 21,416 |
| MI355X 288 GB | Gemma-4-26B | Single decode | Pie 216 · vLLM 186 · SGLang 186 |
| MI355X 288 GB | Gemma-4-26B | Single prefill | Pie 24,712 · SGLang 15,823 · vLLM 15,106 |
| MI355X 288 GB | Gemma-4-26B | Batched decode | vLLM 22,488 · Pie 20,411 · SGLang 15,253 |
| MI355X 288 GB | Gemma-4-26B | Batched prefill | Pie 71,822 · SGLang 67,646 · vLLM 62,748 |
| MI355X 288 GB | Gemma-4-31B | Single decode | Pie 80.4 · vLLM 73.9 · SGLang 72.2 |
| MI355X 288 GB | Gemma-4-31B | Single prefill | Pie 12,234 · vLLM 12,037 · SGLang 11,886 |
| MI355X 288 GB | Gemma-4-31B | Batched decode | vLLM 5,965 · Pie 4,842 · SGLang 3,685 |
| MI355X 288 GB | Gemma-4-31B | Batched prefill | SGLang 18,591 · Pie 18,210 · vLLM 17,599 |
| MI355X 288 GB | Qwen3.8-27B | Single decode | Pie 93.1 · vLLM 84.2 · SGLang 82.7 |
| MI355X 288 GB | Qwen3.8-27B | Single prefill | Pie 11,326 · vLLM 10,430 · SGLang 9,564 |
| MI355X 288 GB | Qwen3.8-27B | Batched decode | vLLM 6,275 · SGLang 6,170 · Pie 5,727 |
| MI355X 288 GB | Qwen3.8-27B | Batched prefill | SGLang 20,599 · vLLM 19,805 · Pie 19,559 |
| MI355X 288 GB | Qwen3.5-9B | Single decode | Pie 252 · SGLang 233 · vLLM 217 |
| MI355X 288 GB | Qwen3.5-9B | Single prefill | Pie 30,068 · vLLM 15,707 · SGLang 15,151 |
| MI355X 288 GB | Qwen3.5-9B | Batched decode | SGLang 19,432 · vLLM 18,307 · Pie 16,037 |
| MI355X 288 GB | Qwen3.5-9B | Batched prefill | SGLang 73,646 · Pie 70,165 · vLLM 62,747 |
| MI350X 288 GB | Qwen3.5-0.8B | Single decode | Pie 714 · SGLang 649 · vLLM 520 |
| MI350X 288 GB | Qwen3.5-0.8B | Single prefill | Pie 67,237 · vLLM 23,457 · SGLang 19,527 |
| MI350X 288 GB | Qwen3-0.6B | Single decode | SGLang 617 · vLLM 602 · Pie 539 |
| MI350X 288 GB | Qwen3-0.6B | Single prefill | Pie 83,807 · vLLM 33,787 · SGLang 32,997 |
| MI350X 288 GB | Gemma-4-E4B | Single decode | Pie 228 · vLLM 204 · SGLang 169 |
| MI350X 288 GB | Gemma-4-E4B | Single prefill | Pie 32,734 · vLLM 18,815 · SGLang 14,895 |
| MI350X 288 GB | Qwen3.6-27B | Single decode | Pie 90.9 · vLLM 82.3 · SGLang 80.2 |
| MI350X 288 GB | Qwen3.6-27B | Single prefill | vLLM 10,066 · Pie 9,753 · SGLang 9,221 |
| MI350X 288 GB | GPT-OSS-20B | Single decode | SGLang 422 · vLLM 350 · Pie 341 |
| MI350X 288 GB | GPT-OSS-20B | Single prefill | vLLM 46,013 · Pie 27,206 · SGLang 17,014 |
| MI350X 288 GB | GPT-OSS-120B | Single decode | SGLang 306 · vLLM 236 · Pie 233 |
| MI350X 288 GB | GPT-OSS-120B | Single prefill | vLLM 25,435 · Pie 15,228 · SGLang 11,543 |
| MI350X 288 GB | Qwen3.6-35B | Single decode | SGLang 265 · Pie 264 · vLLM 162 |
| MI350X 288 GB | Qwen3.6-35B | Single prefill | Pie 20,896 · SGLang 9,465 · vLLM 8,496 |
| MI350X 288 GB | Muse-Glimmer-30B | Single decode | Pie 82.7 · vLLM 81 · SGLang 77.9 |
| MI350X 288 GB | Muse-Glimmer-30B | Single prefill | vLLM 12,071 · Pie 11,603 · SGLang 11,060 |
| MI350X 288 GB | Gemma-4-26B | Single decode | Pie 206 · vLLM 181 · SGLang 179 |
| MI350X 288 GB | Gemma-4-26B | Single prefill | Pie 22,879 · vLLM 17,483 · SGLang 16,370 |
| MI350X 288 GB | Gemma-4-31B | Single decode | Pie 78.8 · vLLM 72.3 · SGLang 70.3 |
| MI350X 288 GB | Gemma-4-31B | Single prefill | vLLM 10,181 · Pie 10,109 · SGLang 9,937 |
| MI350X 288 GB | Qwen3.8-27B | Single decode | Pie 91 · vLLM 82.4 · SGLang 80.4 |
| MI350X 288 GB | Qwen3.8-27B | Single prefill | vLLM 10,164 · Pie 9,755 · SGLang 9,635 |
| MI350X 288 GB | Qwen3.5-9B | Single decode | Pie 246 · vLLM 211 · SGLang 190 |
| MI350X 288 GB | Qwen3.5-9B | Single prefill | Pie 27,697 · vLLM 14,814 · SGLang 14,085 |
| MI300X 192 GB | Qwen3.5-0.8B | Single decode | Pie 673 · SGLang 572 · vLLM 507 |
| MI300X 192 GB | Qwen3.5-0.8B | Single prefill | Pie 42,602 · SGLang 14,059 · vLLM 12,914 |
| MI300X 192 GB | Qwen3.5-0.8B | Batched decode | Pie 30,823 · SGLang 24,012 · vLLM 18,347 |
| MI300X 192 GB | Qwen3.5-0.8B | Batched prefill | Pie 98,705 · SGLang 90,331 · vLLM 40,284 |
| MI300X 192 GB | Qwen3-0.6B | Single decode | vLLM 676 · SGLang 541 · Pie 527 |
| MI300X 192 GB | Qwen3-0.6B | Single prefill | Pie 50,202 · SGLang 19,920 · vLLM 19,633 |
| MI300X 192 GB | Qwen3-0.6B | Batched decode | SGLang 32,132 · vLLM 28,108 · Pie 26,936 |
| MI300X 192 GB | Qwen3-0.6B | Batched prefill | Pie 176,715 · SGLang 151,359 · vLLM 66,983 |
| MI300X 192 GB | Gemma-4-E4B | Single decode | Pie 198 · vLLM 194 |
| MI300X 192 GB | Gemma-4-E4B | Single prefill | Pie 15,259 · vLLM 11,773 |
| MI300X 192 GB | Gemma-4-E4B | Batched decode | Pie 11,449 · vLLM 10,608 · SGLang 9,901 |
| MI300X 192 GB | Gemma-4-E4B | Batched prefill | Pie 43,380 · vLLM 32,117 · SGLang 9,008 |
| MI300X 192 GB | Qwen3.6-27B | Single decode | Pie 67.9 · vLLM 64.7 · SGLang 53.9 |
| MI300X 192 GB | Qwen3.6-27B | Single prefill | SGLang 5,523 · vLLM 5,310 · Pie 4,007 |
| MI300X 192 GB | Qwen3.6-27B | Batched decode | vLLM 2,768 · SGLang 2,727 |
| MI300X 192 GB | Qwen3.6-27B | Batched prefill | SGLang 8,708 · vLLM 8,381 |
| MI300X 192 GB | GPT-OSS-20B | Single decode | Pie 304 · vLLM 292 |
| MI300X 192 GB | GPT-OSS-20B | Single prefill | Pie 17,080 · vLLM 10,868 |
| MI300X 192 GB | GPT-OSS-20B | Batched decode | Pie 9,914 · vLLM 7,061 |
| MI300X 192 GB | GPT-OSS-20B | Batched prefill | Pie 49,237 · vLLM 19,115 |
| MI300X 192 GB | GPT-OSS-120B | Single decode | Pie 206 · vLLM 202 |
| MI300X 192 GB | GPT-OSS-120B | Single prefill | Pie 10,679 · vLLM 5,733 |
| MI300X 192 GB | Qwen3.6-35B | Single decode | Pie 239 · SGLang 216 · vLLM 159 |
| MI300X 192 GB | Qwen3.6-35B | Single prefill | Pie 12,837 · SGLang 6,959 · vLLM 5,699 |
| MI300X 192 GB | Qwen3.6-35B | Batched decode | Pie 7,587 · SGLang 7,464 · vLLM 6,308 |
| MI300X 192 GB | Qwen3.6-35B | Batched prefill | Pie 38,645 · SGLang 23,035 · vLLM 19,437 |
| MI300X 192 GB | Muse-Glimmer-30B | Single decode | Pie 63.9 · vLLM 61.1 · SGLang 51.8 |
| MI300X 192 GB | Muse-Glimmer-30B | Single prefill | vLLM 5,907 · SGLang 5,871 · Pie 4,703 |
| MI300X 192 GB | Muse-Glimmer-30B | Batched decode | Pie 3,853 |
| MI300X 192 GB | Muse-Glimmer-30B | Batched prefill | vLLM 9,048 · Pie 9,023 |
| MI300X 192 GB | Gemma-4-26B | Single decode | Pie 180 · vLLM 179 · SGLang 144 |
| MI300X 192 GB | Gemma-4-26B | Single prefill | Pie 14,101 · vLLM 10,968 · SGLang 9,872 |
| MI300X 192 GB | Gemma-4-26B | Batched decode | vLLM 7,286 · Pie 7,071 · SGLang 6,969 |
| MI300X 192 GB | Gemma-4-26B | Batched prefill | Pie 36,554 · SGLang 14,543 · vLLM 11,497 |
| MI300X 192 GB | Gemma-4-31B | Single decode | Pie 57.7 · vLLM 56.7 |
| MI300X 192 GB | Gemma-4-31B | Single prefill | vLLM 5,229 · Pie 4,094 |
| MI300X 192 GB | Gemma-4-31B | Batched decode | Pie 2,786 · vLLM 2,440 |
| MI300X 192 GB | Gemma-4-31B | Batched prefill | Pie 7,886 · vLLM 7,484 |
| MI300X 192 GB | Qwen3.8-27B | Single decode | Pie 67.8 · vLLM 64.2 · SGLang 52.9 |
| MI300X 192 GB | Qwen3.8-27B | Single prefill | SGLang 5,391 · vLLM 5,134 · Pie 4,002 |
| MI300X 192 GB | Qwen3.8-27B | Batched decode | Pie 3,186 · vLLM 3,087 · SGLang 2,928 |
| MI300X 192 GB | Qwen3.8-27B | Batched prefill | SGLang 9,160 · Pie 8,832 · vLLM 8,013 |

### Google TPU

| Machine | Model | Setup | Engines (tok/s) |
|---|---|---|---|
| TPU v6e 32 GB | Qwen3.5-0.8B | Single decode | Pie 688 · vLLM 348 · SGLang 214 |
| TPU v6e 32 GB | Qwen3.5-0.8B | Single prefill | vLLM 49,984 · Pie 49,751 · SGLang 28,941 |
| TPU v6e 32 GB | Qwen3.5-0.8B | Batched decode | vLLM 16,189 · Pie 14,635 · SGLang 5,464 |
| TPU v6e 32 GB | Qwen3.5-0.8B | Batched prefill | vLLM 140,661 · Pie 116,810 · SGLang 116,589 |
| TPU v6e 32 GB | Qwen3-0.6B | Single decode | Pie 694 · vLLM 421 · SGLang 329 |
| TPU v6e 32 GB | Qwen3-0.6B | Single prefill | vLLM 61,879 · Pie 58,756 · SGLang 38,262 |
| TPU v6e 32 GB | Qwen3-0.6B | Batched decode | vLLM 12,086 · Pie 11,620 · SGLang 7,225 |
| TPU v6e 32 GB | Qwen3-0.6B | Batched prefill | Pie 236,161 · vLLM 230,690 · SGLang 112,732 |
| TPU v6e 32 GB | Gemma-4-E2B | Single decode | vLLM 235 · Pie 231 |
| TPU v6e 32 GB | Gemma-4-E2B | Single prefill | Pie 49,315 · vLLM 36,201 |
| TPU v6e 32 GB | Gemma-4-E2B | Batched decode | vLLM 15,113 · Pie 13,966 |
| TPU v6e 32 GB | Gemma-4-E2B | Batched prefill | vLLM 115,393 · Pie 99,137 |
| TPU v6e 32 GB | Gemma-4-E4B | Single decode | vLLM 117 · Pie 116 |
| TPU v6e 32 GB | Gemma-4-E4B | Single prefill | Pie 34,947 · vLLM 23,540 |
| TPU v6e 32 GB | Gemma-4-E4B | Batched decode | vLLM 8,605 · Pie 8,359 |
| TPU v6e 32 GB | Gemma-4-E4B | Batched prefill | vLLM 65,764 · Pie 61,438 |
| TPU v6e 32 GB | Gemma-4-12B | Single decode | vLLM 45.4 · Pie 45.3 |
| TPU v6e 32 GB | Gemma-4-12B | Single prefill | Pie 13,725 · vLLM 11,182 |
| TPU v6e 32 GB | Gemma-4-12B | Batched decode | vLLM 1,195 · Pie 1,129 |
| TPU v6e 32 GB | Gemma-4-12B | Batched prefill | vLLM 23,773 · Pie 21,796 |
| TPU v6e 32 GB | GPT-OSS-20B | Single decode | Pie 216 · vLLM 110 |
| TPU v6e 32 GB | GPT-OSS-20B | Single prefill | Pie 20,938 · vLLM 15,629 |
| TPU v6e 32 GB | GPT-OSS-20B | Batched decode | Pie 6,710 · vLLM 5,960 |
| TPU v6e 32 GB | GPT-OSS-20B | Batched prefill | Pie 36,384 · vLLM 33,880 |
| TPU v5e 16 GB | Qwen3.5-0.8B | Single decode | Pie 467 · vLLM 271 · SGLang 175 |
| TPU v5e 16 GB | Qwen3.5-0.8B | Single prefill | Pie 43,296 · vLLM 32,315 · SGLang 6,798 |
| TPU v5e 16 GB | Qwen3.5-0.8B | Batched decode | vLLM 8,759 · Pie 8,298 · SGLang 3,001 |
| TPU v5e 16 GB | Qwen3.5-0.8B | Batched prefill | vLLM 73,279 · SGLang 60,116 · Pie 54,344 |
| TPU v5e 16 GB | Qwen3-0.6B | Single decode | Pie 555 · vLLM 326 · SGLang 264 |
| TPU v5e 16 GB | Qwen3-0.6B | Single prefill | Pie 61,458 · vLLM 38,439 · SGLang 26,055 |
| TPU v5e 16 GB | Qwen3-0.6B | Batched decode | vLLM 9,028 · Pie 8,124 · SGLang 5,240 |
| TPU v5e 16 GB | Qwen3-0.6B | Batched prefill | vLLM 109,487 · Pie 106,037 · SGLang 73,324 |
| TPU v5e 16 GB | Gemma-4-E2B | Single decode | Pie 152 · vLLM 148 |
| TPU v5e 16 GB | Gemma-4-E2B | Single prefill | Pie 26,042 · vLLM 17,095 |
| TPU v5e 16 GB | Gemma-4-E2B | Batched decode | vLLM 8,811 · Pie 6,891 |
| TPU v5e 16 GB | Gemma-4-E2B | Batched prefill | vLLM 39,846 · Pie 29,543 |
| TPU v5e 16 GB | Gemma-4-E4B | Single decode | vLLM 74.1 · Pie 68.2 |
| TPU v5e 16 GB | Gemma-4-E4B | Single prefill | Pie 14,526 · vLLM 9,345 |
| TPU v5e 16 GB | Gemma-4-E4B | Batched decode | vLLM 1,370 · Pie 1,309 |
| TPU v5e 16 GB | Gemma-4-E4B | Batched prefill | vLLM 20,053 · Pie 15,908 |

### Browser

| Machine | Model | Setup | Engines (tok/s) |
|---|---|---|---|
| RTX 5090 Chrome 32 GB | Gemma-4-E2B | Single decode | Pie 84.8 · WebLLM 39 · llama.cpp WebGPU 21.5 |
| RTX 5090 Chrome 32 GB | Gemma-4-E2B | Single prefill | Pie 3,055 · WebLLM 495 · llama.cpp WebGPU 62.3 |
| RTX 5090 Chrome 32 GB | Gemma-4-E2B | Batched decode | Pie 486 · llama.cpp WebGPU 23.9 |
| RTX 5090 Chrome 32 GB | Gemma-4-E2B | Batched prefill | Pie 2,523 · llama.cpp WebGPU 62.4 |
| RTX 5090 Chrome 32 GB | Gemma-4-E4B | Single decode | Pie 67.9 · llama.cpp WebGPU 15.2 |
| RTX 5090 Chrome 32 GB | Gemma-4-E4B | Single prefill | Pie 2,098 · llama.cpp WebGPU 32.6 |
| RTX 5090 Chrome 32 GB | Gemma-4-E4B | Batched decode | Pie 351 · llama.cpp WebGPU 12.9 |
| RTX 5090 Chrome 32 GB | Gemma-4-E4B | Batched prefill | Pie 1,677 · llama.cpp WebGPU 32.5 |
| RTX 5090 Chrome 32 GB | Gemma-4-12B | Single decode | Pie 37.8 · llama.cpp WebGPU 33.8 |
| RTX 5090 Chrome 32 GB | Gemma-4-12B | Single prefill | Pie 1,107 · llama.cpp WebGPU 414 |
| RTX 5090 Chrome 32 GB | Gemma-4-12B | Batched decode | Pie 209 · llama.cpp WebGPU 74.7 |
| RTX 5090 Chrome 32 GB | Gemma-4-12B | Batched prefill | Pie 1,213 · llama.cpp WebGPU 445 |
| RTX 5090 Chrome 32 GB | Qwen3.5-4B | Single decode | Pie 97.3 · WebLLM 67 · llama.cpp WebGPU 42 |
| RTX 5090 Chrome 32 GB | Qwen3.5-4B | Single prefill | Pie 1,990 · WebLLM 1,314 · llama.cpp WebGPU 858 |
| RTX 5090 Chrome 32 GB | Qwen3.5-4B | Batched decode | Pie 409 · llama.cpp WebGPU 117 |
| RTX 5090 Chrome 32 GB | Qwen3.5-4B | Batched prefill | Pie 2,092 · llama.cpp WebGPU 1,098 |
| RTX 5090 Chrome 32 GB | Qwen3.5-9B | Single decode | Pie 87.6 · WebLLM 60.2 · llama.cpp WebGPU 40.3 |
| RTX 5090 Chrome 32 GB | Qwen3.5-9B | Single prefill | Pie 1,327 · WebLLM 863 · llama.cpp WebGPU 553 |
| RTX 5090 Chrome 32 GB | Qwen3.5-9B | Batched decode | Pie 292 · llama.cpp WebGPU 91.5 |
| RTX 5090 Chrome 32 GB | Qwen3.5-9B | Batched prefill | Pie 1,326 · llama.cpp WebGPU 662 |
| RTX 4090 Chrome 24 GB | Gemma-4-E2B | Single decode | Pie 92.8 · WebLLM 39 · llama.cpp WebGPU 25.4 |
| RTX 4090 Chrome 24 GB | Gemma-4-E2B | Single prefill | Pie 3,122 · WebLLM 455 · llama.cpp WebGPU 71.9 |
| RTX 4090 Chrome 24 GB | Gemma-4-E2B | Batched decode | Pie 459 · llama.cpp WebGPU 26.6 |
| RTX 4090 Chrome 24 GB | Gemma-4-E2B | Batched prefill | Pie 2,947 · llama.cpp WebGPU 70.3 |
| RTX 4090 Chrome 24 GB | Gemma-4-E4B | Single decode | Pie 72.4 · llama.cpp WebGPU 14.6 |
| RTX 4090 Chrome 24 GB | Gemma-4-E4B | Single prefill | Pie 1,938 · llama.cpp WebGPU 34.2 |
| RTX 4090 Chrome 24 GB | Gemma-4-E4B | Batched decode | Pie 319 · llama.cpp WebGPU 13.8 |
| RTX 4090 Chrome 24 GB | Gemma-4-E4B | Batched prefill | Pie 1,761 · llama.cpp WebGPU 35.6 |
| RTX 4090 Chrome 24 GB | Gemma-4-12B | Single decode | llama.cpp WebGPU 42.7 · Pie 33 |
| RTX 4090 Chrome 24 GB | Gemma-4-12B | Single prefill | Pie 874 · llama.cpp WebGPU 240 |
| RTX 4090 Chrome 24 GB | Gemma-4-12B | Batched decode | Pie 189 · llama.cpp WebGPU 56.4 |
| RTX 4090 Chrome 24 GB | Gemma-4-12B | Batched prefill | Pie 982 · llama.cpp WebGPU 253 |
| RTX 4090 Chrome 24 GB | Qwen3.5-4B | Single decode | Pie 107 · llama.cpp WebGPU 46.3 · WebLLM 33.3 |
| RTX 4090 Chrome 24 GB | Qwen3.5-4B | Single prefill | Pie 1,871 · WebLLM 915 · llama.cpp WebGPU 667 |
| RTX 4090 Chrome 24 GB | Qwen3.5-4B | Batched decode | Pie 331 · llama.cpp WebGPU 99.9 |
| RTX 4090 Chrome 24 GB | Qwen3.5-4B | Batched prefill | Pie 1,985 · llama.cpp WebGPU 683 |
| RTX 4090 Chrome 24 GB | Qwen3.5-9B | Single decode | Pie 84.9 · llama.cpp WebGPU 46 · WebLLM 35.1 |
| RTX 4090 Chrome 24 GB | Qwen3.5-9B | Single prefill | Pie 1,180 · WebLLM 658 · llama.cpp WebGPU 382 |
| RTX 4090 Chrome 24 GB | Qwen3.5-9B | Batched decode | Pie 251 · llama.cpp WebGPU 73 |
| RTX 4090 Chrome 24 GB | Qwen3.5-9B | Batched prefill | Pie 1,220 · llama.cpp WebGPU 391 |

## Correctness

Whether Pie’s output matches a reference engine’s on the same weights.

| Platform | Machine | Model | Result | Check |
|---|---|---|---|---|
| Apple Silicon | M5 Max 128 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M5 Max 128 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M5 Max 64 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M5 Max 64 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M5 Max 64 GB | GPT-OSS-120B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Apple Silicon | M5 Max 64 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M4 Pro 48 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M4 Pro 48 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Apple Silicon | M4 Pro 48 GB | GPT-OSS-120B | pass | Next-token KL divergence vs MLX: 0.0005 nats |
| Apple Silicon | M4 Pro 48 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Apple Silicon | M3 16 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M3 16 GB | Qwen3.8-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M3 16 GB | Qwen3.6-35B | pass | Next-token KL divergence vs MLX: 0.0006 nats |
| Apple Silicon | M3 16 GB | Qwen3.6-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M3 16 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M3 16 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Apple Silicon | M3 16 GB | Gemma-4-31B | pass | Next-token KL divergence vs MLX: 0.0175 nats |
| Apple Silicon | M3 16 GB | Gemma-4-26B | pass | Next-token KL divergence vs MLX: 0.1318 nats |
| Apple Silicon | M3 16 GB | GPT-OSS-120B | pass | Next-token KL divergence vs MLX: 0.0015 nats |
| Apple Silicon | M3 16 GB | GPT-OSS-20B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Apple Silicon | M3 16 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M2 16 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M2 16 GB | Qwen3.8-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M2 16 GB | Qwen3.6-35B | pass | Next-token KL divergence vs MLX: 0.0006 nats |
| Apple Silicon | M2 16 GB | Qwen3.6-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M2 16 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Apple Silicon | M2 16 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Apple Silicon | M2 16 GB | Gemma-4-31B | pass | Next-token KL divergence vs MLX: 0.0062 nats |
| Apple Silicon | M2 16 GB | Gemma-4-26B | pass | Next-token KL divergence vs MLX: 0.0390 nats |
| Apple Silicon | M2 16 GB | GPT-OSS-120B | pass | Next-token KL divergence vs MLX: 0.0019 nats |
| Apple Silicon | M2 16 GB | GPT-OSS-20B | pass | Next-token KL divergence vs MLX: 0.0002 nats |
| Apple Silicon | M2 16 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| NVIDIA GPU | B300 288 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | B300 288 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0005 nats |
| NVIDIA GPU | B300 288 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0007 nats |
| NVIDIA GPU | B300 288 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0009 nats |
| NVIDIA GPU | B300 288 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | B200 180 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | B200 180 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0004 nats |
| NVIDIA GPU | B200 180 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0497 nats |
| NVIDIA GPU | B200 180 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0355 nats |
| NVIDIA GPU | B200 180 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | B200 180 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0211 nats |
| NVIDIA GPU | B200 180 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0005 nats |
| NVIDIA GPU | B200 180 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0168 nats |
| NVIDIA GPU | B200 180 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0004 nats |
| NVIDIA GPU | B200 180 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0012 nats |
| NVIDIA GPU | B200 180 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | B200 180 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0044 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0339 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0122 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0223 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 6000 96 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | RTX 5090 32 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 5090 32 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 5090 32 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0007 nats |
| NVIDIA GPU | RTX 5090 32 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 5090 32 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX 5090 32 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0362 nats |
| NVIDIA GPU | RTX 5090 32 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0090 nats |
| NVIDIA GPU | RTX 5090 32 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 5090 32 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 5090 32 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.1493 nats |
| NVIDIA GPU | RTX 5090 32 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0290 nats |
| NVIDIA GPU | RTX 5090 32 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 5090 32 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0004 nats |
| NVIDIA GPU | RTX 5090 32 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 5090 32 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0004 nats |
| NVIDIA GPU | RTX 5090 32 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0239 nats |
| NVIDIA GPU | RTX 5090 32 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 5090 32 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0005 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0310 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0076 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.0301 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0262 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0228 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX PRO 4500 32 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | H200 SXM 141 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | H200 SXM 141 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | H200 SXM 141 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0101 nats |
| NVIDIA GPU | H200 SXM 141 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0136 nats |
| NVIDIA GPU | H200 SXM 141 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | H200 SXM 141 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0210 nats |
| NVIDIA GPU | H200 SXM 141 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0004 nats |
| NVIDIA GPU | H200 SXM 141 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0452 nats |
| NVIDIA GPU | H200 SXM 141 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0147 nats |
| NVIDIA GPU | H200 SXM 141 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0011 nats |
| NVIDIA GPU | H200 SXM 141 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | H200 SXM 141 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0049 nats |
| NVIDIA GPU | H100 SXM 80 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | H100 SXM 80 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0005 nats |
| NVIDIA GPU | H100 SXM 80 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | H100 SXM 80 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0170 nats |
| NVIDIA GPU | H100 SXM 80 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0078 nats |
| NVIDIA GPU | H100 SXM 80 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | H100 SXM 80 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0165 nats |
| NVIDIA GPU | H100 SXM 80 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | H100 SXM 80 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0059 nats |
| NVIDIA GPU | H100 SXM 80 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | H100 SXM 80 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0010 nats |
| NVIDIA GPU | H100 SXM 80 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | H100 SXM 80 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0010 nats |
| NVIDIA GPU | L40S 46 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0007 nats |
| NVIDIA GPU | L40S 46 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | L40S 46 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0163 nats |
| NVIDIA GPU | L40S 46 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0069 nats |
| NVIDIA GPU | L40S 46 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | L40S 46 GB | Gemma-4-31B | pass | Next-token KL divergence vs vLLM: 0.0017 nats |
| NVIDIA GPU | L40S 46 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.0732 nats |
| NVIDIA GPU | L40S 46 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0251 nats |
| NVIDIA GPU | L40S 46 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | L40S 46 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | L40S 46 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0221 nats |
| NVIDIA GPU | L40S 46 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | L40S 46 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | RTX 4090 24 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 4090 24 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 4090 24 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0004 nats |
| NVIDIA GPU | RTX 4090 24 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 4090 24 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX 4090 24 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0092 nats |
| NVIDIA GPU | RTX 4090 24 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0068 nats |
| NVIDIA GPU | RTX 4090 24 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 4090 24 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 4090 24 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.0620 nats |
| NVIDIA GPU | RTX 4090 24 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0210 nats |
| NVIDIA GPU | RTX 4090 24 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | RTX 4090 24 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX 4090 24 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 4090 24 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | RTX 4090 24 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0231 nats |
| NVIDIA GPU | RTX 4090 24 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | RTX 4090 24 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | A100 SXM 80 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A100 SXM 80 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0006 nats |
| NVIDIA GPU | A100 SXM 80 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | A100 SXM 80 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0111 nats |
| NVIDIA GPU | A100 SXM 80 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0062 nats |
| NVIDIA GPU | A100 SXM 80 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | A100 SXM 80 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0091 nats |
| NVIDIA GPU | A100 SXM 80 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | A100 SXM 80 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0057 nats |
| NVIDIA GPU | A100 SXM 80 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | A100 SXM 80 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0238 nats |
| NVIDIA GPU | A100 SXM 80 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A100 SXM 80 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0012 nats |
| NVIDIA GPU | A40 48 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A40 48 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A40 48 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0007 nats |
| NVIDIA GPU | A40 48 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A40 48 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | A40 48 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0098 nats |
| NVIDIA GPU | A40 48 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0061 nats |
| NVIDIA GPU | A40 48 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | A40 48 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| NVIDIA GPU | A40 48 GB | Gemma-4-31B | pass | Next-token KL divergence vs vLLM: 0.0017 nats |
| NVIDIA GPU | A40 48 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.0896 nats |
| NVIDIA GPU | A40 48 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0243 nats |
| NVIDIA GPU | A40 48 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A40 48 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| NVIDIA GPU | A40 48 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A40 48 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| NVIDIA GPU | A40 48 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0219 nats |
| NVIDIA GPU | A40 48 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| NVIDIA GPU | A40 48 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| AMD GPU | MI355X 288 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| AMD GPU | MI355X 288 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| AMD GPU | MI355X 288 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0425 nats |
| AMD GPU | MI355X 288 GB | DeepSeek-V4-Flash-284B | fail | Next-token KL divergence vs vLLM: 0.7213 nats |
| AMD GPU | MI355X 288 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0123 nats |
| AMD GPU | MI355X 288 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0020 nats |
| AMD GPU | MI355X 288 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0075 nats |
| AMD GPU | MI355X 288 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0116 nats |
| AMD GPU | MI355X 288 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0031 nats |
| AMD GPU | MI355X 288 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| AMD GPU | MI355X 288 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0013 nats |
| AMD GPU | MI350X 288 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| AMD GPU | MI350X 288 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| AMD GPU | MI350X 288 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0327 nats |
| AMD GPU | MI350X 288 GB | DeepSeek-V4-Flash-284B | fail | Next-token KL divergence vs vLLM: 0.4064 nats |
| AMD GPU | MI350X 288 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0131 nats |
| AMD GPU | MI350X 288 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0019 nats |
| AMD GPU | MI350X 288 GB | MiMo-V2.6-Flash-RL-309B | pass | Next-token KL divergence vs vLLM: 0.0120 nats |
| AMD GPU | MI350X 288 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0011 nats |
| AMD GPU | MI350X 288 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| AMD GPU | MI350X 288 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0015 nats |
| AMD GPU | MI300X 192 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0003 nats |
| AMD GPU | MI300X 192 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0180 nats |
| AMD GPU | MI300X 192 GB | DeepSeek-V4-Flash-284B | fail | Next-token KL divergence vs vLLM: 0.3988 nats |
| AMD GPU | MI300X 192 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0021 nats |
| AMD GPU | MI300X 192 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0011 nats |
| AMD GPU | MI300X 192 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| AMD GPU | MI300X 192 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0044 nats |
| Google TPU | TPU v6e 32 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0007 nats |
| Google TPU | TPU v6e 32 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| Google TPU | TPU v6e 32 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0105 nats |
| Google TPU | TPU v6e 32 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0056 nats |
| Google TPU | TPU v6e 32 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0002 nats |
| Google TPU | TPU v6e 32 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | Gemma-4-31B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.0031 nats |
| Google TPU | TPU v6e 32 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0386 nats |
| Google TPU | TPU v6e 32 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0060 nats |
| Google TPU | TPU v6e 32 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| Google TPU | TPU v6e 32 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0009 nats |
| Google TPU | TPU v6e 32 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v6e 32 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| Google TPU | TPU v5e 16 GB | Qwen3.8-Flash-Next-125B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Qwen3.8-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Qwen3.6-35B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| Google TPU | TPU v5e 16 GB | Qwen3.6-27B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Kimi-K3-2.8T | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | DeepSeek-V4.1-Flash-552B | pass | Next-token KL divergence vs vLLM: 0.0142 nats |
| Google TPU | TPU v5e 16 GB | DeepSeek-V4-Flash-284B | pass | Next-token KL divergence vs vLLM: 0.0055 nats |
| Google TPU | TPU v5e 16 GB | Muse-Spark-Open-816B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs vLLM: 0.0001 nats |
| Google TPU | TPU v5e 16 GB | Gemma-4-31B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Gemma-4-26B | pass | Next-token KL divergence vs vLLM: 0.0030 nats |
| Google TPU | TPU v5e 16 GB | Gemma-4-12B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | GPT-OSS-120B | pass | Next-token KL divergence vs vLLM: 0.0139 nats |
| Google TPU | TPU v5e 16 GB | GPT-OSS-20B | pass | Next-token KL divergence vs vLLM: 0.0020 nats |
| Google TPU | TPU v5e 16 GB | GLM-5.3-744B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | GLM-5.3-Flash-320B | pass | Next-token KL divergence vs vLLM: 0.0059 nats |
| Google TPU | TPU v5e 16 GB | MiniMax-M3.1-Flash-428B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Hy4-770B | pass | Next-token KL divergence vs vLLM: 0.0009 nats |
| Google TPU | TPU v5e 16 GB | Nemotron-3-Ultra-550B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Google TPU | TPU v5e 16 GB | Nemotron-3-Super-120B | pass | Next-token KL divergence vs vLLM: 0.0000 nats |
| Browser | RTX 5090 Chrome 32 GB | Qwen3.8-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Browser | RTX 5090 Chrome 32 GB | Qwen3.6-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Browser | RTX 5090 Chrome 32 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Browser | RTX 5090 Chrome 32 GB | Gemma-4-31B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Browser | RTX 5090 Chrome 32 GB | Gemma-4-26B | pass | Next-token KL divergence vs MLX: 0.1224 nats |
| Browser | RTX 5090 Chrome 32 GB | GPT-OSS-120B | pass | Next-token KL divergence vs MLX: 0.0016 nats |
| Browser | RTX 5090 Chrome 32 GB | GPT-OSS-20B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Browser | RTX 4090 Chrome 24 GB | Qwen3.8-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Browser | RTX 4090 Chrome 24 GB | Qwen3.6-27B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Browser | RTX 4090 Chrome 24 GB | Muse-Glimmer-30B | pass | Next-token KL divergence vs MLX: 0.0001 nats |
| Browser | RTX 4090 Chrome 24 GB | Gemma-4-31B | pass | Next-token KL divergence vs MLX: 0.0000 nats |
| Browser | RTX 4090 Chrome 24 GB | Gemma-4-26B | pass | Next-token KL divergence vs MLX: 0.1371 nats |
| Browser | RTX 4090 Chrome 24 GB | GPT-OSS-120B | pass | Next-token KL divergence vs MLX: 0.0014 nats |
| Browser | RTX 4090 Chrome 24 GB | GPT-OSS-20B | pass | Next-token KL divergence vs MLX: 0.0002 nats |