Qwen benchmarks

Results as tablesRelease main-f03640150, as of 2026-11-13

A snapshot taken when this site was last built. The charts above read live data and may be newer. The same tables as markdown.

Throughput on Apple Silicon, tokens per second: every engine on the same model and machine, fastest first
Machine	Model	Setup	Engines (tok/s)
M5 Max 128 GB	Qwen3.6-27B	Single decode	Pie 33.4 · MLX 31.8 · llama.cpp 27
M5 Max 128 GB	Qwen3.6-27B	Single prefill	Pie 900 · MLX 857 · llama.cpp 529
M5 Max 128 GB	Qwen3.6-27B	Batched decode	Pie 234 · llama.cpp 97.1 · MLX 29
M5 Max 128 GB	Qwen3.6-27B	Batched prefill	MLX 772 · Pie 761 · llama.cpp 575
M5 Max 128 GB	Qwen3.6-35B	Single decode	Pie 166 · MLX 130 · llama.cpp 105
M5 Max 128 GB	Qwen3.6-35B	Single prefill	Pie 4,292 · MLX 4,056 · llama.cpp 2,422
M5 Max 128 GB	Qwen3.6-35B	Batched decode	Pie 735 · llama.cpp 367 · MLX 119
M5 Max 128 GB	Qwen3.6-35B	Batched prefill	Pie 6,587 · llama.cpp 2,800
M5 Max 128 GB	Qwen3.8-Flash-Next-125B	Single decode	Pie 75 · llama.cpp 63 · MLX 60.7
M5 Max 128 GB	Qwen3.8-Flash-Next-125B	Single prefill	MLX 1,221 · Pie 1,054 · llama.cpp 966
M5 Max 128 GB	Qwen3.8-Flash-Next-125B	Batched decode	Pie 269 · MLX 94 · llama.cpp 77.6
M5 Max 128 GB	Qwen3.8-Flash-Next-125B	Batched prefill	Pie 1,294 · MLX 1,101
M5 Max 128 GB	Qwen3.8-27B	Single decode	Pie 32.8 · MLX 31.9 · llama.cpp 27.2
M5 Max 128 GB	Qwen3.8-27B	Single prefill	Pie 862 · MLX 856 · llama.cpp 536
M5 Max 128 GB	Qwen3.8-27B	Batched decode	Pie 228 · llama.cpp 97.4 · MLX 28.7
M5 Max 128 GB	Qwen3.8-27B	Batched prefill	MLX 774 · Pie 727 · llama.cpp 572
M5 Max 128 GB	Qwen3.5-4B	Single decode	Pie 167 · MLX 152 · llama.cpp 128
M5 Max 128 GB	Qwen3.5-4B	Single prefill	Pie 4,932 · MLX 4,908 · llama.cpp 2,997
M5 Max 128 GB	Qwen3.5-4B	Batched decode	Pie 1,166 · llama.cpp 407 · MLX 142
M5 Max 128 GB	Qwen3.5-4B	Batched prefill	Pie 5,536 · MLX 4,993 · llama.cpp 3,165
M5 Max 128 GB	Qwen3.5-9B	Single decode	Pie 101 · MLX 96.4 · llama.cpp 82.8
M5 Max 128 GB	Qwen3.5-9B	Single prefill	Pie 2,968 · MLX 2,938 · llama.cpp 1,808
M5 Max 128 GB	Qwen3.5-9B	Batched decode	Pie 800 · llama.cpp 300 · MLX 88
M5 Max 128 GB	Qwen3.5-9B	Batched prefill	MLX 2,866 · Pie 2,852 · llama.cpp 1,984
M5 Max 64 GB	Qwen3.5-0.8B	Single decode	vLLM Metal 266
M5 Max 64 GB	Qwen3.5-0.8B	Single prefill	vLLM Metal 19,134
M5 Max 64 GB	Qwen3.5-0.8B	Batched decode	vLLM Metal 3,606
M5 Max 64 GB	Qwen3.5-0.8B	Batched prefill	vLLM Metal 29,554
M5 Max 64 GB	Qwen3.6-27B	Single decode	Pie 33.6 · vLLM Metal 33 · MLX 32 · llama.cpp 25.3
M5 Max 64 GB	Qwen3.6-27B	Single prefill	Pie 849 · vLLM Metal 823 · MLX 781 · llama.cpp 454
M5 Max 64 GB	Qwen3.6-27B	Batched decode	Pie 220 · vLLM Metal 169 · llama.cpp 92.4 · MLX 24.4
M5 Max 64 GB	Qwen3.6-27B	Batched prefill	vLLM Metal 807 · MLX 714 · Pie 689 · llama.cpp 538
M5 Max 64 GB	Qwen3.6-35B	Single decode	Pie 168 · MLX 130 · vLLM Metal 122 · llama.cpp 104
M5 Max 64 GB	Qwen3.6-35B	Single prefill	Pie 4,535 · MLX 4,072 · vLLM Metal 3,933 · llama.cpp 2,333
M5 Max 64 GB	Qwen3.6-35B	Batched decode	Pie 688 · vLLM Metal 592 · llama.cpp 340 · MLX 121
M5 Max 64 GB	Qwen3.6-35B	Batched prefill	Pie 6,189 · vLLM Metal 5,678 · MLX 3,880 · llama.cpp 2,591
M5 Max 64 GB	Qwen3.8-27B	Single decode	Pie 33.6 · vLLM Metal 33 · MLX 32 · llama.cpp 25.7
M5 Max 64 GB	Qwen3.8-27B	Single prefill	Pie 850 · vLLM Metal 779 · MLX 690 · llama.cpp 457
M5 Max 64 GB	Qwen3.8-27B	Batched decode	Pie 220 · vLLM Metal 169 · llama.cpp 92.4 · MLX 24.6
M5 Max 64 GB	Qwen3.8-27B	Batched prefill	vLLM Metal 807 · MLX 713 · Pie 687 · llama.cpp 538
M5 Max 64 GB	Qwen3.5-4B	Single decode	Pie 166 · vLLM Metal 160 · MLX 152 · llama.cpp 121
M5 Max 64 GB	Qwen3.5-4B	Single prefill	Pie 5,092 · MLX 4,920 · vLLM Metal 4,834 · llama.cpp 2,765
M5 Max 64 GB	Qwen3.5-4B	Batched decode	Pie 1,145 · vLLM Metal 923 · llama.cpp 394 · MLX 137
M5 Max 64 GB	Qwen3.5-4B	Batched prefill	vLLM Metal 5,583 · Pie 5,068 · MLX 4,704 · llama.cpp 2,982
M5 Max 64 GB	Qwen3.5-9B	Single decode	vLLM Metal 102 · Pie 102 · MLX 96.2 · llama.cpp 77.3
M5 Max 64 GB	Qwen3.5-9B	Single prefill	Pie 3,034 · MLX 2,933 · vLLM Metal 2,913 · llama.cpp 1,583
M5 Max 64 GB	Qwen3.5-9B	Batched decode	Pie 732 · vLLM Metal 568 · llama.cpp 282 · MLX 77.3
M5 Max 64 GB	Qwen3.5-9B	Batched prefill	vLLM Metal 2,994 · MLX 2,645 · Pie 2,541 · llama.cpp 1,821
M5 16 GB	Qwen3.5-4B	Batched decode	llama.cpp 70.6 · MLX 34.5
M5 16 GB	Qwen3.5-9B	Batched decode	llama.cpp 31.9
M4 Pro 48 GB	Qwen3.5-0.8B	Single decode	vLLM Metal 136
M4 Pro 48 GB	Qwen3.5-0.8B	Single prefill	vLLM Metal 4,718
M4 Pro 48 GB	Qwen3.5-0.8B	Batched decode	vLLM Metal 970
M4 Pro 48 GB	Qwen3.5-0.8B	Batched prefill	vLLM Metal 5,512
M4 Pro 48 GB	Qwen3.6-27B	Single decode	Pie 15.8 · vLLM Metal 15.6 · MLX 15.1 · llama.cpp 13.2
M4 Pro 48 GB	Qwen3.6-27B	Single prefill	Pie 214 · MLX 126 · vLLM Metal 125 · llama.cpp 118
M4 Pro 48 GB	Qwen3.6-27B	Batched decode	Pie 51.6 · llama.cpp 32.9
M4 Pro 48 GB	Qwen3.6-27B	Batched prefill	Pie 132 · MLX 126
M4 Pro 48 GB	Qwen3.6-35B	Single decode	Pie 98.5 · vLLM Metal 86 · MLX 79.8 · llama.cpp 54.4
M4 Pro 48 GB	Qwen3.6-35B	Single prefill	Pie 989 · MLX 946 · vLLM Metal 890 · llama.cpp 745
M4 Pro 48 GB	Qwen3.6-35B	Batched decode	Pie 245 · vLLM Metal 227 · llama.cpp 143 · MLX 69.4
M4 Pro 48 GB	Qwen3.6-35B	Batched prefill	Pie 1,171 · vLLM Metal 1,041 · MLX 948 · llama.cpp 820
M4 Pro 48 GB	Qwen3.8-27B	Single decode	Pie 15.8 · vLLM Metal 15.6 · MLX 15.1 · llama.cpp 12.8
M4 Pro 48 GB	Qwen3.8-27B	Single prefill	Pie 214 · MLX 127 · vLLM Metal 125 · llama.cpp 116
M4 Pro 48 GB	Qwen3.8-27B	Batched decode	Pie 51.6 · llama.cpp 33.8 · MLX 13
M4 Pro 48 GB	Qwen3.8-27B	Batched prefill	Pie 132 · MLX 126
M4 Pro 48 GB	Qwen3.5-4B	Single decode	Pie 87.3 · vLLM Metal 84.2 · MLX 78 · llama.cpp 61.7
M4 Pro 48 GB	Qwen3.5-4B	Single prefill	Pie 1,212 · MLX 825 · vLLM Metal 788 · llama.cpp 711
M4 Pro 48 GB	Qwen3.5-4B	Batched decode	Pie 288 · vLLM Metal 282 · llama.cpp 169 · MLX 68.3
M4 Pro 48 GB	Qwen3.5-4B	Batched prefill	Pie 849 · vLLM Metal 845 · MLX 825 · llama.cpp 726
M4 Pro 48 GB	Qwen3.5-9B	Single decode	Pie 51.7 · vLLM Metal 50.5 · MLX 47.9 · llama.cpp 38.9
M4 Pro 48 GB	Qwen3.5-9B	Single prefill	Pie 723 · MLX 449 · vLLM Metal 434 · llama.cpp 397
M4 Pro 48 GB	Qwen3.5-9B	Batched decode	Pie 173 · vLLM Metal 167 · llama.cpp 116 · MLX 41.3
M4 Pro 48 GB	Qwen3.5-9B	Batched prefill	Pie 472 · vLLM Metal 463 · MLX 448 · llama.cpp 411
M3 16 GB	Qwen3.5-0.8B	Single decode	vLLM Metal 55.7
M3 16 GB	Qwen3.5-0.8B	Single prefill	vLLM Metal 2,146
M3 16 GB	Qwen3.5-0.8B	Batched decode	vLLM Metal 441
M3 16 GB	Qwen3.5-0.8B	Batched prefill	vLLM Metal 2,319
M3 16 GB	Qwen3.5-4B	Single decode	Pie 35.1 · vLLM Metal 34.7 · MLX 33.6 · llama.cpp 26.2
M3 16 GB	Qwen3.5-4B	Single prefill	Pie 515 · MLX 346 · vLLM Metal 334 · llama.cpp 276
M3 16 GB	Qwen3.5-4B	Batched decode	Pie 98.5 · vLLM Metal 86.9 · llama.cpp 40.6 · MLX 24.9
M3 16 GB	Qwen3.5-4B	Batched prefill	Pie 538 · MLX 335
M3 16 GB	Qwen3.5-9B	Single decode	Pie 19.6 · vLLM Metal 19.2 · MLX 19 · llama.cpp 12.3
M3 16 GB	Qwen3.5-9B	Single prefill	Pie 283 · MLX 177 · vLLM Metal 171 · llama.cpp 140
M3 16 GB	Qwen3.5-9B	Batched decode	Pie 68.3 · llama.cpp 31.4 · MLX 11.7
M3 16 GB	Qwen3.5-9B	Batched prefill	Pie 306 · MLX 131
M2 16 GB	Qwen3.5-0.8B	Single decode	vLLM Metal 48.5
M2 16 GB	Qwen3.5-0.8B	Single prefill	vLLM Metal 1,529
M2 16 GB	Qwen3.5-0.8B	Batched decode	vLLM Metal 362
M2 16 GB	Qwen3.5-0.8B	Batched prefill	vLLM Metal 1,304
M2 16 GB	Qwen3.5-4B	Single decode	Pie 33.8 · MLX 27.6 · vLLM Metal 26.7 · llama.cpp 26
M2 16 GB	Qwen3.5-4B	Single prefill	Pie 331 · llama.cpp 277 · MLX 242 · vLLM Metal 139
M2 16 GB	Qwen3.5-4B	Batched decode	Pie 47.9 · llama.cpp 43.2 · MLX 23.2
M2 16 GB	Qwen3.5-4B	Batched prefill	Pie 372 · MLX 245
M2 16 GB	Qwen3.5-9B	Single decode	MLX 16.7 · Pie 16.2 · vLLM Metal 16 · llama.cpp 12.9
M2 16 GB	Qwen3.5-9B	Single prefill	Pie 136 · llama.cpp 135 · MLX 131 · vLLM Metal 82.1
M2 16 GB	Qwen3.5-9B	Batched decode	Pie 31 · llama.cpp 25.4 · MLX 12.8
M2 16 GB	Qwen3.5-9B	Batched prefill	Pie 212 · MLX 117

Throughput on NVIDIA GPU, tokens per second: every engine on the same model and machine, fastest first
Machine	Model	Setup	Engines (tok/s)
B200 180 GB	Qwen3.5-0.8B	Single decode	Pie 675 · SGLang 558 · vLLM 397 · TensorRT-LLM 275
B200 180 GB	Qwen3.5-0.8B	Single prefill	Pie 45,816 · SGLang 17,750 · vLLM 10,914 · TensorRT-LLM 4,764
B200 180 GB	Qwen3.5-0.8B	Batched decode	Pie 84,398 · SGLang 55,409 · vLLM 39,249 · TensorRT-LLM 6,454
B200 180 GB	Qwen3.5-0.8B	Batched prefill	Pie 470,170 · SGLang 439,719 · vLLM 254,530 · TensorRT-LLM 146,073
B200 180 GB	Qwen3-0.6B	Single decode	Pie 969 · SGLang 637 · vLLM 443 · TensorRT-LLM 297
B200 180 GB	Qwen3-0.6B	Single prefill	Pie 59,499 · SGLang 32,206 · vLLM 12,628 · TensorRT-LLM 7,172
B200 180 GB	Qwen3-0.6B	Batched decode	Pie 90,301 · SGLang 56,847 · vLLM 43,917 · TensorRT-LLM 9,524
B200 180 GB	Qwen3-0.6B	Batched prefill	Pie 662,963 · SGLang 377,453 · vLLM 211,925 · TensorRT-LLM 72,704
B200 180 GB	Qwen3.6-27B	Single decode	TensorRT-LLM 103 · Pie 102 · SGLang 102 · vLLM 95.2
B200 180 GB	Qwen3.6-27B	Single prefill	Pie 10,947 · SGLang 8,752 · vLLM 4,872 · TensorRT-LLM 3,601
B200 180 GB	Qwen3.6-27B	Batched decode	Pie 7,765 · vLLM 7,538 · SGLang 6,519 · TensorRT-LLM 6,171
B200 180 GB	Qwen3.6-27B	Batched prefill	vLLM 23,790 · TensorRT-LLM 23,567 · SGLang 21,905 · Pie 21,249
B200 180 GB	Qwen3.6-35B	Single decode	SGLang 396 · Pie 367 · vLLM 325 · TensorRT-LLM 279
B200 180 GB	Qwen3.6-35B	Single prefill	Pie 24,922 · SGLang 13,101 · vLLM 3,924 · TensorRT-LLM 3,257
B200 180 GB	Qwen3.6-35B	Batched decode	vLLM 22,721 · Pie 22,450 · SGLang 18,545 · TensorRT-LLM 10,779
B200 180 GB	Qwen3.6-35B	Batched prefill	vLLM 126,246 · SGLang 112,748 · Pie 107,876 · TensorRT-LLM 81,207
B200 180 GB	Qwen3.8-27B	Single decode	TensorRT-LLM 103 · SGLang 102 · Pie 102 · vLLM 94.5
B200 180 GB	Qwen3.8-27B	Single prefill	Pie 11,245 · SGLang 8,779 · vLLM 4,909 · TensorRT-LLM 3,839
B200 180 GB	Qwen3.8-27B	Batched decode	vLLM 7,539 · Pie 6,802 · SGLang 6,540 · TensorRT-LLM 5,549
B200 180 GB	Qwen3.8-27B	Batched prefill	TensorRT-LLM 22,873 · SGLang 22,350 · vLLM 21,806 · Pie 21,218
B200 180 GB	Qwen3.5-9B	Single decode	Pie 284 · SGLang 280 · TensorRT-LLM 275 · vLLM 248
B200 180 GB	Qwen3.5-9B	Single prefill	Pie 26,681 · SGLang 20,861 · TensorRT-LLM 11,563 · vLLM 6,042
B200 180 GB	Qwen3.5-9B	Batched decode	Pie 23,777 · vLLM 22,510 · SGLang 20,434 · TensorRT-LLM 10,027
B200 180 GB	Qwen3.5-9B	Batched prefill	SGLang 73,903 · vLLM 72,879 · Pie 71,320 · TensorRT-LLM 70,860
RTX PRO 6000 96 GB	Qwen3.5-0.8B	Single decode	TensorRT-LLM 573 · SGLang 567 · Pie 562 · vLLM 385
RTX PRO 6000 96 GB	Qwen3.5-0.8B	Single prefill	Pie 63,884 · SGLang 29,290 · vLLM 14,907 · TensorRT-LLM 11,588
RTX PRO 6000 96 GB	Qwen3.5-0.8B	Batched decode	TensorRT-LLM 21,984
RTX PRO 6000 96 GB	Qwen3.5-0.8B	Batched prefill	Pie 217,989 · vLLM 191,574 · SGLang 186,448 · TensorRT-LLM 175,675
RTX PRO 6000 96 GB	Qwen3-0.6B	Single decode	Pie 681 · SGLang 636 · vLLM 626 · TensorRT-LLM 580
RTX PRO 6000 96 GB	Qwen3-0.6B	Single prefill	Pie 85,375 · SGLang 73,009 · vLLM 68,556 · TensorRT-LLM 16,932
RTX PRO 6000 96 GB	Qwen3-0.6B	Batched decode	Pie 31,859 · vLLM 29,507 · SGLang 29,228 · TensorRT-LLM 23,224
RTX PRO 6000 96 GB	Qwen3-0.6B	Batched prefill	Pie 290,442 · vLLM 261,470 · SGLang 248,360 · TensorRT-LLM 198,607
RTX PRO 6000 96 GB	Qwen3.6-27B	Single decode	Pie 27.7 · TensorRT-LLM 27.2 · SGLang 27.1 · vLLM 26.7
RTX PRO 6000 96 GB	Qwen3.6-27B	Single prefill	Pie 5,341 · SGLang 5,210 · TensorRT-LLM 4,707 · vLLM 4,676
RTX PRO 6000 96 GB	Qwen3.6-27B	Batched decode	Pie 1,834 · TensorRT-LLM 1,814 · vLLM 1,560 · SGLang 1,260
RTX PRO 6000 96 GB	Qwen3.6-27B	Batched prefill	TensorRT-LLM 7,025 · vLLM 6,945 · Pie 6,865 · SGLang 6,731
RTX PRO 6000 96 GB	Qwen3.6-35B	Single decode	SGLang 192 · Pie 186 · TensorRT-LLM 179 · vLLM 167
RTX PRO 6000 96 GB	Qwen3.6-35B	Single prefill	Pie 12,824 · SGLang 12,686 · vLLM 11,143 · TensorRT-LLM 5,868
RTX PRO 6000 96 GB	Qwen3.6-35B	Batched decode	TensorRT-LLM 4,831 · Pie 4,747 · vLLM 3,522 · SGLang 2,701
RTX PRO 6000 96 GB	Qwen3.6-35B	Batched prefill	SGLang 41,034 · Pie 40,099 · TensorRT-LLM 39,497 · vLLM 38,140
RTX PRO 6000 96 GB	Qwen3.8-27B	Single decode	Pie 27.9 · SGLang 27.2 · TensorRT-LLM 27.2 · vLLM 26.6
RTX PRO 6000 96 GB	Qwen3.8-27B	Single prefill	Pie 6,061 · SGLang 5,553 · vLLM 5,073 · TensorRT-LLM 4,801
RTX PRO 6000 96 GB	Qwen3.8-27B	Batched decode	Pie 1,833 · TensorRT-LLM 1,811 · vLLM 1,561 · SGLang 1,264
RTX PRO 6000 96 GB	Qwen3.8-27B	Batched prefill	TensorRT-LLM 7,125 · vLLM 6,948 · Pie 6,869 · SGLang 6,805
RTX PRO 6000 96 GB	Qwen3.5-9B	Single decode	Pie 88.6 · TensorRT-LLM 86.3 · SGLang 86.2 · vLLM 83.8
RTX PRO 6000 96 GB	Qwen3.5-9B	Single prefill	Pie 18,324 · SGLang 14,774 · TensorRT-LLM 12,756 · vLLM 9,927
RTX PRO 6000 96 GB	Qwen3.5-9B	Batched decode	Pie 8,394 · TensorRT-LLM 7,854 · vLLM 6,422 · SGLang 6,072
RTX PRO 6000 96 GB	Qwen3.5-9B	Batched prefill	vLLM 24,245 · Pie 24,141 · TensorRT-LLM 23,713 · SGLang 22,828
RTX 5090 32 GB	Qwen3.5-0.8B	Single decode	TensorRT-LLM 730 · Pie 724 · SGLang 519 · vLLM 465
RTX 5090 32 GB	Qwen3.5-0.8B	Single prefill	Pie 65,629 · SGLang 19,275 · TensorRT-LLM 15,961 · vLLM 15,449
RTX 5090 32 GB	Qwen3.5-0.8B	Batched decode	Pie 30,630 · TensorRT-LLM 25,679 · SGLang 21,929 · vLLM 21,112
RTX 5090 32 GB	Qwen3.5-0.8B	Batched prefill	Pie 152,028 · TensorRT-LLM 142,701 · SGLang 127,016 · vLLM 98,301
RTX 5090 32 GB	Qwen3-0.6B	Single decode	Pie 769 · TensorRT-LLM 710 · SGLang 708 · vLLM 692
RTX 5090 32 GB	Qwen3-0.6B	Single prefill	Pie 71,535 · SGLang 53,548 · vLLM 29,465 · TensorRT-LLM 21,992
RTX 5090 32 GB	Qwen3-0.6B	Batched decode	Pie 29,583 · SGLang 26,623 · TensorRT-LLM 25,638 · vLLM 25,440
RTX 5090 32 GB	Qwen3-0.6B	Batched prefill	TensorRT-LLM 181,100 · Pie 179,831 · vLLM 140,466 · SGLang 135,364
RTX 5090 32 GB	Qwen3.5-9B	Single decode	TensorRT-LLM 98.6 · Pie 98 · SGLang 98 · vLLM 95.8
RTX 5090 32 GB	Qwen3.5-9B	Single prefill	Pie 12,248 · SGLang 11,026 · TensorRT-LLM 10,457 · vLLM 9,412
RTX 5090 32 GB	Qwen3.5-9B	Batched decode	Pie 5,597 · TensorRT-LLM 5,448 · vLLM 4,249
RTX 5090 32 GB	Qwen3.5-9B	Batched prefill	Pie 15,093 · vLLM 14,320 · TensorRT-LLM 14,224
RTX PRO 4500 32 GB	Qwen3.5-0.8B	Single decode	SGLang 444 · TensorRT-LLM 437 · Pie 430 · vLLM 418
RTX PRO 4500 32 GB	Qwen3.5-0.8B	Single prefill	Pie 52,836 · SGLang 22,415 · vLLM 19,021 · TensorRT-LLM 16,945
RTX PRO 4500 32 GB	Qwen3.5-0.8B	Batched decode	Pie 17,703 · TensorRT-LLM 17,227 · vLLM 12,462 · SGLang 11,719
RTX PRO 4500 32 GB	Qwen3.5-0.8B	Batched prefill	Pie 93,699 · TensorRT-LLM 89,324 · SGLang 73,789 · vLLM 60,105
RTX PRO 4500 32 GB	Qwen3-0.6B	Single decode	SGLang 482 · Pie 482 · vLLM 473 · TensorRT-LLM 470
RTX PRO 4500 32 GB	Qwen3-0.6B	Single prefill	Pie 73,254 · SGLang 42,978 · vLLM 27,066 · TensorRT-LLM 22,853
RTX PRO 4500 32 GB	Qwen3-0.6B	Batched decode	Pie 16,795 · TensorRT-LLM 15,826 · vLLM 15,682 · SGLang 15,279
RTX PRO 4500 32 GB	Qwen3-0.6B	Batched prefill	Pie 119,505 · TensorRT-LLM 109,821 · SGLang 105,519 · vLLM 104,344
RTX PRO 4500 32 GB	Qwen3.5-9B	Single decode	TensorRT-LLM 50 · Pie 49.8 · SGLang 49.7 · vLLM 49.4
RTX PRO 4500 32 GB	Qwen3.5-9B	Single prefill	Pie 9,139 · SGLang 7,069 · TensorRT-LLM 6,983 · vLLM 6,660
RTX PRO 4500 32 GB	Qwen3.5-9B	Batched decode	Pie 3,018 · TensorRT-LLM 2,889 · vLLM 2,267
RTX PRO 4500 32 GB	Qwen3.5-9B	Batched prefill	Pie 10,052 · TensorRT-LLM 8,398 · vLLM 8,365
H200 SXM 141 GB	Qwen3.5-0.8B	Single decode	Pie 929 · TensorRT-LLM 713 · SGLang 643 · vLLM 553
H200 SXM 141 GB	Qwen3.5-0.8B	Single prefill	Pie 69,360 · SGLang 25,328 · vLLM 20,707 · TensorRT-LLM 15,841
H200 SXM 141 GB	Qwen3.5-0.8B	Batched decode	Pie 60,127 · SGLang 41,846 · vLLM 32,182 · TensorRT-LLM 27,648
H200 SXM 141 GB	Qwen3.5-0.8B	Batched prefill	vLLM 340,786 · Pie 305,796 · SGLang 210,282 · TensorRT-LLM 185,697
H200 SXM 141 GB	Qwen3-0.6B	Single decode	Pie 888 · SGLang 664 · vLLM 611 · TensorRT-LLM 582
H200 SXM 141 GB	Qwen3-0.6B	Single prefill	Pie 104,679 · SGLang 56,488 · vLLM 40,553 · TensorRT-LLM 21,679
H200 SXM 141 GB	Qwen3-0.6B	Batched decode	Pie 65,059 · SGLang 54,795 · vLLM 53,991 · TensorRT-LLM 31,471
H200 SXM 141 GB	Qwen3-0.6B	Batched prefill	SGLang 434,754 · vLLM 412,407 · Pie 400,528 · TensorRT-LLM 252,916
H200 SXM 141 GB	Qwen3.6-27B	Single decode	Pie 73 · SGLang 68.5 · TensorRT-LLM 67.9 · vLLM 65.6
H200 SXM 141 GB	Qwen3.6-27B	Single prefill	SGLang 9,642 · Pie 9,342 · vLLM 8,851 · TensorRT-LLM 6,764
H200 SXM 141 GB	Qwen3.6-27B	Batched decode	vLLM 4,369 · Pie 4,368 · TensorRT-LLM 4,104 · SGLang 3,906
H200 SXM 141 GB	Qwen3.6-27B	Batched prefill	vLLM 12,330 · SGLang 12,215 · TensorRT-LLM 11,450 · Pie 11,441
H200 SXM 141 GB	Qwen3.6-35B	Single decode	Pie 302 · SGLang 274 · TensorRT-LLM 248 · vLLM 233
H200 SXM 141 GB	Qwen3.6-35B	Single prefill	Pie 21,639 · SGLang 19,159 · vLLM 8,030 · TensorRT-LLM 7,553
H200 SXM 141 GB	Qwen3.6-35B	Batched decode	TensorRT-LLM 13,645 · Pie 12,512 · vLLM 12,114 · SGLang 9,941
H200 SXM 141 GB	Qwen3.6-35B	Batched prefill	SGLang 63,236 · TensorRT-LLM 62,843 · Pie 55,118 · vLLM 55,085
H200 SXM 141 GB	Qwen3.8-27B	Single decode	Pie 72.9 · SGLang 70.6 · TensorRT-LLM 67.6 · vLLM 67.3
H200 SXM 141 GB	Qwen3.8-27B	Single prefill	SGLang 10,072 · vLLM 9,919 · Pie 9,222 · TensorRT-LLM 6,502
H200 SXM 141 GB	Qwen3.8-27B	Batched decode	Pie 4,366 · vLLM 4,363 · TensorRT-LLM 4,102 · SGLang 3,905
H200 SXM 141 GB	Qwen3.8-27B	Batched prefill	SGLang 12,287 · vLLM 12,277 · TensorRT-LLM 11,585 · Pie 11,406
H200 SXM 141 GB	Qwen3.5-9B	Single decode	Pie 211 · SGLang 201 · TensorRT-LLM 196 · vLLM 190
H200 SXM 141 GB	Qwen3.5-9B	Single prefill	Pie 26,944 · SGLang 23,042 · vLLM 16,979 · TensorRT-LLM 9,612
H200 SXM 141 GB	Qwen3.5-9B	Batched decode	TensorRT-LLM 15,246 · Pie 14,401 · vLLM 13,785 · SGLang 12,969
H200 SXM 141 GB	Qwen3.5-9B	Batched prefill	vLLM 40,827 · SGLang 40,086 · Pie 38,957 · TensorRT-LLM 37,158
H100 SXM 80 GB	Qwen3.5-0.8B	Single decode	Pie 841 · SGLang 742 · vLLM 655 · TensorRT-LLM 622
H100 SXM 80 GB	Qwen3.5-0.8B	Single prefill	Pie 75,991 · SGLang 28,921 · vLLM 22,931 · TensorRT-LLM 12,971
H100 SXM 80 GB	Qwen3.5-0.8B	Batched decode	Pie 51,807 · SGLang 37,149 · vLLM 34,065 · TensorRT-LLM 11,627
H100 SXM 80 GB	Qwen3.5-0.8B	Batched prefill	Pie 269,828 · vLLM 252,214 · SGLang 219,193 · TensorRT-LLM 143,986
H100 SXM 80 GB	Qwen3-0.6B	Single decode	Pie 802 · SGLang 695 · vLLM 645 · TensorRT-LLM 613
H100 SXM 80 GB	Qwen3-0.6B	Single prefill	Pie 70,306 · SGLang 34,297 · vLLM 27,078 · TensorRT-LLM 18,154
H100 SXM 80 GB	Qwen3-0.6B	Batched decode	SGLang 40,040 · vLLM 37,141 · Pie 34,322 · TensorRT-LLM 11,661
H100 SXM 80 GB	Qwen3-0.6B	Batched prefill	Pie 390,411 · SGLang 375,806 · vLLM 178,333 · TensorRT-LLM 173,238
H100 SXM 80 GB	Qwen3.6-27B	Single decode	Pie 53.8 · SGLang 52.2 · TensorRT-LLM 51.4 · vLLM 50.6
H100 SXM 80 GB	Qwen3.6-27B	Single prefill	Pie 9,945 · SGLang 9,423 · vLLM 6,686 · TensorRT-LLM 6,486
H100 SXM 80 GB	Qwen3.6-27B	Batched decode	TensorRT-LLM 3,289 · Pie 3,244 · vLLM 2,389 · SGLang 1,598
H100 SXM 80 GB	Qwen3.6-27B	Batched prefill	vLLM 11,837 · SGLang 11,829 · Pie 11,302 · TensorRT-LLM 10,988
H100 SXM 80 GB	Qwen3.8-27B	Single decode	Pie 52.6 · SGLang 51.9 · TensorRT-LLM 51.4 · vLLM 48.5
H100 SXM 80 GB	Qwen3.8-27B	Single prefill	Pie 9,556 · SGLang 9,066 · TensorRT-LLM 6,706 · vLLM 6,343
H100 SXM 80 GB	Qwen3.8-27B	Batched decode	TensorRT-LLM 3,286 · Pie 3,011 · vLLM 2,395 · SGLang 1,704
H100 SXM 80 GB	Qwen3.8-27B	Batched prefill	vLLM 12,067 · TensorRT-LLM 11,768 · SGLang 11,371 · Pie 11,150
H100 SXM 80 GB	Qwen3.5-9B	Single decode	Pie 165 · SGLang 157 · TensorRT-LLM 155 · vLLM 150
H100 SXM 80 GB	Qwen3.5-9B	Single prefill	Pie 22,342 · SGLang 19,806 · TensorRT-LLM 11,078 · vLLM 10,052
H100 SXM 80 GB	Qwen3.5-9B	Batched decode	Pie 13,843 · vLLM 12,182 · SGLang 11,934 · TensorRT-LLM 10,885
H100 SXM 80 GB	Qwen3.5-9B	Batched prefill	vLLM 39,385 · SGLang 39,172 · TensorRT-LLM 38,538 · Pie 38,040
L40S 46 GB	Qwen3.5-0.8B	Single decode	Pie 381 · SGLang 380 · TensorRT-LLM 378 · vLLM 357
L40S 46 GB	Qwen3.5-0.8B	Single prefill	Pie 54,756 · SGLang 27,600 · vLLM 17,299 · TensorRT-LLM 10,602
L40S 46 GB	Qwen3.5-0.8B	Batched decode	TensorRT-LLM 11,314
L40S 46 GB	Qwen3.5-0.8B	Batched prefill	Pie 107,529 · SGLang 102,654 · TensorRT-LLM 83,972 · vLLM 50,178
L40S 46 GB	Qwen3-0.6B	Single decode	Pie 418 · SGLang 415 · vLLM 405 · TensorRT-LLM 398
L40S 46 GB	Qwen3-0.6B	Single prefill	Pie 71,065 · SGLang 55,591 · vLLM 34,553 · TensorRT-LLM 15,391
L40S 46 GB	Qwen3-0.6B	Batched decode	Pie 15,940 · vLLM 15,785 · SGLang 15,098 · TensorRT-LLM 12,600
L40S 46 GB	Qwen3-0.6B	Batched prefill	Pie 152,452 · vLLM 152,450 · SGLang 141,947 · TensorRT-LLM 135,406
L40S 46 GB	Qwen3.5-9B	Single decode	Pie 45.3 · TensorRT-LLM 45 · SGLang 44.9 · vLLM 44.5
L40S 46 GB	Qwen3.5-9B	Single prefill	SGLang 9,305 · vLLM 9,091 · Pie 8,933 · TensorRT-LLM 7,641
L40S 46 GB	Qwen3.5-9B	Batched decode	Pie 3,571 · TensorRT-LLM 3,521 · vLLM 2,933 · SGLang 2,620
L40S 46 GB	Qwen3.5-9B	Batched prefill	vLLM 12,587 · SGLang 12,001 · TensorRT-LLM 11,554 · Pie 10,951
RTX 4090 24 GB	Qwen3.5-0.8B	Single decode	SGLang 490 · Pie 470 · vLLM 462 · TensorRT-LLM 459
RTX 4090 24 GB	Qwen3.5-0.8B	Single prefill	Pie 56,365 · SGLang 25,522 · vLLM 15,752 · TensorRT-LLM 10,261
RTX 4090 24 GB	Qwen3.5-0.8B	Batched decode	TensorRT-LLM 13,359
RTX 4090 24 GB	Qwen3.5-0.8B	Batched prefill	Pie 94,322 · TensorRT-LLM 86,157 · SGLang 67,051 · vLLM 27,981
RTX 4090 24 GB	Qwen3-0.6B	Single decode	Pie 482 · TensorRT-LLM 472 · SGLang 464 · vLLM 441
RTX 4090 24 GB	Qwen3-0.6B	Single prefill	Pie 61,808 · SGLang 39,689 · vLLM 24,408 · TensorRT-LLM 13,446
RTX 4090 24 GB	Qwen3-0.6B	Batched decode	vLLM 18,108 · SGLang 17,243 · Pie 16,473 · TensorRT-LLM 15,168
RTX 4090 24 GB	Qwen3-0.6B	Batched prefill	vLLM 125,628 · TensorRT-LLM 121,096 · Pie 114,619 · SGLang 114,218
RTX 4090 24 GB	Qwen3.5-9B	Single decode	SGLang 57.1 · vLLM 56.6 · Pie 54.9
RTX 4090 24 GB	Qwen3.5-9B	Single prefill	SGLang 7,932 · vLLM 7,291 · Pie 6,641
RTX 4090 24 GB	Qwen3.5-9B	Batched decode	vLLM 810 · SGLang 727 · Pie 465
RTX 4090 24 GB	Qwen3.5-9B	Batched prefill	vLLM 10,124 · SGLang 9,789 · Pie 7,952
A100 SXM 80 GB	Qwen3.5-0.8B	Single decode	Pie 527 · SGLang 517 · TensorRT-LLM 482 · vLLM 457
A100 SXM 80 GB	Qwen3.5-0.8B	Single prefill	Pie 41,156 · SGLang 20,161 · vLLM 14,213 · TensorRT-LLM 12,735
A100 SXM 80 GB	Qwen3.5-0.8B	Batched decode	Pie 28,011 · SGLang 20,756 · TensorRT-LLM 18,708 · vLLM 16,682
A100 SXM 80 GB	Qwen3.5-0.8B	Batched prefill	Pie 121,257 · TensorRT-LLM 110,830 · SGLang 93,624 · vLLM 50,468
A100 SXM 80 GB	Qwen3-0.6B	Single decode	SGLang 518 · Pie 515 · vLLM 482 · TensorRT-LLM 441
A100 SXM 80 GB	Qwen3-0.6B	Single prefill	Pie 60,557 · SGLang 45,587 · vLLM 28,642 · TensorRT-LLM 19,416
A100 SXM 80 GB	Qwen3-0.6B	Batched decode	Pie 26,067 · SGLang 24,483 · vLLM 24,243 · TensorRT-LLM 22,263
A100 SXM 80 GB	Qwen3-0.6B	Batched prefill	Pie 163,016 · TensorRT-LLM 154,723 · SGLang 146,912 · vLLM 133,029
A100 SXM 80 GB	Qwen3.6-27B	Single decode	Pie 31.2 · SGLang 29.4 · TensorRT-LLM 29.2 · vLLM 28.8
A100 SXM 80 GB	Qwen3.6-27B	Single prefill	Pie 3,763 · SGLang 3,601 · TensorRT-LLM 3,469 · vLLM 3,364
A100 SXM 80 GB	Qwen3.6-27B	Batched decode	TensorRT-LLM 1,570 · Pie 1,544 · vLLM 1,157 · SGLang 828
A100 SXM 80 GB	Qwen3.6-27B	Batched prefill	SGLang 4,470 · TensorRT-LLM 4,415 · Pie 4,364 · vLLM 4,256
A100 SXM 80 GB	Qwen3.8-27B	Single decode	Pie 31.3 · SGLang 29.5 · TensorRT-LLM 29.2 · vLLM 29
A100 SXM 80 GB	Qwen3.8-27B	Single prefill	Pie 3,606 · SGLang 3,510 · vLLM 3,485 · TensorRT-LLM 3,436
A100 SXM 80 GB	Qwen3.8-27B	Batched decode	TensorRT-LLM 1,573 · Pie 1,542 · vLLM 1,156 · SGLang 825
A100 SXM 80 GB	Qwen3.8-27B	Batched prefill	TensorRT-LLM 4,408 · SGLang 4,118 · Pie 4,083 · vLLM 3,977
A100 SXM 80 GB	Qwen3.5-9B	Single decode	Pie 95 · SGLang 89.9 · TensorRT-LLM 88.9 · vLLM 88.9
A100 SXM 80 GB	Qwen3.5-9B	Single prefill	Pie 11,199 · SGLang 10,823 · vLLM 10,094 · TensorRT-LLM 9,285
A100 SXM 80 GB	Qwen3.5-9B	Batched decode	TensorRT-LLM 5,937 · Pie 5,544 · vLLM 5,219 · SGLang 4,944
A100 SXM 80 GB	Qwen3.5-9B	Batched prefill	SGLang 15,182 · vLLM 14,682 · TensorRT-LLM 14,515 · Pie 14,345
A40 48 GB	Qwen3.5-0.8B	Single decode	SGLang 292 · Pie 285 · TensorRT-LLM 282 · vLLM 280
A40 48 GB	Qwen3.5-0.8B	Single prefill	Pie 24,869 · SGLang 19,181 · vLLM 10,666 · TensorRT-LLM 9,341
A40 48 GB	Qwen3.5-0.8B	Batched decode	Pie 11,295 · vLLM 8,557 · SGLang 8,277 · TensorRT-LLM 7,934
A40 48 GB	Qwen3.5-0.8B	Batched prefill	Pie 57,102 · TensorRT-LLM 52,794 · SGLang 50,291 · vLLM 48,465
A40 48 GB	Qwen3-0.6B	Single decode	Pie 340 · SGLang 325 · vLLM 316 · TensorRT-LLM 291
A40 48 GB	Qwen3-0.6B	Single prefill	Pie 33,643 · SGLang 26,093 · vLLM 18,365 · TensorRT-LLM 13,701
A40 48 GB	Qwen3-0.6B	Batched decode	Pie 11,383 · vLLM 10,604 · SGLang 10,350 · TensorRT-LLM 8,483
A40 48 GB	Qwen3-0.6B	Batched prefill	Pie 77,990 · TensorRT-LLM 73,093 · vLLM 69,917 · SGLang 61,985
A40 48 GB	Qwen3.5-9B	Single decode	Pie 36.5 · SGLang 33.2 · TensorRT-LLM 33.1 · vLLM 32.9
A40 48 GB	Qwen3.5-9B	Single prefill	Pie 5,596 · SGLang 5,152 · TensorRT-LLM 4,928 · vLLM 4,334
A40 48 GB	Qwen3.5-9B	Batched decode	Pie 2,522 · TensorRT-LLM 2,338 · vLLM 1,973 · SGLang 1,827
A40 48 GB	Qwen3.5-9B	Batched prefill	SGLang 6,534 · Pie 6,492 · vLLM 6,486 · TensorRT-LLM 6,315

Throughput on AMD GPU, tokens per second: every engine on the same model and machine, fastest first
Machine	Model	Setup	Engines (tok/s)
MI355X 288 GB	Qwen3.5-0.8B	Single decode	Pie 751 · SGLang 718 · vLLM 546
MI355X 288 GB	Qwen3.5-0.8B	Single prefill	Pie 80,944 · vLLM 24,473 · SGLang 20,517
MI355X 288 GB	Qwen3.5-0.8B	Batched decode	SGLang 58,959 · Pie 41,995 · vLLM 32,585
MI355X 288 GB	Qwen3.5-0.8B	Batched prefill	Pie 323,319 · SGLang 308,832 · vLLM 88,270
MI355X 288 GB	Qwen3-0.6B	Single decode	SGLang 665 · vLLM 631 · Pie 558
MI355X 288 GB	Qwen3-0.6B	Single prefill	Pie 91,340 · vLLM 35,877 · SGLang 34,159
MI355X 288 GB	Qwen3-0.6B	Batched decode	SGLang 64,176 · Pie 59,280 · vLLM 50,204
MI355X 288 GB	Qwen3-0.6B	Batched prefill	SGLang 495,664 · Pie 450,305 · vLLM 146,985
MI355X 288 GB	Qwen3.6-27B	Single decode	Pie 93 · vLLM 84.3 · SGLang 82.7
MI355X 288 GB	Qwen3.6-27B	Single prefill	Pie 11,244 · vLLM 10,554 · SGLang 9,606
MI355X 288 GB	Qwen3.6-27B	Batched decode	vLLM 6,267 · SGLang 6,204 · Pie 5,706
MI355X 288 GB	Qwen3.6-27B	Batched prefill	SGLang 23,034 · Pie 20,333 · vLLM 20,078
MI355X 288 GB	Qwen3.6-35B	Single decode	SGLang 276 · Pie 273 · vLLM 169
MI355X 288 GB	Qwen3.6-35B	Single prefill	Pie 22,703 · SGLang 9,868 · vLLM 8,539
MI355X 288 GB	Qwen3.6-35B	Batched decode	SGLang 17,564 · vLLM 13,895 · Pie 11,278
MI355X 288 GB	Qwen3.6-35B	Batched prefill	SGLang 90,253 · Pie 72,563 · vLLM 33,737
MI355X 288 GB	Qwen3.8-27B	Single decode	Pie 93.1 · vLLM 84.2 · SGLang 82.7
MI355X 288 GB	Qwen3.8-27B	Single prefill	Pie 11,326 · vLLM 10,430 · SGLang 9,564
MI355X 288 GB	Qwen3.8-27B	Batched decode	vLLM 6,275 · SGLang 6,170 · Pie 5,727
MI355X 288 GB	Qwen3.8-27B	Batched prefill	SGLang 20,599 · vLLM 19,805 · Pie 19,559
MI355X 288 GB	Qwen3.5-9B	Single decode	Pie 252 · SGLang 233 · vLLM 217
MI355X 288 GB	Qwen3.5-9B	Single prefill	Pie 30,068 · vLLM 15,707 · SGLang 15,151
MI355X 288 GB	Qwen3.5-9B	Batched decode	SGLang 19,432 · vLLM 18,307 · Pie 16,037
MI355X 288 GB	Qwen3.5-9B	Batched prefill	SGLang 73,646 · Pie 70,165 · vLLM 62,747
MI350X 288 GB	Qwen3.5-0.8B	Single decode	Pie 714 · SGLang 649 · vLLM 520
MI350X 288 GB	Qwen3.5-0.8B	Single prefill	Pie 67,237 · vLLM 23,457 · SGLang 19,527
MI350X 288 GB	Qwen3-0.6B	Single decode	SGLang 617 · vLLM 602 · Pie 539
MI350X 288 GB	Qwen3-0.6B	Single prefill	Pie 83,807 · vLLM 33,787 · SGLang 32,997
MI350X 288 GB	Qwen3.6-27B	Single decode	Pie 90.9 · vLLM 82.3 · SGLang 80.2
MI350X 288 GB	Qwen3.6-27B	Single prefill	vLLM 10,066 · Pie 9,753 · SGLang 9,221
MI350X 288 GB	Qwen3.6-35B	Single decode	SGLang 265 · Pie 264 · vLLM 162
MI350X 288 GB	Qwen3.6-35B	Single prefill	Pie 20,896 · SGLang 9,465 · vLLM 8,496
MI350X 288 GB	Qwen3.8-27B	Single decode	Pie 91 · vLLM 82.4 · SGLang 80.4
MI350X 288 GB	Qwen3.8-27B	Single prefill	vLLM 10,164 · Pie 9,755 · SGLang 9,635
MI350X 288 GB	Qwen3.5-9B	Single decode	Pie 246 · vLLM 211 · SGLang 190
MI350X 288 GB	Qwen3.5-9B	Single prefill	Pie 27,697 · vLLM 14,814 · SGLang 14,085
MI300X 192 GB	Qwen3.5-0.8B	Single decode	Pie 673 · SGLang 572 · vLLM 507
MI300X 192 GB	Qwen3.5-0.8B	Single prefill	Pie 42,602 · SGLang 14,059 · vLLM 12,914
MI300X 192 GB	Qwen3.5-0.8B	Batched decode	Pie 30,823 · SGLang 24,012 · vLLM 18,347
MI300X 192 GB	Qwen3.5-0.8B	Batched prefill	Pie 98,705 · SGLang 90,331 · vLLM 40,284
MI300X 192 GB	Qwen3-0.6B	Single decode	vLLM 676 · SGLang 541 · Pie 527
MI300X 192 GB	Qwen3-0.6B	Single prefill	Pie 50,202 · SGLang 19,920 · vLLM 19,633
MI300X 192 GB	Qwen3-0.6B	Batched decode	SGLang 32,132 · vLLM 28,108 · Pie 26,936
MI300X 192 GB	Qwen3-0.6B	Batched prefill	Pie 176,715 · SGLang 151,359 · vLLM 66,983
MI300X 192 GB	Qwen3.6-27B	Single decode	Pie 67.9 · vLLM 64.7 · SGLang 53.9
MI300X 192 GB	Qwen3.6-27B	Single prefill	SGLang 5,523 · vLLM 5,310 · Pie 4,007
MI300X 192 GB	Qwen3.6-27B	Batched decode	vLLM 2,768 · SGLang 2,727
MI300X 192 GB	Qwen3.6-27B	Batched prefill	SGLang 8,708 · vLLM 8,381
MI300X 192 GB	Qwen3.6-35B	Single decode	Pie 239 · SGLang 216 · vLLM 159
MI300X 192 GB	Qwen3.6-35B	Single prefill	Pie 12,837 · SGLang 6,959 · vLLM 5,699
MI300X 192 GB	Qwen3.6-35B	Batched decode	Pie 7,587 · SGLang 7,464 · vLLM 6,308
MI300X 192 GB	Qwen3.6-35B	Batched prefill	Pie 38,645 · SGLang 23,035 · vLLM 19,437
MI300X 192 GB	Qwen3.8-27B	Single decode	Pie 67.8 · vLLM 64.2 · SGLang 52.9
MI300X 192 GB	Qwen3.8-27B	Single prefill	SGLang 5,391 · vLLM 5,134 · Pie 4,002
MI300X 192 GB	Qwen3.8-27B	Batched decode	Pie 3,186 · vLLM 3,087 · SGLang 2,928
MI300X 192 GB	Qwen3.8-27B	Batched prefill	SGLang 9,160 · Pie 8,832 · vLLM 8,013

Throughput on Google TPU, tokens per second: every engine on the same model and machine, fastest first
Machine	Model	Setup	Engines (tok/s)
TPU v6e 32 GB	Qwen3.5-0.8B	Single decode	Pie 688 · vLLM 348 · SGLang 214
TPU v6e 32 GB	Qwen3.5-0.8B	Single prefill	vLLM 49,984 · Pie 49,751 · SGLang 28,941
TPU v6e 32 GB	Qwen3.5-0.8B	Batched decode	vLLM 16,189 · Pie 14,635 · SGLang 5,464
TPU v6e 32 GB	Qwen3.5-0.8B	Batched prefill	vLLM 140,661 · Pie 116,810 · SGLang 116,589
TPU v6e 32 GB	Qwen3-0.6B	Single decode	Pie 694 · vLLM 421 · SGLang 329
TPU v6e 32 GB	Qwen3-0.6B	Single prefill	vLLM 61,879 · Pie 58,756 · SGLang 38,262
TPU v6e 32 GB	Qwen3-0.6B	Batched decode	vLLM 12,086 · Pie 11,620 · SGLang 7,225
TPU v6e 32 GB	Qwen3-0.6B	Batched prefill	Pie 236,161 · vLLM 230,690 · SGLang 112,732
TPU v5e 16 GB	Qwen3.5-0.8B	Single decode	Pie 467 · vLLM 271 · SGLang 175
TPU v5e 16 GB	Qwen3.5-0.8B	Single prefill	Pie 43,296 · vLLM 32,315 · SGLang 6,798
TPU v5e 16 GB	Qwen3.5-0.8B	Batched decode	vLLM 8,759 · Pie 8,298 · SGLang 3,001
TPU v5e 16 GB	Qwen3.5-0.8B	Batched prefill	vLLM 73,279 · SGLang 60,116 · Pie 54,344
TPU v5e 16 GB	Qwen3-0.6B	Single decode	Pie 555 · vLLM 326 · SGLang 264
TPU v5e 16 GB	Qwen3-0.6B	Single prefill	Pie 61,458 · vLLM 38,439 · SGLang 26,055
TPU v5e 16 GB	Qwen3-0.6B	Batched decode	vLLM 9,028 · Pie 8,124 · SGLang 5,240
TPU v5e 16 GB	Qwen3-0.6B	Batched prefill	vLLM 109,487 · Pie 106,037 · SGLang 73,324

Throughput on Browser, tokens per second: every engine on the same model and machine, fastest first
Machine	Model	Setup	Engines (tok/s)
RTX 5090 Chrome 32 GB	Qwen3.5-4B	Single decode	Pie 97.3 · WebLLM 67 · llama.cpp WebGPU 42
RTX 5090 Chrome 32 GB	Qwen3.5-4B	Single prefill	Pie 1,990 · WebLLM 1,314 · llama.cpp WebGPU 858
RTX 5090 Chrome 32 GB	Qwen3.5-4B	Batched decode	Pie 409 · llama.cpp WebGPU 117
RTX 5090 Chrome 32 GB	Qwen3.5-4B	Batched prefill	Pie 2,092 · llama.cpp WebGPU 1,098
RTX 5090 Chrome 32 GB	Qwen3.5-9B	Single decode	Pie 87.6 · WebLLM 60.2 · llama.cpp WebGPU 40.3
RTX 5090 Chrome 32 GB	Qwen3.5-9B	Single prefill	Pie 1,327 · WebLLM 863 · llama.cpp WebGPU 553
RTX 5090 Chrome 32 GB	Qwen3.5-9B	Batched decode	Pie 292 · llama.cpp WebGPU 91.5
RTX 5090 Chrome 32 GB	Qwen3.5-9B	Batched prefill	Pie 1,326 · llama.cpp WebGPU 662
RTX 4090 Chrome 24 GB	Qwen3.5-4B	Single decode	Pie 107 · llama.cpp WebGPU 46.3 · WebLLM 33.3
RTX 4090 Chrome 24 GB	Qwen3.5-4B	Single prefill	Pie 1,871 · WebLLM 915 · llama.cpp WebGPU 667
RTX 4090 Chrome 24 GB	Qwen3.5-4B	Batched decode	Pie 331 · llama.cpp WebGPU 99.9
RTX 4090 Chrome 24 GB	Qwen3.5-4B	Batched prefill	Pie 1,985 · llama.cpp WebGPU 683
RTX 4090 Chrome 24 GB	Qwen3.5-9B	Single decode	Pie 84.9 · llama.cpp WebGPU 46 · WebLLM 35.1
RTX 4090 Chrome 24 GB	Qwen3.5-9B	Single prefill	Pie 1,180 · WebLLM 658 · llama.cpp WebGPU 382
RTX 4090 Chrome 24 GB	Qwen3.5-9B	Batched decode	Pie 251 · llama.cpp WebGPU 73
RTX 4090 Chrome 24 GB	Qwen3.5-9B	Batched prefill	Pie 1,220 · llama.cpp WebGPU 391