NVIDIA GPU
Results as tablesRelease main-f03640150, as of 2026-11-13
A snapshot taken when this site was last built. The charts above read live data and may be newer. The same tables as markdown.
| Machine | Model | Setup | Engines (tok/s) |
|---|---|---|---|
| B200 180 GB | Qwen3.5-0.8B | Single decode | Pie 675 · SGLang 558 · vLLM 397 · TensorRT-LLM 275 |
| B200 180 GB | Qwen3.5-0.8B | Single prefill | Pie 45,816 · SGLang 17,750 · vLLM 10,914 · TensorRT-LLM 4,764 |
| B200 180 GB | Qwen3.5-0.8B | Batched decode | Pie 84,398 · SGLang 55,409 · vLLM 39,249 · TensorRT-LLM 6,454 |
| B200 180 GB | Qwen3.5-0.8B | Batched prefill | Pie 470,170 · SGLang 439,719 · vLLM 254,530 · TensorRT-LLM 146,073 |
| B200 180 GB | Qwen3-0.6B | Single decode | Pie 969 · SGLang 637 · vLLM 443 · TensorRT-LLM 297 |
| B200 180 GB | Qwen3-0.6B | Single prefill | Pie 59,499 · SGLang 32,206 · vLLM 12,628 · TensorRT-LLM 7,172 |
| B200 180 GB | Qwen3-0.6B | Batched decode | Pie 90,301 · SGLang 56,847 · vLLM 43,917 · TensorRT-LLM 9,524 |
| B200 180 GB | Qwen3-0.6B | Batched prefill | Pie 662,963 · SGLang 377,453 · vLLM 211,925 · TensorRT-LLM 72,704 |
| B200 180 GB | Gemma-4-E4B | Single decode | Pie 290 · vLLM 252 · SGLang 237 · TensorRT-LLM 229 |
| B200 180 GB | Gemma-4-E4B | Single prefill | Pie 36,744 · SGLang 9,256 · vLLM 8,622 · TensorRT-LLM 4,258 |
| B200 180 GB | Gemma-4-E4B | Batched decode | SGLang 40,383 · Pie 37,249 · vLLM 33,247 · TensorRT-LLM 8,754 |
| B200 180 GB | Gemma-4-E4B | Batched prefill | vLLM 126,822 · SGLang 121,701 · Pie 116,759 · TensorRT-LLM 100,267 |
| B200 180 GB | Qwen3.6-27B | Single decode | TensorRT-LLM 103 · Pie 102 · SGLang 102 · vLLM 95.2 |
| B200 180 GB | Qwen3.6-27B | Single prefill | Pie 10,947 · SGLang 8,752 · vLLM 4,872 · TensorRT-LLM 3,601 |
| B200 180 GB | Qwen3.6-27B | Batched decode | Pie 7,765 · vLLM 7,538 · SGLang 6,519 · TensorRT-LLM 6,171 |
| B200 180 GB | Qwen3.6-27B | Batched prefill | vLLM 23,790 · TensorRT-LLM 23,567 · SGLang 21,905 · Pie 21,249 |
| B200 180 GB | GPT-OSS-20B | Single decode | SGLang 569 · Pie 550 · vLLM 456 · TensorRT-LLM 285 |
| B200 180 GB | GPT-OSS-20B | Single prefill | Pie 36,203 · SGLang 30,644 · vLLM 27,768 · TensorRT-LLM 7,475 |
| B200 180 GB | GPT-OSS-20B | Batched decode | vLLM 43,895 · SGLang 38,625 · Pie 35,733 · TensorRT-LLM 11,287 |
| B200 180 GB | GPT-OSS-20B | Batched prefill | vLLM 138,139 · Pie 115,227 · SGLang 94,130 · TensorRT-LLM 79,131 |
| B200 180 GB | GPT-OSS-120B | Single decode | TensorRT-LLM 428 · SGLang 418 · vLLM 397 · Pie 378 |
| B200 180 GB | GPT-OSS-120B | Single prefill | Pie 19,318 · vLLM 18,538 · SGLang 18,177 · TensorRT-LLM 13,497 |
| B200 180 GB | GPT-OSS-120B | Batched decode | vLLM 28,699 · SGLang 24,257 · Pie 20,408 · TensorRT-LLM 11,495 |
| B200 180 GB | GPT-OSS-120B | Batched prefill | vLLM 90,339 · Pie 72,754 · SGLang 62,902 · TensorRT-LLM 62,034 |
| B200 180 GB | Qwen3.6-35B | Single decode | SGLang 396 · Pie 367 · vLLM 325 · TensorRT-LLM 279 |
| B200 180 GB | Qwen3.6-35B | Single prefill | Pie 24,922 · SGLang 13,101 · vLLM 3,924 · TensorRT-LLM 3,257 |
| B200 180 GB | Qwen3.6-35B | Batched decode | vLLM 22,721 · Pie 22,450 · SGLang 18,545 · TensorRT-LLM 10,779 |
| B200 180 GB | Qwen3.6-35B | Batched prefill | vLLM 126,246 · SGLang 112,748 · Pie 107,876 · TensorRT-LLM 81,207 |
| B200 180 GB | Muse-Glimmer-30B | Single decode | Pie 102 · SGLang 95.8 · vLLM 87.9 |
| B200 180 GB | Muse-Glimmer-30B | Single prefill | Pie 14,663 · SGLang 8,779 · vLLM 7,309 |
| B200 180 GB | Muse-Glimmer-30B | Batched decode | Pie 12,979 · vLLM 12,602 · SGLang 7,343 |
| B200 180 GB | Muse-Glimmer-30B | Batched prefill | SGLang 24,352 · vLLM 23,581 · Pie 22,238 |
| B200 180 GB | Gemma-4-26B | Single decode | Pie 341 · SGLang 291 · vLLM 256 · TensorRT-LLM 217 |
| B200 180 GB | Gemma-4-26B | Single prefill | Pie 28,266 · SGLang 7,926 · vLLM 6,521 · TensorRT-LLM 3,699 |
| B200 180 GB | Gemma-4-26B | Batched decode | vLLM 31,100 · Pie 27,712 · SGLang 27,511 · TensorRT-LLM 9,763 |
| B200 180 GB | Gemma-4-26B | Batched prefill | vLLM 114,958 · Pie 114,220 · SGLang 104,786 · TensorRT-LLM 49,721 |
| B200 180 GB | Gemma-4-31B | Single decode | SGLang 82.2 · Pie 81.5 · TensorRT-LLM 80.9 · vLLM 76.9 |
| B200 180 GB | Gemma-4-31B | Single prefill | Pie 12,555 · SGLang 8,174 · vLLM 7,335 · TensorRT-LLM 3,534 |
| B200 180 GB | Gemma-4-31B | Batched decode | vLLM 6,493 · Pie 6,134 · SGLang 6,021 · TensorRT-LLM 4,930 |
| B200 180 GB | Gemma-4-31B | Batched prefill | SGLang 19,932 · vLLM 19,717 · TensorRT-LLM 18,880 · Pie 18,509 |
| B200 180 GB | Qwen3.8-27B | Single decode | TensorRT-LLM 103 · SGLang 102 · Pie 102 · vLLM 94.5 |
| B200 180 GB | Qwen3.8-27B | Single prefill | Pie 11,245 · SGLang 8,779 · vLLM 4,909 · TensorRT-LLM 3,839 |
| B200 180 GB | Qwen3.8-27B | Batched decode | vLLM 7,539 · Pie 6,802 · SGLang 6,540 · TensorRT-LLM 5,549 |
| B200 180 GB | Qwen3.8-27B | Batched prefill | TensorRT-LLM 22,873 · SGLang 22,350 · vLLM 21,806 · Pie 21,218 |
| B200 180 GB | Qwen3.5-9B | Single decode | Pie 284 · SGLang 280 · TensorRT-LLM 275 · vLLM 248 |
| B200 180 GB | Qwen3.5-9B | Single prefill | Pie 26,681 · SGLang 20,861 · TensorRT-LLM 11,563 · vLLM 6,042 |
| B200 180 GB | Qwen3.5-9B | Batched decode | Pie 23,777 · vLLM 22,510 · SGLang 20,434 · TensorRT-LLM 10,027 |
| B200 180 GB | Qwen3.5-9B | Batched prefill | SGLang 73,903 · vLLM 72,879 · Pie 71,320 · TensorRT-LLM 70,860 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Single decode | TensorRT-LLM 573 · SGLang 567 · Pie 562 · vLLM 385 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Single prefill | Pie 63,884 · SGLang 29,290 · vLLM 14,907 · TensorRT-LLM 11,588 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Batched decode | TensorRT-LLM 21,984 |
| RTX PRO 6000 96 GB | Qwen3.5-0.8B | Batched prefill | Pie 217,989 · vLLM 191,574 · SGLang 186,448 · TensorRT-LLM 175,675 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Single decode | Pie 681 · SGLang 636 · vLLM 626 · TensorRT-LLM 580 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Single prefill | Pie 85,375 · SGLang 73,009 · vLLM 68,556 · TensorRT-LLM 16,932 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Batched decode | Pie 31,859 · vLLM 29,507 · SGLang 29,228 · TensorRT-LLM 23,224 |
| RTX PRO 6000 96 GB | Qwen3-0.6B | Batched prefill | Pie 290,442 · vLLM 261,470 · SGLang 248,360 · TensorRT-LLM 198,607 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Single decode | Pie 133 · vLLM 117 · SGLang 114 · TensorRT-LLM 105 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Single prefill | Pie 22,255 · vLLM 14,936 · SGLang 11,302 · TensorRT-LLM 7,373 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Batched decode | Pie 14,827 · vLLM 14,451 · SGLang 13,615 · TensorRT-LLM 11,876 |
| RTX PRO 6000 96 GB | Gemma-4-E4B | Batched prefill | vLLM 40,618 · Pie 38,977 · TensorRT-LLM 37,114 · SGLang 35,100 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Single decode | Pie 27.7 · TensorRT-LLM 27.2 · SGLang 27.1 · vLLM 26.7 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Single prefill | Pie 5,341 · SGLang 5,210 · TensorRT-LLM 4,707 · vLLM 4,676 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Batched decode | Pie 1,834 · TensorRT-LLM 1,814 · vLLM 1,560 · SGLang 1,260 |
| RTX PRO 6000 96 GB | Qwen3.6-27B | Batched prefill | TensorRT-LLM 7,025 · vLLM 6,945 · Pie 6,865 · SGLang 6,731 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Single decode | Pie 280 · vLLM 260 · SGLang 245 · TensorRT-LLM 227 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Single prefill | Pie 24,159 · SGLang 19,338 · vLLM 18,268 · TensorRT-LLM 13,991 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Batched decode | TensorRT-LLM 23,048 · Pie 19,240 · vLLM 17,141 · SGLang 13,874 |
| RTX PRO 6000 96 GB | GPT-OSS-20B | Batched prefill | TensorRT-LLM 65,736 · Pie 49,242 · vLLM 41,689 · SGLang 31,920 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Single decode | Pie 206 · vLLM 188 · SGLang 178 · TensorRT-LLM 156 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Single prefill | Pie 12,849 · vLLM 11,919 · TensorRT-LLM 11,822 · SGLang 10,070 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Batched decode | TensorRT-LLM 11,517 · vLLM 8,470 · Pie 7,998 · SGLang 3,673 |
| RTX PRO 6000 96 GB | GPT-OSS-120B | Batched prefill | TensorRT-LLM 42,843 · Pie 30,372 · vLLM 27,725 · SGLang 21,200 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Single decode | SGLang 192 · Pie 186 · TensorRT-LLM 179 · vLLM 167 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Single prefill | Pie 12,824 · SGLang 12,686 · vLLM 11,143 · TensorRT-LLM 5,868 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Batched decode | TensorRT-LLM 4,831 · Pie 4,747 · vLLM 3,522 · SGLang 2,701 |
| RTX PRO 6000 96 GB | Qwen3.6-35B | Batched prefill | SGLang 41,034 · Pie 40,099 · TensorRT-LLM 39,497 · vLLM 38,140 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Single decode | Pie 27.1 · SGLang 26 · vLLM 25.9 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Single prefill | Pie 6,293 · SGLang 5,484 · vLLM 5,454 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Batched decode | Pie 3,483 · vLLM 3,244 · SGLang 3,158 |
| RTX PRO 6000 96 GB | Muse-Glimmer-30B | Batched prefill | Pie 7,499 · SGLang 6,966 · vLLM 6,958 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Single decode | Pie 154 · SGLang 139 · vLLM 133 · TensorRT-LLM 118 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Single prefill | Pie 14,030 · vLLM 11,111 · SGLang 9,599 · TensorRT-LLM 6,621 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Batched decode | Pie 6,892 · vLLM 6,730 · SGLang 4,927 · TensorRT-LLM 4,730 |
| RTX PRO 6000 96 GB | Gemma-4-26B | Batched prefill | vLLM 38,562 · Pie 37,461 · TensorRT-LLM 35,825 · SGLang 32,018 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Single decode | Pie 23.4 · SGLang 22.7 · vLLM 22.5 · TensorRT-LLM 22.5 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Single prefill | Pie 5,304 · TensorRT-LLM 4,209 · SGLang 4,201 · vLLM 4,139 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Batched decode | Pie 1,088 · vLLM 1,047 · SGLang 889 · TensorRT-LLM 729 |
| RTX PRO 6000 96 GB | Gemma-4-31B | Batched prefill | vLLM 6,088 · Pie 5,965 · TensorRT-LLM 5,939 · SGLang 5,437 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Single decode | Pie 27.9 · SGLang 27.2 · TensorRT-LLM 27.2 · vLLM 26.6 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Single prefill | Pie 6,061 · SGLang 5,553 · vLLM 5,073 · TensorRT-LLM 4,801 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Batched decode | Pie 1,833 · TensorRT-LLM 1,811 · vLLM 1,561 · SGLang 1,264 |
| RTX PRO 6000 96 GB | Qwen3.8-27B | Batched prefill | TensorRT-LLM 7,125 · vLLM 6,948 · Pie 6,869 · SGLang 6,805 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Single decode | Pie 88.6 · TensorRT-LLM 86.3 · SGLang 86.2 · vLLM 83.8 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Single prefill | Pie 18,324 · SGLang 14,774 · TensorRT-LLM 12,756 · vLLM 9,927 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Batched decode | Pie 8,394 · TensorRT-LLM 7,854 · vLLM 6,422 · SGLang 6,072 |
| RTX PRO 6000 96 GB | Qwen3.5-9B | Batched prefill | vLLM 24,245 · Pie 24,141 · TensorRT-LLM 23,713 · SGLang 22,828 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Single decode | TensorRT-LLM 730 · Pie 724 · SGLang 519 · vLLM 465 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Single prefill | Pie 65,629 · SGLang 19,275 · TensorRT-LLM 15,961 · vLLM 15,449 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Batched decode | Pie 30,630 · TensorRT-LLM 25,679 · SGLang 21,929 · vLLM 21,112 |
| RTX 5090 32 GB | Qwen3.5-0.8B | Batched prefill | Pie 152,028 · TensorRT-LLM 142,701 · SGLang 127,016 · vLLM 98,301 |
| RTX 5090 32 GB | Qwen3-0.6B | Single decode | Pie 769 · TensorRT-LLM 710 · SGLang 708 · vLLM 692 |
| RTX 5090 32 GB | Qwen3-0.6B | Single prefill | Pie 71,535 · SGLang 53,548 · vLLM 29,465 · TensorRT-LLM 21,992 |
| RTX 5090 32 GB | Qwen3-0.6B | Batched decode | Pie 29,583 · SGLang 26,623 · TensorRT-LLM 25,638 · vLLM 25,440 |
| RTX 5090 32 GB | Qwen3-0.6B | Batched prefill | TensorRT-LLM 181,100 · Pie 179,831 · vLLM 140,466 · SGLang 135,364 |
| RTX 5090 32 GB | Gemma-4-E4B | Single decode | Pie 146 · vLLM 140 · SGLang 138 · TensorRT-LLM 132 |
| RTX 5090 32 GB | Gemma-4-E4B | Single prefill | Pie 16,612 · vLLM 13,576 · SGLang 11,174 · TensorRT-LLM 9,338 |
| RTX 5090 32 GB | Gemma-4-E4B | Batched decode | vLLM 9,748 · Pie 9,429 · TensorRT-LLM 4,443 · SGLang 1,453 |
| RTX 5090 32 GB | Gemma-4-E4B | Batched prefill | Pie 24,563 · vLLM 24,181 · TensorRT-LLM 23,879 · SGLang 20,770 |
| RTX 5090 32 GB | GPT-OSS-20B | Single decode | Pie 318 · vLLM 293 · SGLang 279 · TensorRT-LLM 233 |
| RTX 5090 32 GB | GPT-OSS-20B | Single prefill | Pie 19,891 · vLLM 16,812 · SGLang 16,244 · TensorRT-LLM 15,240 |
| RTX 5090 32 GB | GPT-OSS-20B | Batched decode | TensorRT-LLM 16,098 · Pie 11,771 · vLLM 10,893 · SGLang 9,979 |
| RTX 5090 32 GB | GPT-OSS-20B | Batched prefill | TensorRT-LLM 52,472 · Pie 32,873 · vLLM 27,370 · SGLang 23,606 |
| RTX 5090 32 GB | Qwen3.5-9B | Single decode | TensorRT-LLM 98.6 · Pie 98 · SGLang 98 · vLLM 95.8 |
| RTX 5090 32 GB | Qwen3.5-9B | Single prefill | Pie 12,248 · SGLang 11,026 · TensorRT-LLM 10,457 · vLLM 9,412 |
| RTX 5090 32 GB | Qwen3.5-9B | Batched decode | Pie 5,597 · TensorRT-LLM 5,448 · vLLM 4,249 |
| RTX 5090 32 GB | Qwen3.5-9B | Batched prefill | Pie 15,093 · vLLM 14,320 · TensorRT-LLM 14,224 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Single decode | SGLang 444 · TensorRT-LLM 437 · Pie 430 · vLLM 418 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Single prefill | Pie 52,836 · SGLang 22,415 · vLLM 19,021 · TensorRT-LLM 16,945 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Batched decode | Pie 17,703 · TensorRT-LLM 17,227 · vLLM 12,462 · SGLang 11,719 |
| RTX PRO 4500 32 GB | Qwen3.5-0.8B | Batched prefill | Pie 93,699 · TensorRT-LLM 89,324 · SGLang 73,789 · vLLM 60,105 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Single decode | SGLang 482 · Pie 482 · vLLM 473 · TensorRT-LLM 470 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Single prefill | Pie 73,254 · SGLang 42,978 · vLLM 27,066 · TensorRT-LLM 22,853 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Batched decode | Pie 16,795 · TensorRT-LLM 15,826 · vLLM 15,682 · SGLang 15,279 |
| RTX PRO 4500 32 GB | Qwen3-0.6B | Batched prefill | Pie 119,505 · TensorRT-LLM 109,821 · SGLang 105,519 · vLLM 104,344 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Single decode | Pie 79.7 · vLLM 76.8 · SGLang 75.7 · TensorRT-LLM 73.9 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Single prefill | Pie 13,530 · vLLM 10,489 · SGLang 10,066 · TensorRT-LLM 9,639 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Batched decode | Pie 6,424 · vLLM 6,100 · TensorRT-LLM 2,948 · SGLang 2,223 |
| RTX PRO 4500 32 GB | Gemma-4-E4B | Batched prefill | Pie 16,795 · vLLM 15,887 · TensorRT-LLM 14,882 · SGLang 14,107 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Single decode | Pie 183 · vLLM 174 · SGLang 170 · TensorRT-LLM 140 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Single prefill | TensorRT-LLM 14,148 · Pie 13,893 · vLLM 11,703 · SGLang 10,567 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Batched decode | TensorRT-LLM 8,977 · Pie 7,652 · vLLM 6,851 · SGLang 5,833 |
| RTX PRO 4500 32 GB | GPT-OSS-20B | Batched prefill | TensorRT-LLM 27,230 · Pie 20,379 · vLLM 17,659 · SGLang 13,804 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Single decode | TensorRT-LLM 50 · Pie 49.8 · SGLang 49.7 · vLLM 49.4 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Single prefill | Pie 9,139 · SGLang 7,069 · TensorRT-LLM 6,983 · vLLM 6,660 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Batched decode | Pie 3,018 · TensorRT-LLM 2,889 · vLLM 2,267 |
| RTX PRO 4500 32 GB | Qwen3.5-9B | Batched prefill | Pie 10,052 · TensorRT-LLM 8,398 · vLLM 8,365 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Single decode | Pie 929 · TensorRT-LLM 713 · SGLang 643 · vLLM 553 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Single prefill | Pie 69,360 · SGLang 25,328 · vLLM 20,707 · TensorRT-LLM 15,841 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Batched decode | Pie 60,127 · SGLang 41,846 · vLLM 32,182 · TensorRT-LLM 27,648 |
| H200 SXM 141 GB | Qwen3.5-0.8B | Batched prefill | vLLM 340,786 · Pie 305,796 · SGLang 210,282 · TensorRT-LLM 185,697 |
| H200 SXM 141 GB | Qwen3-0.6B | Single decode | Pie 888 · SGLang 664 · vLLM 611 · TensorRT-LLM 582 |
| H200 SXM 141 GB | Qwen3-0.6B | Single prefill | Pie 104,679 · SGLang 56,488 · vLLM 40,553 · TensorRT-LLM 21,679 |
| H200 SXM 141 GB | Qwen3-0.6B | Batched decode | Pie 65,059 · SGLang 54,795 · vLLM 53,991 · TensorRT-LLM 31,471 |
| H200 SXM 141 GB | Qwen3-0.6B | Batched prefill | SGLang 434,754 · vLLM 412,407 · Pie 400,528 · TensorRT-LLM 252,916 |
| H200 SXM 141 GB | Gemma-4-E4B | Single decode | Pie 228 · SGLang 191 · vLLM 189 · TensorRT-LLM 162 |
| H200 SXM 141 GB | Gemma-4-E4B | Single prefill | Pie 37,639 · vLLM 25,378 · SGLang 13,493 · TensorRT-LLM 8,769 |
| H200 SXM 141 GB | Gemma-4-E4B | Batched decode | SGLang 24,229 · vLLM 23,987 · Pie 23,721 · TensorRT-LLM 21,491 |
| H200 SXM 141 GB | Gemma-4-E4B | Batched prefill | vLLM 69,777 · Pie 65,371 · SGLang 61,185 · TensorRT-LLM 59,231 |
| H200 SXM 141 GB | Qwen3.6-27B | Single decode | Pie 73 · SGLang 68.5 · TensorRT-LLM 67.9 · vLLM 65.6 |
| H200 SXM 141 GB | Qwen3.6-27B | Single prefill | SGLang 9,642 · Pie 9,342 · vLLM 8,851 · TensorRT-LLM 6,764 |
| H200 SXM 141 GB | Qwen3.6-27B | Batched decode | vLLM 4,369 · Pie 4,368 · TensorRT-LLM 4,104 · SGLang 3,906 |
| H200 SXM 141 GB | Qwen3.6-27B | Batched prefill | vLLM 12,330 · SGLang 12,215 · TensorRT-LLM 11,450 · Pie 11,441 |
| H200 SXM 141 GB | GPT-OSS-20B | Single decode | Pie 438 · vLLM 328 · TensorRT-LLM 324 · SGLang 317 |
| H200 SXM 141 GB | GPT-OSS-20B | Single prefill | vLLM 29,595 · Pie 28,862 · SGLang 25,162 · TensorRT-LLM 10,227 |
| H200 SXM 141 GB | GPT-OSS-20B | Batched decode | Pie 24,792 · SGLang 22,032 · vLLM 22,012 · TensorRT-LLM 20,447 |
| H200 SXM 141 GB | GPT-OSS-20B | Batched prefill | SGLang 66,327 · Pie 60,913 · TensorRT-LLM 60,737 · vLLM 58,424 |
| H200 SXM 141 GB | GPT-OSS-120B | Single decode | Pie 290 · vLLM 225 · TensorRT-LLM 224 · SGLang 216 |
| H200 SXM 141 GB | GPT-OSS-120B | Single prefill | SGLang 13,915 · vLLM 13,488 · Pie 9,874 · TensorRT-LLM 7,125 |
| H200 SXM 141 GB | GPT-OSS-120B | Batched decode | SGLang 12,529 · TensorRT-LLM 11,346 · vLLM 10,981 · Pie 9,688 |
| H200 SXM 141 GB | GPT-OSS-120B | Batched prefill | vLLM 41,652 · SGLang 36,554 · TensorRT-LLM 33,697 · Pie 31,120 |
| H200 SXM 141 GB | Qwen3.6-35B | Single decode | Pie 302 · SGLang 274 · TensorRT-LLM 248 · vLLM 233 |
| H200 SXM 141 GB | Qwen3.6-35B | Single prefill | Pie 21,639 · SGLang 19,159 · vLLM 8,030 · TensorRT-LLM 7,553 |
| H200 SXM 141 GB | Qwen3.6-35B | Batched decode | TensorRT-LLM 13,645 · Pie 12,512 · vLLM 12,114 · SGLang 9,941 |
| H200 SXM 141 GB | Qwen3.6-35B | Batched prefill | SGLang 63,236 · TensorRT-LLM 62,843 · Pie 55,118 · vLLM 55,085 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Single decode | Pie 73.1 · SGLang 65.8 · vLLM 63.6 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Single prefill | SGLang 11,588 · Pie 11,277 · vLLM 11,114 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Batched decode | SGLang 6,810 · vLLM 6,573 · Pie 6,356 |
| H200 SXM 141 GB | Muse-Glimmer-30B | Batched prefill | SGLang 12,971 · Pie 12,922 · vLLM 12,324 |
| H200 SXM 141 GB | Gemma-4-26B | Single decode | Pie 293 · SGLang 234 · vLLM 212 · TensorRT-LLM 165 |
| H200 SXM 141 GB | Gemma-4-26B | Single prefill | Pie 24,493 · vLLM 17,341 · SGLang 11,971 · TensorRT-LLM 8,015 |
| H200 SXM 141 GB | Gemma-4-26B | Batched decode | Pie 15,430 · vLLM 14,330 · SGLang 12,808 · TensorRT-LLM 10,657 |
| H200 SXM 141 GB | Gemma-4-26B | Batched prefill | vLLM 64,793 · TensorRT-LLM 56,762 · SGLang 55,088 · Pie 54,256 |
| H200 SXM 141 GB | Gemma-4-31B | Single decode | Pie 60.6 · SGLang 57.7 · vLLM 54 |
| H200 SXM 141 GB | Gemma-4-31B | Single prefill | vLLM 9,503 · Pie 9,090 · SGLang 8,724 |
| H200 SXM 141 GB | Gemma-4-31B | Batched decode | Pie 3,317 · SGLang 3,272 · vLLM 3,053 |
| H200 SXM 141 GB | Gemma-4-31B | Batched prefill | vLLM 10,398 · Pie 9,902 · SGLang 9,646 |
| H200 SXM 141 GB | Qwen3.8-27B | Single decode | Pie 72.9 · SGLang 70.6 · TensorRT-LLM 67.6 · vLLM 67.3 |
| H200 SXM 141 GB | Qwen3.8-27B | Single prefill | SGLang 10,072 · vLLM 9,919 · Pie 9,222 · TensorRT-LLM 6,502 |
| H200 SXM 141 GB | Qwen3.8-27B | Batched decode | Pie 4,366 · vLLM 4,363 · TensorRT-LLM 4,102 · SGLang 3,905 |
| H200 SXM 141 GB | Qwen3.8-27B | Batched prefill | SGLang 12,287 · vLLM 12,277 · TensorRT-LLM 11,585 · Pie 11,406 |
| H200 SXM 141 GB | Qwen3.5-9B | Single decode | Pie 211 · SGLang 201 · TensorRT-LLM 196 · vLLM 190 |
| H200 SXM 141 GB | Qwen3.5-9B | Single prefill | Pie 26,944 · SGLang 23,042 · vLLM 16,979 · TensorRT-LLM 9,612 |
| H200 SXM 141 GB | Qwen3.5-9B | Batched decode | TensorRT-LLM 15,246 · Pie 14,401 · vLLM 13,785 · SGLang 12,969 |
| H200 SXM 141 GB | Qwen3.5-9B | Batched prefill | vLLM 40,827 · SGLang 40,086 · Pie 38,957 · TensorRT-LLM 37,158 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Single decode | Pie 841 · SGLang 742 · vLLM 655 · TensorRT-LLM 622 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Single prefill | Pie 75,991 · SGLang 28,921 · vLLM 22,931 · TensorRT-LLM 12,971 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Batched decode | Pie 51,807 · SGLang 37,149 · vLLM 34,065 · TensorRT-LLM 11,627 |
| H100 SXM 80 GB | Qwen3.5-0.8B | Batched prefill | Pie 269,828 · vLLM 252,214 · SGLang 219,193 · TensorRT-LLM 143,986 |
| H100 SXM 80 GB | Qwen3-0.6B | Single decode | Pie 802 · SGLang 695 · vLLM 645 · TensorRT-LLM 613 |
| H100 SXM 80 GB | Qwen3-0.6B | Single prefill | Pie 70,306 · SGLang 34,297 · vLLM 27,078 · TensorRT-LLM 18,154 |
| H100 SXM 80 GB | Qwen3-0.6B | Batched decode | SGLang 40,040 · vLLM 37,141 · Pie 34,322 · TensorRT-LLM 11,661 |
| H100 SXM 80 GB | Qwen3-0.6B | Batched prefill | Pie 390,411 · SGLang 375,806 · vLLM 178,333 · TensorRT-LLM 173,238 |
| H100 SXM 80 GB | Gemma-4-E4B | Single decode | Pie 195 · SGLang 166 · vLLM 163 · TensorRT-LLM 146 |
| H100 SXM 80 GB | Gemma-4-E4B | Single prefill | Pie 29,796 · SGLang 12,950 · vLLM 11,085 · TensorRT-LLM 8,045 |
| H100 SXM 80 GB | Gemma-4-E4B | Batched decode | vLLM 19,351 · SGLang 18,527 · Pie 17,349 · TensorRT-LLM 10,616 |
| H100 SXM 80 GB | Gemma-4-E4B | Batched prefill | vLLM 64,417 · Pie 61,472 · SGLang 59,471 · TensorRT-LLM 57,388 |
| H100 SXM 80 GB | Qwen3.6-27B | Single decode | Pie 53.8 · SGLang 52.2 · TensorRT-LLM 51.4 · vLLM 50.6 |
| H100 SXM 80 GB | Qwen3.6-27B | Single prefill | Pie 9,945 · SGLang 9,423 · vLLM 6,686 · TensorRT-LLM 6,486 |
| H100 SXM 80 GB | Qwen3.6-27B | Batched decode | TensorRT-LLM 3,289 · Pie 3,244 · vLLM 2,389 · SGLang 1,598 |
| H100 SXM 80 GB | Qwen3.6-27B | Batched prefill | vLLM 11,837 · SGLang 11,829 · Pie 11,302 · TensorRT-LLM 10,988 |
| H100 SXM 80 GB | GPT-OSS-20B | Single decode | Pie 385 · vLLM 305 · TensorRT-LLM 299 · SGLang 298 |
| H100 SXM 80 GB | GPT-OSS-20B | Single prefill | Pie 22,779 · vLLM 22,379 · SGLang 20,203 · TensorRT-LLM 10,169 |
| H100 SXM 80 GB | GPT-OSS-20B | Batched decode | SGLang 21,360 · vLLM 18,398 · Pie 15,291 · TensorRT-LLM 10,500 |
| H100 SXM 80 GB | GPT-OSS-20B | Batched prefill | vLLM 66,370 · SGLang 63,978 · Pie 59,779 · TensorRT-LLM 58,560 |
| H100 SXM 80 GB | GPT-OSS-120B | Single decode | Pie 263 · vLLM 214 · SGLang 204 · TensorRT-LLM 203 |
| H100 SXM 80 GB | GPT-OSS-120B | Single prefill | SGLang 12,998 · vLLM 11,811 · Pie 10,177 · TensorRT-LLM 6,216 |
| H100 SXM 80 GB | GPT-OSS-120B | Batched decode | TensorRT-LLM 7,349 · vLLM 6,290 · Pie 5,225 · SGLang 3,683 |
| H100 SXM 80 GB | GPT-OSS-120B | Batched prefill | SGLang 37,544 · TensorRT-LLM 34,291 · Pie 31,886 · vLLM 25,417 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Single decode | Pie 51.6 · SGLang 49.4 · vLLM 48.5 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Single prefill | Pie 10,495 · SGLang 10,008 · vLLM 8,002 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Batched decode | SGLang 5,894 · vLLM 5,811 · Pie 5,611 |
| H100 SXM 80 GB | Muse-Glimmer-30B | Batched prefill | SGLang 12,313 · Pie 11,990 · vLLM 11,863 |
| H100 SXM 80 GB | Gemma-4-26B | Single decode | Pie 226 · SGLang 205 · vLLM 190 · TensorRT-LLM 151 |
| H100 SXM 80 GB | Gemma-4-26B | Single prefill | Pie 19,908 · SGLang 11,625 · vLLM 8,991 · TensorRT-LLM 4,848 |
| H100 SXM 80 GB | Gemma-4-26B | Batched decode | vLLM 9,151 · Pie 8,358 · SGLang 7,489 · TensorRT-LLM 5,085 |
| H100 SXM 80 GB | Gemma-4-26B | Batched prefill | vLLM 60,782 · TensorRT-LLM 52,397 · Pie 50,121 · SGLang 47,118 |
| H100 SXM 80 GB | Gemma-4-31B | Single decode | Pie 43.2 · SGLang 42.3 · vLLM 41.5 |
| H100 SXM 80 GB | Gemma-4-31B | Single prefill | Pie 8,619 · SGLang 7,961 · vLLM 6,872 |
| H100 SXM 80 GB | Gemma-4-31B | Batched decode | vLLM 1,317 · SGLang 787 · Pie 695 |
| H100 SXM 80 GB | Gemma-4-31B | Batched prefill | vLLM 10,000 · Pie 9,555 · SGLang 9,461 |
| H100 SXM 80 GB | Qwen3.8-27B | Single decode | Pie 52.6 · SGLang 51.9 · TensorRT-LLM 51.4 · vLLM 48.5 |
| H100 SXM 80 GB | Qwen3.8-27B | Single prefill | Pie 9,556 · SGLang 9,066 · TensorRT-LLM 6,706 · vLLM 6,343 |
| H100 SXM 80 GB | Qwen3.8-27B | Batched decode | TensorRT-LLM 3,286 · Pie 3,011 · vLLM 2,395 · SGLang 1,704 |
| H100 SXM 80 GB | Qwen3.8-27B | Batched prefill | vLLM 12,067 · TensorRT-LLM 11,768 · SGLang 11,371 · Pie 11,150 |
| H100 SXM 80 GB | Qwen3.5-9B | Single decode | Pie 165 · SGLang 157 · TensorRT-LLM 155 · vLLM 150 |
| H100 SXM 80 GB | Qwen3.5-9B | Single prefill | Pie 22,342 · SGLang 19,806 · TensorRT-LLM 11,078 · vLLM 10,052 |
| H100 SXM 80 GB | Qwen3.5-9B | Batched decode | Pie 13,843 · vLLM 12,182 · SGLang 11,934 · TensorRT-LLM 10,885 |
| H100 SXM 80 GB | Qwen3.5-9B | Batched prefill | vLLM 39,385 · SGLang 39,172 · TensorRT-LLM 38,538 · Pie 38,040 |
| L40S 46 GB | Qwen3.5-0.8B | Single decode | Pie 381 · SGLang 380 · TensorRT-LLM 378 · vLLM 357 |
| L40S 46 GB | Qwen3.5-0.8B | Single prefill | Pie 54,756 · SGLang 27,600 · vLLM 17,299 · TensorRT-LLM 10,602 |
| L40S 46 GB | Qwen3.5-0.8B | Batched decode | TensorRT-LLM 11,314 |
| L40S 46 GB | Qwen3.5-0.8B | Batched prefill | Pie 107,529 · SGLang 102,654 · TensorRT-LLM 83,972 · vLLM 50,178 |
| L40S 46 GB | Qwen3-0.6B | Single decode | Pie 418 · SGLang 415 · vLLM 405 · TensorRT-LLM 398 |
| L40S 46 GB | Qwen3-0.6B | Single prefill | Pie 71,065 · SGLang 55,591 · vLLM 34,553 · TensorRT-LLM 15,391 |
| L40S 46 GB | Qwen3-0.6B | Batched decode | Pie 15,940 · vLLM 15,785 · SGLang 15,098 · TensorRT-LLM 12,600 |
| L40S 46 GB | Qwen3-0.6B | Batched prefill | Pie 152,452 · vLLM 152,450 · SGLang 141,947 · TensorRT-LLM 135,406 |
| L40S 46 GB | Gemma-4-E4B | Single decode | Pie 70 · vLLM 69.7 · SGLang 67.9 · TensorRT-LLM 66 |
| L40S 46 GB | Gemma-4-E4B | Single prefill | Pie 15,492 · vLLM 14,639 · SGLang 13,777 · TensorRT-LLM 6,631 |
| L40S 46 GB | Gemma-4-E4B | Batched decode | vLLM 6,379 · Pie 6,337 · SGLang 6,034 · TensorRT-LLM 5,506 |
| L40S 46 GB | Gemma-4-E4B | Batched prefill | vLLM 20,657 · Pie 19,739 · TensorRT-LLM 19,553 · SGLang 18,160 |
| L40S 46 GB | GPT-OSS-20B | Single decode | Pie 162 · vLLM 158 · SGLang 137 |
| L40S 46 GB | GPT-OSS-20B | Single prefill | Pie 18,678 · vLLM 16,734 · SGLang 639 |
| L40S 46 GB | GPT-OSS-20B | Batched prefill | Pie 27,648 · vLLM 23,099 · SGLang 1,066 |
| L40S 46 GB | Qwen3.5-9B | Single decode | Pie 45.3 · TensorRT-LLM 45 · SGLang 44.9 · vLLM 44.5 |
| L40S 46 GB | Qwen3.5-9B | Single prefill | SGLang 9,305 · vLLM 9,091 · Pie 8,933 · TensorRT-LLM 7,641 |
| L40S 46 GB | Qwen3.5-9B | Batched decode | Pie 3,571 · TensorRT-LLM 3,521 · vLLM 2,933 · SGLang 2,620 |
| L40S 46 GB | Qwen3.5-9B | Batched prefill | vLLM 12,587 · SGLang 12,001 · TensorRT-LLM 11,554 · Pie 10,951 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Single decode | SGLang 490 · Pie 470 · vLLM 462 · TensorRT-LLM 459 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Single prefill | Pie 56,365 · SGLang 25,522 · vLLM 15,752 · TensorRT-LLM 10,261 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Batched decode | TensorRT-LLM 13,359 |
| RTX 4090 24 GB | Qwen3.5-0.8B | Batched prefill | Pie 94,322 · TensorRT-LLM 86,157 · SGLang 67,051 · vLLM 27,981 |
| RTX 4090 24 GB | Qwen3-0.6B | Single decode | Pie 482 · TensorRT-LLM 472 · SGLang 464 · vLLM 441 |
| RTX 4090 24 GB | Qwen3-0.6B | Single prefill | Pie 61,808 · SGLang 39,689 · vLLM 24,408 · TensorRT-LLM 13,446 |
| RTX 4090 24 GB | Qwen3-0.6B | Batched decode | vLLM 18,108 · SGLang 17,243 · Pie 16,473 · TensorRT-LLM 15,168 |
| RTX 4090 24 GB | Qwen3-0.6B | Batched prefill | vLLM 125,628 · TensorRT-LLM 121,096 · Pie 114,619 · SGLang 114,218 |
| RTX 4090 24 GB | Gemma-4-E4B | Single decode | Pie 78.8 · vLLM 78.6 · SGLang 76.7 |
| RTX 4090 24 GB | Gemma-4-E4B | Single prefill | Pie 12,922 · vLLM 11,880 · SGLang 9,562 |
| RTX 4090 24 GB | Gemma-4-E4B | Batched decode | vLLM 5,841 · Pie 4,566 · SGLang 3,675 |
| RTX 4090 24 GB | Gemma-4-E4B | Batched prefill | vLLM 17,175 · Pie 16,776 · SGLang 15,330 |
| RTX 4090 24 GB | GPT-OSS-20B | Single decode | Pie 182 · vLLM 176 · SGLang 150 |
| RTX 4090 24 GB | GPT-OSS-20B | Single prefill | Pie 14,393 · vLLM 12,569 · SGLang 806 |
| RTX 4090 24 GB | GPT-OSS-20B | Batched prefill | Pie 20,423 · vLLM 19,483 · SGLang 1,529 |
| RTX 4090 24 GB | Qwen3.5-9B | Single decode | SGLang 57.1 · vLLM 56.6 · Pie 54.9 |
| RTX 4090 24 GB | Qwen3.5-9B | Single prefill | SGLang 7,932 · vLLM 7,291 · Pie 6,641 |
| RTX 4090 24 GB | Qwen3.5-9B | Batched decode | vLLM 810 · SGLang 727 · Pie 465 |
| RTX 4090 24 GB | Qwen3.5-9B | Batched prefill | vLLM 10,124 · SGLang 9,789 · Pie 7,952 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Single decode | Pie 527 · SGLang 517 · TensorRT-LLM 482 · vLLM 457 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Single prefill | Pie 41,156 · SGLang 20,161 · vLLM 14,213 · TensorRT-LLM 12,735 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Batched decode | Pie 28,011 · SGLang 20,756 · TensorRT-LLM 18,708 · vLLM 16,682 |
| A100 SXM 80 GB | Qwen3.5-0.8B | Batched prefill | Pie 121,257 · TensorRT-LLM 110,830 · SGLang 93,624 · vLLM 50,468 |
| A100 SXM 80 GB | Qwen3-0.6B | Single decode | SGLang 518 · Pie 515 · vLLM 482 · TensorRT-LLM 441 |
| A100 SXM 80 GB | Qwen3-0.6B | Single prefill | Pie 60,557 · SGLang 45,587 · vLLM 28,642 · TensorRT-LLM 19,416 |
| A100 SXM 80 GB | Qwen3-0.6B | Batched decode | Pie 26,067 · SGLang 24,483 · vLLM 24,243 · TensorRT-LLM 22,263 |
| A100 SXM 80 GB | Qwen3-0.6B | Batched prefill | Pie 163,016 · TensorRT-LLM 154,723 · SGLang 146,912 · vLLM 133,029 |
| A100 SXM 80 GB | Gemma-4-E4B | Single decode | Pie 128 · vLLM 112 · SGLang 107 · TensorRT-LLM 94.6 |
| A100 SXM 80 GB | Gemma-4-E4B | Single prefill | Pie 17,592 · vLLM 14,396 · SGLang 9,281 · TensorRT-LLM 7,470 |
| A100 SXM 80 GB | Gemma-4-E4B | Batched decode | vLLM 10,539 · Pie 9,494 · SGLang 9,162 · TensorRT-LLM 8,998 |
| A100 SXM 80 GB | Gemma-4-E4B | Batched prefill | vLLM 24,724 · Pie 23,997 · TensorRT-LLM 23,781 · SGLang 14,922 |
| A100 SXM 80 GB | Qwen3.6-27B | Single decode | Pie 31.2 · SGLang 29.4 · TensorRT-LLM 29.2 · vLLM 28.8 |
| A100 SXM 80 GB | Qwen3.6-27B | Single prefill | Pie 3,763 · SGLang 3,601 · TensorRT-LLM 3,469 · vLLM 3,364 |
| A100 SXM 80 GB | Qwen3.6-27B | Batched decode | TensorRT-LLM 1,570 · Pie 1,544 · vLLM 1,157 · SGLang 828 |
| A100 SXM 80 GB | Qwen3.6-27B | Batched prefill | SGLang 4,470 · TensorRT-LLM 4,415 · Pie 4,364 · vLLM 4,256 |
| A100 SXM 80 GB | GPT-OSS-20B | Single decode | Pie 257 · vLLM 238 · SGLang 217 |
| A100 SXM 80 GB | GPT-OSS-20B | Single prefill | Pie 15,977 · vLLM 13,818 · SGLang 11,657 |
| A100 SXM 80 GB | GPT-OSS-20B | Batched decode | Pie 9,277 · vLLM 8,807 · SGLang 7,450 |
| A100 SXM 80 GB | GPT-OSS-20B | Batched prefill | Pie 24,086 · vLLM 20,222 · SGLang 16,265 |
| A100 SXM 80 GB | GPT-OSS-120B | Single decode | Pie 182 · vLLM 166 · SGLang 150 |
| A100 SXM 80 GB | GPT-OSS-120B | Single prefill | Pie 6,972 · vLLM 6,930 · SGLang 6,104 |
| A100 SXM 80 GB | GPT-OSS-120B | Batched decode | vLLM 2,598 · Pie 2,177 · SGLang 1,117 |
| A100 SXM 80 GB | GPT-OSS-120B | Batched prefill | Pie 12,262 · vLLM 10,017 · SGLang 9,320 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Single decode | Pie 31.1 · SGLang 28.7 · vLLM 28.7 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Single prefill | Pie 3,538 · SGLang 3,470 · vLLM 3,299 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Batched decode | Pie 2,490 · SGLang 2,354 · vLLM 2,345 |
| A100 SXM 80 GB | Muse-Glimmer-30B | Batched prefill | SGLang 4,500 · vLLM 4,439 · Pie 4,287 |
| A100 SXM 80 GB | Gemma-4-26B | Single decode | Pie 148 · SGLang 131 · vLLM 130 · TensorRT-LLM 104 |
| A100 SXM 80 GB | Gemma-4-26B | Single prefill | Pie 13,410 · vLLM 9,836 · SGLang 7,577 · TensorRT-LLM 6,782 |
| A100 SXM 80 GB | Gemma-4-26B | Batched decode | vLLM 4,701 · Pie 4,492 · SGLang 3,791 · TensorRT-LLM 3,239 |
| A100 SXM 80 GB | Gemma-4-26B | Batched prefill | TensorRT-LLM 22,818 · Pie 21,647 · vLLM 20,449 · SGLang 11,783 |
| A100 SXM 80 GB | Gemma-4-31B | Single decode | Pie 25.8 · SGLang 24.4 · vLLM 23.9 |
| A100 SXM 80 GB | Gemma-4-31B | Single prefill | Pie 3,189 · vLLM 3,169 · SGLang 2,096 |
| A100 SXM 80 GB | Gemma-4-31B | Batched decode | vLLM 638 · Pie 466 · SGLang 407 |
| A100 SXM 80 GB | Gemma-4-31B | Batched prefill | Pie 3,620 · vLLM 3,585 · SGLang 2,363 |
| A100 SXM 80 GB | Qwen3.8-27B | Single decode | Pie 31.3 · SGLang 29.5 · TensorRT-LLM 29.2 · vLLM 29 |
| A100 SXM 80 GB | Qwen3.8-27B | Single prefill | Pie 3,606 · SGLang 3,510 · vLLM 3,485 · TensorRT-LLM 3,436 |
| A100 SXM 80 GB | Qwen3.8-27B | Batched decode | TensorRT-LLM 1,573 · Pie 1,542 · vLLM 1,156 · SGLang 825 |
| A100 SXM 80 GB | Qwen3.8-27B | Batched prefill | TensorRT-LLM 4,408 · SGLang 4,118 · Pie 4,083 · vLLM 3,977 |
| A100 SXM 80 GB | Qwen3.5-9B | Single decode | Pie 95 · SGLang 89.9 · TensorRT-LLM 88.9 · vLLM 88.9 |
| A100 SXM 80 GB | Qwen3.5-9B | Single prefill | Pie 11,199 · SGLang 10,823 · vLLM 10,094 · TensorRT-LLM 9,285 |
| A100 SXM 80 GB | Qwen3.5-9B | Batched decode | TensorRT-LLM 5,937 · Pie 5,544 · vLLM 5,219 · SGLang 4,944 |
| A100 SXM 80 GB | Qwen3.5-9B | Batched prefill | SGLang 15,182 · vLLM 14,682 · TensorRT-LLM 14,515 · Pie 14,345 |
| A40 48 GB | Qwen3.5-0.8B | Single decode | SGLang 292 · Pie 285 · TensorRT-LLM 282 · vLLM 280 |
| A40 48 GB | Qwen3.5-0.8B | Single prefill | Pie 24,869 · SGLang 19,181 · vLLM 10,666 · TensorRT-LLM 9,341 |
| A40 48 GB | Qwen3.5-0.8B | Batched decode | Pie 11,295 · vLLM 8,557 · SGLang 8,277 · TensorRT-LLM 7,934 |
| A40 48 GB | Qwen3.5-0.8B | Batched prefill | Pie 57,102 · TensorRT-LLM 52,794 · SGLang 50,291 · vLLM 48,465 |
| A40 48 GB | Qwen3-0.6B | Single decode | Pie 340 · SGLang 325 · vLLM 316 · TensorRT-LLM 291 |
| A40 48 GB | Qwen3-0.6B | Single prefill | Pie 33,643 · SGLang 26,093 · vLLM 18,365 · TensorRT-LLM 13,701 |
| A40 48 GB | Qwen3-0.6B | Batched decode | Pie 11,383 · vLLM 10,604 · SGLang 10,350 · TensorRT-LLM 8,483 |
| A40 48 GB | Qwen3-0.6B | Batched prefill | Pie 77,990 · TensorRT-LLM 73,093 · vLLM 69,917 · SGLang 61,985 |
| A40 48 GB | Gemma-4-E4B | Single decode | Pie 57.9 · vLLM 51.5 · SGLang 50.2 · TensorRT-LLM 48.2 |
| A40 48 GB | Gemma-4-E4B | Single prefill | Pie 8,066 · SGLang 6,717 · vLLM 6,536 · TensorRT-LLM 6,290 |
| A40 48 GB | Gemma-4-E4B | Batched decode | Pie 4,462 · vLLM 4,367 · SGLang 4,039 · TensorRT-LLM 3,941 |
| A40 48 GB | Gemma-4-E4B | Batched prefill | vLLM 11,584 · Pie 11,180 · TensorRT-LLM 10,631 · SGLang 8,954 |
| A40 48 GB | GPT-OSS-20B | Single decode | Pie 129 · vLLM 123 · SGLang 120 |
| A40 48 GB | GPT-OSS-20B | Single prefill | Pie 7,941 · vLLM 7,625 · SGLang 6,390 |
| A40 48 GB | GPT-OSS-20B | Batched decode | vLLM 4,477 · Pie 3,761 · SGLang 3,330 |
| A40 48 GB | GPT-OSS-20B | Batched prefill | Pie 11,893 · vLLM 11,573 · SGLang 7,937 |
| A40 48 GB | Qwen3.5-9B | Single decode | Pie 36.5 · SGLang 33.2 · TensorRT-LLM 33.1 · vLLM 32.9 |
| A40 48 GB | Qwen3.5-9B | Single prefill | Pie 5,596 · SGLang 5,152 · TensorRT-LLM 4,928 · vLLM 4,334 |
| A40 48 GB | Qwen3.5-9B | Batched decode | Pie 2,522 · TensorRT-LLM 2,338 · vLLM 1,973 · SGLang 1,827 |
| A40 48 GB | Qwen3.5-9B | Batched prefill | SGLang 6,534 · Pie 6,492 · vLLM 6,486 · TensorRT-LLM 6,315 |