The future ofinference isprogrammable

Pie is a blazingly fast inference engine that empowers your application to achieve more.

Get startedTry it in your browser

Applications

  • Coding agent
  • Deep research
  • Voice assistant
  • Batch evals

Inferlets

Control plane

  • Earliest deadline firstdeadline.wasm
  • Cache-aware routingcache-aware.wasm
  • Fair share (VTC)fair.wasm
  • Prefill/decode disaggregationdisaggregate.wasm
  • Priority schedulingpriority.wasm
  • Session affinitysession-affinity.wasm
  • Session KV retentionkeep-session.wasm
  • Load rebalancingrebalance.wasm
  • Prefill offloadoffload.wasm
  • Persistent prefix cachepersist-prefix.wasm
  • KV replicationreplicate.wasm
  • Autoscalingscale-by-prop.wasm
  • Drain and migratedrain-migrate.wasm
  • Pinned placementpin.wasm

Data plane

  • Qwen 3qwen_3.poem
  • Qwen 4qwen_4.poem
  • Gemma 4gemma_4.poem
  • DeepSeek V4deepseek_v4.poem
  • GLM 5glm_5.poem
  • Kimi K3kimi_k3.poem
  • gpt-ossgpt_oss.poem
  • MiniMax H3minimax_h3.poem
  • Diffusion Gemmagemma_4.poem
  • FLUX.2flux_2.poem
  • Z-Imagez_image.poem
  • HunyuanImage 3hunyuan_image_3.poem
  • Wan 2wan_2.poem
  • LTX-2ltx_2.poem

Hardware

  • NVIDIA GPUs
  • AMD GPUs
  • Google TPUs
  • Apple silicon
  • Vulkan
  • Android
  • Web browsers

Latest from the blog

All posts

Release

Pie 0.4.0: Eliminating the programmability tax, and becoming multimodal

Speculative execution hides the runtime's per-token overhead behind GPU compute, and a model-agnostic media API makes Pie multimodal.

Read the post

High performance

  • 5–55% faster than vLLM and SGLang on datacenter GPUs, and ahead of vllm-tpu on Google TPUs.
  • 30–200% faster than llama.cpp and MLX on Apple silicon and consumer GPUs.
  • No programmability tax: Inferlet's novel programming model hides runtime overhead behind GPU compute, while custom samplers compile to native device kernels.
See the full benchmarks

How much faster Pie is than each engine

NVIDIA H100
vLLM1.04×
SGLang1.05×
Apple M4 Pro
mlx-lm1.49×
llama.cpp2.75×
Google TPU v6e
vllm-tpu1.36×

Each bar is the geometric mean over the models measured on that platform. The bright part of a bar is Pie's lead beyond parity.

Lightweight

  • 60 MB single binary—over 153× smaller than a vLLM or SGLang container, with no Python or PyTorch runtime.
  • Written from scratch in Rust with zero CUDA container layers or pip dependency chains to untangle.
  • Instant cold starts whether deployed on a cluster, run locally on a laptop, or embedded inside an application.
Install the binary

What you download to serve a model, area to scale

SGLang15.3 GB · docker image
vLLM9.2 GB · docker image
Pie60 MB · single binary153× smaller

Programmable

  • You own the loop: every request is an inferlet in Python, JavaScript, or Rust that drives the model one forward pass at a time.
  • First-class KV state: fork prefixes across parallel branches, rewind draft tokens, or keep multi-turn sessions resident in memory.
  • Batched automatically: custom decoding loops and device epilogues share GPU launches with every other request on the server.
Read the programming guide

29 inferlets ship in the repository

top_a.py
from inferlet.eta import (
    softmax, reduce_max, broadcast,
    ge, select, gumbel_max,
)

def top_a(logits, vocab, a, rng):
    # Keep tokens with p >= a * p_max^2, then sample.
    probs = softmax(logits)
    p_max = reduce_max(probs)
    cut = broadcast(a * p_max * p_max, [vocab])
    keep = ge(probs, cut)
    floor = broadcast(float("-inf"), [vocab])
    return gumbel_max(select(keep, logits, floor), rng)
Write a short paragraph about Paris.
  1. Paris0.46
  2. The0.21
  3. Nestled0.09
  4. Known0.06
  5. As0.04
  6. With0.03
  7. In0.02
p_max 0.46cut 0.042kept 4 of 7a = 0.2 · probabilities illustrative

Cross-platform

  • One binary across the fleet: native backends for NVIDIA CUDA, AMD ROCm, Google Cloud TPU, Apple Metal, and Vulkan.
  • Down to phones and browsers: runs natively on Android and compiles to WebAssembly and WebGPU to run inside a browser tab.
  • Write once, run anywhere: the same inferlets and model definitions execute unchanged on every target.
See supported platforms
  • NVIDIA GPUs
  • AMD GPUs
  • Google TPUs
  • Apple silicon
  • Vulkan
  • Android
  • Web browsers

Extensible

  • Models are programs: every architecture is a short file in Poem, Pie’s model language—add or edit a model without rebuilding the engine.
  • Every modality in one runtime: serve autoregressive text, diffusion language, vision, audio, image, and video models side by side.
  • Customize the forward pass: edit layers, attach per-request LoRA adapters, or expose custom intermediate outputs to your inferlets.
See how models are defined

Some model definitions shipped by default.

Language
  • Qwen 3
  • Qwen 4
  • Gemma 4
  • DeepSeek V4
  • GLM 5
  • Kimi K3
  • gpt-oss
  • MiniMax H3
Diffusion language
  • Diffusion Gemma
Multimodal
  • Qwen 3
  • Gemma 4
  • GLM 5
Image generation
  • FLUX.2
  • Z-Image
  • HunyuanImage 3
Video generation
  • Wan 2
  • LTX-2

Build on Pie.Open source under Apache 2.0.

For users

Serve with the clients you already use.

Start pie serve and point your OpenAI, Anthropic, or Gemini SDK at it unchanged. Import models from Hugging Face, quantize weights and KV cache at runtime, and scale across GPUs with cache-aware routing.

For developers

Move your agent loop inside the engine.

When a chat endpoint isn’t enough, write an inferlet in Python, JavaScript, or Rust. Tool calls, structured output, and parallel rollouts run next to the model and fork KV state in place—with no network round trips between turns.

For researchers

Turn a paper into an inferlet in an afternoon.

Every built-in feature in Pie is built from user-space primitives. Ship speculative decoders, KV eviction policies, device-compiled samplers, or new model architectures in Poem without touching engine code.

Stay in sync

New releases, benchmarks, and posts from the Pie team, in your inbox.