What is Pie
An inference engine you can program, from a phone to a TPU pod.
Pie is an LLM and multimodal inference engine. You can start pie serve, point an OpenAI, Anthropic, or Gemini client at it, and use it as a standard server.
What makes Pie different is how it runs requests. In Pie, every request is a small program called an inferlet. The inferlet drives the model one forward pass at a time, and you can read it, change it, or write your own in Python, JavaScript, or Rust.
Because the generation loop is a user program rather than fixed engine code, the same system serves drop-in APIs, runs multi-step agents, and executes custom decoding algorithms.
Three layers of control
The figure above shows Pie's stack from top to bottom. Start at the top layer and drop down only when you need more control.
Serve
Run Pie as a server: drop-in APIs, models, hardware backends, and clusters.
Program
Write inferlets for agents, structured output, branching, and multi-turn sessions.
Invent
Build custom decoding algorithms, KV cache policies, and model architectures.
These layers are not separate systems. The OpenAI, Anthropic, and Gemini APIs in Serve are built-in inferlets that ship with Pie. The helpers in Program are built from the low-level primitives in Invent. You can open up any layer and change how it works without modifying the engine.
Key design ideas
Every algorithm is an inferlet. Speculative decoding, beam search, constrained decoding, watermarking, and agent loops are user programs, not engine flags. Anything Pie ships, you can copy and modify. See How Pie thinks.
Every model is a program. Model architectures across text, vision, image, and video are short files written in Poem, Pie's model definition language. You can edit a model's forward pass or add a new architecture without recompiling Pie.
One 60 MB binary across hardware. Pie is written in Rust with no Python or PyTorch runtime. It runs on NVIDIA and AMD GPUs, Apple silicon, Vulkan devices (including Android phones), web browsers via WebGPU, and Google TPUs. See Hardware.
Programmability without losing batching. Each inferlet writes its own loop, and the runtime batches its forward passes with passes from all other running inferlets. The compiler fuses custom sampling code into device kernels, so programmability does not add a host round trip per token. See the benchmarks.