Program
Write inferlets for custom decoding, tool calls, structured output, sessions, and agents.
When a stateless chat endpoint is not enough (for example, when you need multi-step agents, constrained output, branching search, or stateful sessions), you write an inferlet.
As the figure below shows, an inferlet is a short program that runs inside the engine next to the model, while the runtime batches its forward passes with everyone else's.
You do not need to write GPU kernels or manage batching. Every page in this section starts from the same generate loop and adds one concept at a time.
Your first inferlet
Stream a chat reply in about 30 lines of Python.
Chat, tools & reasoning
Format chat turns, tool calls, and thinking blocks across models.
Structured output
Enforce JSON Schema, regex, and EBNF grammars.
Sessions
Keep KV cache state alive across multi-turn conversations.
Branching
Fork one working set into parallel continuations.
Agents
Run tool loops inside the engine without network round trips.
Ship it
Install inferlets on a server and call them from clients.
Inferlets can be written in Python, JavaScript, or Rust. These pages use Python; the Cookbook has versions of the main examples in all three languages.
Go deeperWhat a forward pass can do →Lanes, attention masks, adapters, and device epilogues: the primitives under every helper.