Invent

Low-level primitives: forward passes, device epilogues, KV cache control, scheduling policies, and Poem models.

You can implement new inference algorithms and model architectures in Pie without modifying the engine. Built-in features such as the OpenAI API, speculative decoding, and cache-aware routing are written using the same primitives described in this section.

inferletsthe loopepiloguessampling on deviceruntimebatches passes · keeps cache pages · moves statepolicieswho runs, wherePoem modelthe forward pass, as a programeta compilertraces → device codeCUDAROCmMetalVulkanWebGPUTPU
Dashed outlines are programs you can write: inferlets, epilogues, policies and models.

As the figure above shows, four programmable pieces meet here:

  • Inferlets control the generation loop: what each ForwardPass reads and writes, and how its outputs are used.
  • Epilogues run on the GPU right after each forward pass, compiled from Python, JavaScript, or Rust.
  • Policies configure which process runs next, which KV pages stay cached, and how requests route across a cluster.
  • Models are Poem files that define the forward pass itself.

The forward pass

Lanes, state geometry, attention masks, readouts, and adapters.

Device epilogues

Samplers compiled to GPU kernels, and decode loops that stay on the device.

Speculative decoding

Draft, verify, and commit tokens as an inferlet.

KV cache

Shared pages, eviction algorithms, and memory tiers.

Scheduling policies

Configure how work is ordered, cached, and routed.

Write a model in Poem

Define or edit a model architecture in a short file.

Backends & kernels

How one codebase targets CUDA, ROCm, Metal, Vulkan, WebGPU, and TPUs.

The Cookbook contains more than fifty complete inferlets covering samplers, guidance, tree search, speculative decoding, KV eviction, and watermarks.