KV cache
Content-addressed pages, prefix sharing, eviction algorithms, and memory tiers.
Pie stores model state (both attention KV caches and recurrent state) in fixed-size pages. Each frozen page is keyed by a hash of its content: the model, the dtype, and the exact token sequence that produced it. Prefix sharing, zero-copy forking, and tiered offloading all follow from that design.
Automatic prefix sharing
As the figure above shows, two sequences that start with the same tokens produce pages with the same keys. The second sequence maps the existing pages instead of recomputing them. This happens automatically on every prefill; ws.cached_tokens() returns how many tokens a working set reused from cache.
When multiple requests prefill the same uncached prefix at the same time, the runtime computes the prefix once and shares the resulting pages across all waiting requests. In a cluster, the cache-aware router sends each request to the worker that already holds its prefix.
Forks
ws.fork(pipe) creates a child working set that shares the parent's frozen pages and copies only its writable buffer. Forking lets beam search, prefix trees, and consensus decoding branch from a shared prompt without duplicating memory or prefill compute.
Eviction as an inferlet
Because each ForwardPass specifies which KV slots it writes (StateGeometry) and attends to (fwd.mask), an inferlet can implement KV cache eviction directly in user code:
| Method | Idea | Example |
|---|---|---|
| Attention sink | Keep the first few sink tokens plus a sliding window of recent tokens. | attention-sink |
| SnapKV | Score prompt positions by attention from an observation window and drop low-scoring slots. | snapkv-eviction |
| TOVA | Drop the KV slot that receives the lowest attention weight at each step. | tova-attention |
| H2O | Keep recent tokens plus heavy-hitter tokens with the highest cumulative attention. | trackb-h2o |
Memory tiers
When GPU memory runs short, the runtime moves idle KV pages to host RAM and then to disk instead of dropping them. When an idle conversation resumes on a later turn, the runtime brings its pages back into GPU memory rather than recomputing the prefill. Set the tier budgets under [model]; see Configure & tune.
In a cluster, pages can also live on shared pie-pool nodes in host RAM, on disk, or in an object store. See Scale out.
Mechanism vs. policy
Inferlets decide which KV slots a sequence reads and writes. Scheduling policies decide which pages stay cached and where: for example, keep-session retains a finished process's pages for its next turn, and persist-prefix keeps hot prefixes on disk across restarts.
Use worker and cluster policies to control page retention, replication, and migration.