Scale out
Scale from one GPU to a cluster with prefill/decode disaggregation and cache-aware routing.
On one machine, pie serve runs three components in a single process: a gateway that accepts client connections, a controller that tracks workers and cached prefixes, and a worker that runs the model. To scale across machines, run those components as separate processes.
Cluster binaries
| Binary | Role |
|---|---|
pie-gateway | Accepts client connections and routes each session to a worker. |
pie-controller | Maintains the worker registry, routing table, and directory of cached prefixes. |
pie-worker | Loads the model on its GPUs/TPUs and runs inferlets. |
pie-pool | Stores KV cache pages in host RAM, on disk, or in an object store for workers to share. It loads no model. |
A worker joins a cluster through its [cluster] config section:
[cluster]
controller = "tcp://10.0.0.1:9102"
role = "decode" # or "prefill"
gateways = ["tcp://10.0.0.2:8081"]Prefill and decode disaggregation
Processing a long prompt (prefill) is compute-bound, while generating tokens (decode) is memory-bandwidth-bound. If you assign some workers role = "prefill", decode workers send their prompt prefill work to those workers. The resulting KV cache pages return over the network, using NIXL (RDMA) when available.
Cluster routing policies
Routing is controlled by policies listed under [cluster] cluster_policies. Built-in policies include:
cache-aware: Sends a request to the worker that already holds most of its prompt prefix, unless that worker is overloaded.session-affinity: Keeps a multi-turn session on the same worker.disaggregate: Sends prefill work toprefillworkers.replicate,persist-prefix,drain-migrate,rebalance: Copy hot prefixes across workers, persist them to disk, and move sessions off workers that are shutting down.
Sharing KV cache across machines
Under [offload], a worker can send prefill work to peer workers and store evicted KV pages on pie-pool nodes. When a prompt prefix has already been computed anywhere in the cluster, workers fetch its pages over the network instead of recomputing them:
[offload]
listen = "10.0.0.1:7100"
peers = ["10.0.0.2:7100"]
pools = ["10.0.0.3:7201"]Combine routing, KV replication, and live migration policies for your cluster workload.