# Scale out

> Scale from one GPU to a cluster with prefill/decode disaggregation and cache-aware routing.

Source: https://pie-project.org/docs/serve/scale-out  

On one machine, `pie serve` runs three components in a single process: a **gateway** that accepts client connections, a **controller** that tracks workers and cached prefixes, and a **worker** that runs the model. To scale across machines, run those components as separate processes.

> **Figure.** A Pie cluster: clients reach gateways, a controller keeps the routing table and cache directory, prefill and decode workers run the model, and pages move between workers. pie serve is all of these in one process. A cluster runs them as separate processes.

## Cluster binaries

| Binary           | Role                                                                                                       |
| ---------------- | ---------------------------------------------------------------------------------------------------------- |
| `pie-gateway`    | Accepts client connections and routes each session to a worker.                                            |
| `pie-controller` | Maintains the worker registry, routing table, and directory of cached prefixes.                            |
| `pie-worker`     | Loads the model on its GPUs/TPUs and runs inferlets.                                                       |
| `pie-pool`       | Stores KV cache pages in host RAM, on disk, or in an object store for workers to share. It loads no model. |

A worker joins a cluster through its `[cluster]` config section:

```toml
[cluster]
controller = "tcp://10.0.0.1:9102"
role = "decode"                      # or "prefill"
gateways = ["tcp://10.0.0.2:8081"]
```

## Prefill and decode disaggregation

Processing a long prompt (prefill) is compute-bound, while generating tokens (decode) is memory-bandwidth-bound. If you assign some workers `role = "prefill"`, decode workers send their prompt prefill work to those workers. The resulting KV cache pages return over the network, using NIXL (RDMA) when available.

## Cluster routing policies

Routing is controlled by policies listed under `[cluster] cluster_policies`. Built-in policies include:

* **`cache-aware`**: Sends a request to the worker that already holds most of its prompt prefix, unless that worker is overloaded.
* **`session-affinity`**: Keeps a multi-turn session on the same worker.
* **`disaggregate`**: Sends prefill work to `prefill` workers.
* **`replicate`**, **`persist-prefix`**, **`drain-migrate`**, **`rebalance`**: Copy hot prefixes across workers, persist them to disk, and move sessions off workers that are shutting down.

## Sharing KV cache across machines

Under `[offload]`, a worker can send prefill work to peer workers and store evicted KV pages on `pie-pool` nodes. When a prompt prefix has already been computed anywhere in the cluster, workers fetch its pages over the network instead of recomputing them:

```toml
[offload]
listen = "10.0.0.1:7100"
peers  = ["10.0.0.2:7100"]
pools  = ["10.0.0.3:7201"]
```

> **Go deeper:** [Configure cluster routing policies](https://pie-project.org/docs/invent/scheduling). Combine routing, KV replication, and live migration policies for your cluster workload.
