Scheduling policies
Configure how Pie orders requests, retains KV cache pages, and routes sessions across a cluster.
Pie separates scheduling and routing decisions into modular policies configured in config.toml. Policies decide which inferlet process runs next on a worker, which KV cache pages stay resident, and which worker handles an incoming session in a cluster.
Two policy planes
As the figure above shows, policies operate at two levels.
Worker policies run inside a single worker and order its local processes. Set one with [runtime] worker_policy:
| Policy | Behavior |
|---|---|
fair | Fair-share scheduling across clients using a virtual token counter (VTC). |
priority | Runs processes with a higher priority property first. |
deadline | Earliest-deadline-first (EDF) scheduling. |
keep-session | Retains a finished process's KV pages for a follow-up turn. |
offload | Sends prefill work to peer workers when local compute is busy. |
Cluster policies run in pie-controller across all workers. List them in [cluster] cluster_policies:
| Policy | Behavior |
|---|---|
cache-aware | Routes a request to the worker that already holds its prompt prefix, unless that worker is overloaded. |
session-affinity | Keeps a multi-turn session on the same worker. |
disaggregate | Sends prefill work to dedicated prefill workers. |
replicate, pool-replicate | Copies hot prefixes to additional workers or to a pie-pool. |
persist-prefix | Keeps hot prefixes on persistent disk across restarts. |
drain-migrate, rebalance, relocate | Moves active sessions and cache pages between workers. |
cluster-priority, scale-by-prop, pin, retain-all | Cluster-wide priority ordering, autoscaling hints, worker pinning, and retention. |
Sending SIGHUP reloads both worker and cluster policies without restarting the server.
Request hints (props)
Clients and inferlets can attach small typed key-value attributes called props to a process. Policies read those props when making decisions: for example, priority reads the integer priority prop, and keep-session reads keep-ms.