Configure & tune
Configure Pie with one TOML file and tune kernel and runtime settings for your hardware.
Pie reads a single TOML configuration file. In most setups, you only set the model and device, and let pie tune choose the kernel and batching parameters.
The config file
pie config init # write a commented default config.toml
pie config show # print the active config
pie config list # list every key, its current value, and what it does
pie config set server.listen 0.0.0.0:8080
pie config edit # open the config file in $EDITORPie looks for the config file in this order: the --config CLI flag, $PIE_CONFIG, and $PIE_HOME/config.toml. Every key has a default value, so deleting a line restores its default.
Byte sizes and durations include their units, such as "4GiB" or "50ms".
Sections at a glance
| Section | Purpose |
|---|---|
[server] | Bind address, worker threads, and upload limits. |
[model] | Model name and memory budgets for device, host RAM, and disk KV cache tiers. |
[engine] | Backend type, device list, activation dtype, and multi-GPU sharding. |
[runtime] | Batching deadlines, concurrency limits, and the worker scheduling policy. |
[sandbox] | Resource limits and permissions for inferlets: filesystem, network, and memory. |
[cluster] | Distributed serving: controller address, worker role, and gateways. |
[offload] | Peer prefill offloading and remote KV cache pools (pie-pool). |
[telemetry] | OpenTelemetry (OTLP) collector endpoint. |
The Config reference lists every key and default.
KV cache memory tiers
By default, the engine reserves 90% of free GPU memory (device_memory_share = 0.90). Any reserved memory not used by the model weights becomes GPU KV cache.
When GPU KV cache fills up, the runtime moves idle pages to host RAM and then to disk instead of dropping them. When an idle conversation resumes, the runtime pages its state back into GPU memory rather than recomputing the prefill.
[model]
device_memory_share = 0.90
host_kv_budget = "32GiB"
disk_kv_budget = "200GiB" # set to "0B" to disable the disk tierTune for your device
pie tunepie tune benchmarks kernel variants on your device, fits a profile of the fastest settings for each tensor shape, and verifies the profile end to end on your smallest imported model. It saves the new profile only if end-to-end speed improves. The default run takes a few minutes; --full sweeps the entire grid in about an hour.
pie config tune tunes runtime batching settings for latency or throughput on a workload you specify:
pie config tune --for throughput --program text-completion --writeWithout --write, it prints the best settings it found without editing config.toml.
Sandbox
Inferlets run in a WebAssembly sandbox. By default, they can access the network (for tool calls) but cannot read or write local files:
[sandbox]
allow_fs = false
allow_network = true
network_allowed_hosts = ["*"]
max_memory = "4GiB"Restrict network_allowed_hosts when running untrusted inferlets.
Logs and metrics
Pass --log-level debug to increase log verbosity. Pass --metrics-listen 127.0.0.1:9100 to serve Prometheus metrics, and set [telemetry] in config.toml to export traces to an OTLP collector.
Set worker_policy to switch between fair-share, priority, deadline, and session-retention schedulers.