Agents
Run tool loops, outbound HTTP requests, and multi-agent pipelines inside the engine.
In a typical setup, an agent loop runs in an external application: on every step it sends the full conversation to the model server, parses a tool call, runs the tool, and sends the growing transcript back again.
In Pie, the agent loop runs inside the inferlet next to the model. As the figure below shows, the WorkingSet stays in memory across steps, so the prompt and earlier turns are never sent or prefilled twice.
A calculator agent
This inferlet answers a question and calls a calculator tool whenever it needs arithmetic. Even a 0.8B model computes 1234 * 5678 + 91 accurately this way:
import json
from inferlet import chat, model, tools
from inferlet.eta import WorkingSet, Pipeline
# generate() as in "Your first inferlet"
CALCULATOR = json.dumps({
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate an arithmetic expression.",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string"}},
"required": ["expression"],
},
},
})
def calculate(expression: str) -> str:
return str(eval(expression, {"__builtins__": {}}))
async def main(input: dict) -> dict:
ws, pipe = WorkingSet(), Pipeline()
pending = (chat.system("Always call the calculate tool for arithmetic.")
+ tools.equip([CALCULATOR])
+ chat.user(input["question"])
+ chat.cue_without_thinking())
steps = []
for _ in range(4): # at most four tool calls
decoder, call, reply = tools.Decoder(), None, []
async for token in generate(ws, pipe, pending):
reply.append(token)
event = decoder.feed([token])
if isinstance(event, tools.Event.Call):
call = event.call
if call is None: # no tool call: return the final answer
pipe.close()
return {"answer": model.decode(reply), "steps": steps}
args = json.loads(call.arguments_json)
result = calculate(args["expression"])
steps.append({"call": call.name, "args": args, "result": result})
pending = chat.seal() + tools.answer(call.name, result) + chat.cue_without_thinking()
pipe.close()
return {"answer": None, "steps": steps}pie run agent.py -- --question "What is 1234 * 5678 + 91?"{"answer": "The result of the calculation is 7006743.",
"steps": [{"call": "calculate", "args": {"expression": "1234 * 5678 + 91"}, "result": "7006743"}]}Each iteration appends only the tool result (tools.answer(...)) to ws. The system prompt, tool schemas, and question are prefilled once at the start.
Calling external tools over HTTP
Inferlets can make outbound HTTP requests through wasi:http, so a tool can call a web API, search service, or MCP server directly. The [sandbox] section in config.toml controls which hosts are reachable. Local file access is disabled by default.
Multi-agent messaging
Multiple inferlet processes launched by the same client can message each other directly inside the engine without routing through the client. In the Rust SDK, session::send_to(process, message) delivers a message to another process's session.receive(). A planner inferlet can delegate work to specialist agents this way while each agent keeps its own warm WorkingSet.
To try multiple plans in parallel and keep the best result, fork the working set before each attempt.
Keeping agent loops fast
- Automatic prefix sharing. Agents that share a system prompt and tool schemas share those KV cache pages automatically.
- Scheduler hints. A process can set properties such as
prioritythat scheduling policies read. - Session retention. With the
keep-sessionpolicy enabled, a finished agent's KV pages stay cached for follow-up runs.
Configure worker and cluster scheduling policies to prioritize interactive agents and retain session state.