Sessions

An inferlet does not have to exit after answering one prompt. It can wait for the next user message, answer again, and keep its WorkingSet in memory between turns. As the figure below shows, each turn then computes only its new tokens instead of re-reading the conversation history.

A multi-turn chat inferletCopy Anchor Link

await session.receive() waits for the next message from the client, and session.send() sends a message back. Because ws stays alive across the while loop, the model retains the KV cache from earlier turns:

from inferlet import chat, model, session
from inferlet.eta import WorkingSet, Pipeline
# generate() as in "Your first inferlet"


async def main(input: dict) -> dict:
    ws, pipe = WorkingSet(), Pipeline()
    pending = chat.system("You are a concise assistant.")
    turns = 0
    while True:
        message = await session.receive()
        if message == "/bye":
            break
        pending += chat.user(message) + chat.cue_without_thinking()
        reply = []
        async for token in generate(ws, pipe, pending):
            reply.append(token)
        session.send(model.decode(reply))
        pending = chat.seal()     # close this turn; the next turn starts from here
        turns += 1
    pipe.close()
    return {"turns": turns}

pending holds tokens that have not been fed into ws yet. On the first turn it holds the system prompt and first user message; on later turns it holds the closing delimiter (chat.seal()) from the previous turn plus the new user message.

Connecting from a clientCopy Anchor Link

Sessions run over Pie's WebSocket endpoint at /v1/ws. Using the Python pie-client package:

from pie_client import PieClient, Event

async with PieClient("ws://127.0.0.1:8080") as client:
    await client.authenticate("me")
    name = await client.install_program("chat.py")
    process = await client.launch_process(name)

    for question in ["What is the capital of France?", "And of Italy?"]:
        await process.signal(question)
        event, reply = await process.recv()
        while event != Event.Message:              # skip stdout and log events
            event, reply = await process.recv()
        print(reply)

    await process.signal("/bye")
    print(await process.result())

The capital of France is **Paris**.
The capital of Italy is **Rome**.
{'turns': 2}

The second question ("And of Italy?") works because ws still holds the KV state from the first turn.

StreamingCopy Anchor Link

You can call session.send() as often as you like during a turn, such as once per token in Your first inferlet. Each call arrives at the client as an Event.Message.

Files and mediaCopy Anchor Link

session.send_file(data) and await session.receive_file() transfer binary data such as images and audio. Files attached at launch (pie run --file name=path) can be read inside the inferlet with session.file(name).

Idle sessions and KV offloadingCopy Anchor Link

While a session waits on session.receive(), the runtime can move its KV cache pages from GPU memory to host RAM or disk to free space for active requests, and page them back in when the next message arrives. Two built-in scheduling policies also help when sessions reconnect:

• keep-session keeps a finished process's KV pages cached for a window of time so a new process can reuse them.
• session-affinity routes a returning session to the same worker in a cluster.