Your first inferlet

An inferlet is an async def main function that takes a JSON object, runs the model as many times as needed, and returns a JSON object. Below is a complete inferlet that answers a question and streams the reply token by token.

The codeCopy Anchor Link

from inferlet import chat, model, session
from inferlet.eta import ForwardPass, Pipeline, StateGeometry, WorkingSet, intrinsics, reduce_argmax

STOP = set(chat.stop_tokens())


async def generate(ws, pipe, tokens, max_tokens=256):
    """Feed `tokens`, then decode greedily until a stop token. Yields each new token."""
    for _ in range(max_tokens):
        fwd = ForwardPass()
        fwd.embed(tokens)
        fwd.state([ws], StateGeometry.folding())
        nxt = fwd.epilogue(lambda: reduce_argmax(intrinsics.output(0, "logits")))
        pipe.submit(fwd)
        token = await nxt
        if token in STOP:
            return
        yield token
        tokens = [token]


async def main(input: dict) -> dict:
    prompt = (chat.system("You are a concise assistant.")
              + chat.user(input["question"])
              + chat.cue_without_thinking())
    ws, pipe = WorkingSet(), Pipeline()

    reply, sent = [], ""
    async for token in generate(ws, pipe, prompt):
        reply.append(token)
        text = model.decode(reply)
        session.send(text[len(sent):])     # stream the newly decoded text
        sent = text
    pipe.close()
    return {"text": model.decode(reply)}

Run itCopy Anchor Link

pie run main.py -- --question "What is the capital of France?"

pie run prints the streamed text as tokens arrive, followed by the returned JSON object:

Paris is the capital of France.
{"text": "Paris is the capital of France."}

Line by lineCopy Anchor Link

chat.system, chat.user, chat.cue_without_thinking format messages into token IDs using the active model's chat template, so the same code works on Qwen, Llama, Gemma, and other supported models. A cue marks the start of the assistant's turn. chat.cue_without_thinking() tells reasoning models to skip their thinking block; chat.cue() lets them think first.

WorkingSet() holds the sequence's KV cache (and recurrent state) inside the model. Every forward pass that references ws extends that state.

generate is the decoding loop. Each iteration builds one ForwardPass:

1. fwd.embed(tokens) feeds the entire prompt on the first step, and only [token] on each step after.
2. fwd.state([ws], StateGeometry.folding()) appends the new tokens to ws and freezes them into the KV cache.
3. fwd.epilogue(...) runs on the GPU after the forward pass and picks the highest-scoring next token (reduce_argmax).

Pipeline runs passes in submission order. pipe.submit(fwd) does not block; await nxt waits for the chosen token ID.

session.send streams text back to whoever started the inferlet: the terminal, a WebSocket client, or an embedded app.

Changing the loopCopy Anchor Link

Because the loop lives in your script, small edits change how generation works:

• Sample instead of taking the top token. Replace reduce_argmax in the epilogue with gumbel_max(logits / temperature, rng). See Device epilogues.
• Stop on your own condition, such as a phrase, token budget, or time limit.
• Let the model think before answering by using chat.cue(). Chat, tools & reasoning shows how to separate the thinking block from the final answer.

The rest of the Program pages reuse generate unchanged.