Chat, tools & reasoning
Format chat turns, tool calls, and thinking blocks across models.
Different model families use different special tokens for chat turns, tool calls, and thinking blocks. The chat, tools, and reasoning modules apply the active model's template automatically, so one inferlet runs across model families without change.
Chat turns
Each function returns a list of token IDs ready for fwd.embed():
| Function | Returns tokens for |
|---|---|
chat.system(text) | A system message. |
chat.user(text) | A user message. |
chat.assistant(text) | A past assistant message (when replaying conversation history). |
chat.cue() | The start of the assistant's reply. |
chat.cue_without_thinking() | The start of the reply, instructing a reasoning model to skip its thinking block. |
chat.seal() | The end of the current assistant turn. |
chat.stop_tokens() | Token IDs that mark the end of a reply. |
chat.prefix() | The model's opening (BOS) tokens for raw, non-chat prompts. |
prompt = chat.system("You are terse.") + chat.user("Hi!") + chat.cue_without_thinking()Tool calls
Describe each tool as a JSON function schema (the same format the OpenAI API uses). tools.equip() formats the schemas into prompt tokens, and tools.Decoder() parses generated tokens to detect tool calls:
import json
from inferlet import chat, tools
CALCULATOR = json.dumps({
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate an arithmetic expression.",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string"}},
"required": ["expression"],
},
},
})
prompt = (chat.system("Always call the calculate tool for arithmetic.")
+ tools.equip([CALCULATOR])
+ chat.user("What is 1234 * 5678 + 91?")
+ chat.cue_without_thinking())
decoder = tools.Decoder()
async for token in generate(ws, pipe, prompt):
event = decoder.feed([token])
if isinstance(event, tools.Event.Call):
print(event.call.name, event.call.arguments_json)
# calculate {"expression": "1234 * 5678 + 91"}To return the tool's output to the model, append chat.seal() + tools.answer(name, result) + chat.cue() to the same working set. Agents puts this into a multi-step loop.
To force the model to emit valid tool-call syntax, call tools.create_matcher(specs) to build a grammar matcher for the model's tool format, then apply it as shown in Structured output.
Reasoning
Reasoning models emit a thinking block before their final answer. reasoning.Decoder separates the two streams. Pass the cue tokens you used to reasoning.Decoder.for_cue(cue), because some templates open the <think> tag inside the cue itself:
from inferlet import reasoning
cue = chat.cue()
thinking = reasoning.Decoder.for_cue(cue)
async for token in generate(ws, pipe, chat.user(question) + cue):
match thinking.feed([token]):
case reasoning.Event.Delta(text=t):
... # thinking text as it streams
case reasoning.Event.End(text=t):
... # complete thinking block; the answer followsFor templates that support configurable reasoning budgets, chat.system_thinking(message, effort="medium") creates a system turn that sets the reasoning effort.
Apply a grammar matcher so tool calls always parse as valid JSON.