Inferlet API
The inferlet SDK for Python, JavaScript, and Rust.
The Python (inferlet), JavaScript (@pie-project/inferlet), and Rust (inferlet) SDKs expose the same interface and compile to identical device programs. This page uses the Python names; JavaScript and Rust follow them closely.
Entry point
async def main(input: dict) -> dict: ...#[inferlet::main]
async fn main(input: String) -> Result<String, String> { ... }model
| Function | Returns |
|---|---|
model.encode(text) · model.decode(tokens) | Token IDs from text, or text from token IDs. |
model.name() · model.architecture() | Name and architecture of the loaded model. |
model.output_vocab_size() | Vocabulary size of the output logits. |
model.kv_page_size() · model.frame_size() · model.channel_capacity() · model.run_ahead_window() | Engine runtime constants. |
model.draft_block() | The model's speculative draft head, if one was imported. |
model.switches() · model.switch(name) | Execution modes declared by the model, such as diffusion steps. |
chat
| Function | Returns tokens for |
|---|---|
chat.prefix() | The model's opening (BOS) tokens for raw prompts. |
chat.system(text) · chat.user(text) · chat.assistant(text) | A single system, user, or assistant turn. |
chat.system_user(system, user) · chat.first_user(text) | An opening turn for templates that combine system and first user messages. |
chat.system_thinking(text, effort) | A system turn that sets reasoning effort. |
chat.cue() · chat.cue_without_thinking() · chat.cue_thinking(bool) | The start of the assistant's reply. |
chat.seal() | The end of the current assistant turn. |
chat.stop_tokens() | Token IDs that end a reply. |
tools
| Function | Description |
|---|---|
tools.equip(specs) | Format JSON function schemas into prompt token IDs. |
tools.answer(name, value) | Format a tool's return value into token IDs. |
tools.Decoder() | Streaming tool parser: .feed(tokens) returns Event.Start or Event.Call(call) with call.name and call.arguments_json. |
tools.format(specs) · tools.create_matcher(specs) | Build the grammar or Matcher for valid tool calls. |
reasoning
| Function | Description |
|---|---|
reasoning.Decoder.for_cue(cue) | Create a decoder for the turn started by cue. |
.feed(tokens) | Feed generated tokens; returns Event.Start, Event.Delta(text), Event.End(text), or Event.Idle. |
grammar and mask
| Function | Description |
|---|---|
Grammar.from_json_schema(s) · from_regex(p) · from_ebnf(g) · json() | Compile a grammar for the model's vocabulary. |
Matcher(grammar) | Track valid next tokens across one generation. |
.mask() · .accept_tokens(t) · .is_terminated() | Get the packed valid-token mask, advance state with chosen tokens, or check if done. |
.fork() · .rollback(n) · .reset() | Clone matcher state, undo $n$ tokens, or reset to the start. |
mask.unpack_mask(packed, vocab) | Unpack a bitmask into a boolean list of length vocab for a channel. |
session
| Function | Description |
|---|---|
session.send(message) · await session.receive() | Send or receive text and JSON messages between the inferlet and the client. |
session.send_file(data) · await session.receive_file() | Send or receive binary data. |
session.file(name) | Read a file attached at launch via pie run --file name=path. |
session::send_to(process, message) (Rust) | Send a message to another inferlet process owned by the same client. |
WorkingSet
| Method | Description |
|---|---|
WorkingSet() | Create an empty sequence state handle (KV cache and recurrent state). |
.fork(pipe) | Create a child working set that shares the frozen prefix. |
.folded() | Number of tokens in the frozen prefix. |
.cached_tokens() | Number of prefix tokens reused from existing cached pages. |
.reserve(end) | Reserve writable slots up to end when slots are chosen on the device. |
ForwardPass
| Method | Description |
|---|---|
.embed(tokens, indptr=None) | Set input tokens; indptr splits them into multiple lanes. |
.state(working_sets, geometry) | Set which working sets the lanes update and how slots are placed. |
.readout(indices) | Select which input rows send logits to the epilogue (defaults to the last row of each lane). |
.mask(rows) | Set a per-row attention mask. |
.adapter(site, f) | Attach an adapter expression f(x, y) at projection site. |
.media(lane, spans) | Attach image or audio spans to a lane. |
.input(lane, name, channel) · .switch(lane, name, value) | Pass extra model inputs or mode switches. |
.checkpoints(slots) | Save recurrent-state checkpoints at given slots for rollback. |
.group(lane, id) | Group lanes that share attention state. |
.prologue(f) · .epilogue(f) | Attach device code before or after the model pass; epilogue returns an awaitable result. |
StateGeometry(positions=, slots=, kv_len=, fold=) places rows explicitly; StateGeometry.folding() appends rows causally and freezes them.
Pipeline
| Method | Description |
|---|---|
.submit(fwd) | Queue one forward pass. |
.submit_frame([fwd, ...]) | Queue passes that run in back-to-back scheduler waves. |
await .run_ahead(fwd, budget, on_token) | Keep a repeating pass in flight on the GPU until on_token returns False. |
.close() · .park() | Close the pipeline, or step out of batching while idle. |
Channel
| Method | Description |
|---|---|
Channel(shape, dtype) · Channel.from_(values, dtype) | Create an empty or pre-filled queue in GPU memory. |
.capacity(n) | Set how many values the channel can buffer. |
.put(x) · .take() | Enqueue or dequeue a tensor from the host or inside an epilogue. |
await .take_scalar() | Read one scalar value on the host. |
eta ops and intrinsics
Used inside prologue and epilogue functions:
| Kind | Operations |
|---|---|
| Arithmetic | add, mul, div, abs, exp, log, cos, l2norm, matmul, max_elem, min_elem |
| Comparison and logic | eq, ne, lt, le, gt, ge, and_, select, mask_apply |
| Shape | reshape, broadcast, cast, constant, iota, indptr |
| Reductions | reduce_max, reduce_sum, reduce_argmax, cumsum, cumprod, cummass_le |
| Distributions | softmax, log_softmax, entropy, entropy_from_logprobs, top_k, nucleus_sample, gumbel, gumbel_max, masked_argmax |
| Indexing | gather, gather_row |
| Masks | causal_mask |
| Intrinsics | intrinsics.output(lane, name) reads a named model output, such as "logits". |