# 04 — Wire Protocol JSON frames over a single WebSocket. One connection per device. Text frames are JSON; media travels as **binary frames** (chunked) referenced by a header frame. ## Envelope Every frame: ```json { "v": 1, "id": 42, // optional; present on requests + their responses "type": "message", // frame type (below) "chat_id": "default", // optional; scope for chat-scoped frames "thread_id": "t_123", // optional "payload": { } // type-specific object } ``` - `v` — protocol version (currently `1`). Server rejects unknown major versions. - `id` — request id (client-chosen). Responses/acks echo it. Events have no `id`. - `chat_id` / `thread_id` — top-level for convenience; may also be in `payload`. - `cursor` — outbox cursor the frame was parked under. Present **only** on frames replayed by `sync` (live frames carry none). The app compares it against `last_pushed_cursor` from `hello.ack` to skip re-notifying frames that already woke the device via push (`08-push.md` §8.7). - Unknown `type`s are ignored (forward-compat); unknown `payload` fields ignored. **Binary media frames** are not JSON. A media transfer is: one JSON header frame (`media.upload.start` / `media.pull` ack) followed by raw binary frames, then a JSON `media.upload.end` / final ack. See `07-media.md`. ## Server → App (events / responses) ### `hello.ack` Pairing succeeded. ```json {"type":"hello.ack","payload":{ "server_caps":{"streaming":true,"reasoning":true,"tools":true,"media":true, "search":true,"push":"fcm","pickers":true}, "sync_cursor":1042, "last_pushed_cursor":1040, "channels":[{"chat_id":"default","name":"Default","kind":"default","is_default":true}] }} ``` `last_pushed_cursor` is the highest outbox cursor already delivered to THIS device via the push backend (0 = never). The app skips system notifications for sync-replayed frames with `cursor <= last_pushed_cursor` — they already woke the device via push (dedupe, `08-push.md` §8.7). ### `message` A final / standalone message. ```json {"type":"message","chat_id":"default","thread_id":null, "payload":{ "message_id":"m_9001","role":"assistant", "text":"Here is the answer…", "reasoning":"The user asked… so I will…", // optional; render ABOVE text "media":[{"media_id":"md_5","kind":"video","mime":"video/mp4","size":123456, "filename":"clip.mp4"}], // optional "reply_to":"m_8999", // optional "model":"qwen3-27b","tokens":11,"ts":1724000000000, "runtime":{"model":"qwen3-27b","context_pct":38,"cwd":"~", "latency":22.5,"cost":0.0012} // optional; see below }} ``` `role` ∈ `user | assistant | system | cron`. `reasoning` present only when the agent produced reasoning and `show_reasoning` is on. `runtime` (optional) is the **structured runtime-metadata footer** the app renders under final assistant messages (Telegram-style). The gateway ALWAYS sends it on final assistant messages; whether/what is shown is a **per-app setting** (Settings → Runtime footer), NOT a hermes config. Keys (all optional; absent when the data is unavailable, e.g. local models have no cost): - `model` — bare model id, vendor prefix dropped (`gpt-5.4`) - `context_pct` — last-call context occupancy, 0-100 (int) - `cwd` — home-relative working dir (`~`) - `latency` — wall-clock turn duration, seconds (float) - `cost` — turn cost, USD (float) ### `message.start` / `message.update` / `message.stop` Streaming a bubble. `update` carries the **full** current text (app replaces). ```json {"type":"message.start","chat_id":"…","payload":{"message_id":"m_9002","role":"assistant"}} {"type":"message.update","chat_id":"…","payload":{"message_id":"m_9002","text":"partial…"}} {"type":"message.stop","chat_id":"…","payload":{"message_id":"m_9002","final_text":"full…", "reasoning":"…","model":"…","tokens":11, "runtime":{"model":"…","context_pct":38,"cwd":"~","latency":22.5}}} ``` ### `message.deleted` The given message(s) were deleted from a chat/thread. Response to a `message.delete` request (`id` set) **and** broadcast to every device so all of them drop the message(s) from their cache. Also outboxed, so a device that was offline learns of the deletion on its next `sync`. ```json {"type":"message.deleted","id":30,"chat_id":"default","thread_id":null, "payload":{"message_ids":["m_9001","m_9002"]}} ``` ### `commentary` Intermediate assistant beat (between tool iterations). ```json {"type":"commentary","chat_id":"…","payload":{"message_id":"m_9003","text":"Let me inspect the repo first."}} ``` ### `tool.start` / `tool.progress` / `tool.end` **Structured** tool events. The app decides how much to show (everything / truncated / nothing). ```json {"type":"tool.start","chat_id":"…","payload":{ "index":3,"name":"terminal","preview":"pytest -q","args":{"command":"pytest -q"},"emoji":"💻"}} {"type":"tool.progress","chat_id":"…","payload":{"index":3,"name":"terminal","note":"running…"}} {"type":"tool.end","chat_id":"…","payload":{"index":3,"name":"terminal","ok":true,"duration":12.4, "output_preview":"12 passed"}} ``` `args` may be large; the app truncates per its setting. `output_preview` is a short tail (full output is not streamed — it lives in agent history). `emoji` is a cosmetic per-tool glyph resolved server-side via hermes' `get_tool_emoji` (active-skin overrides, then the tool registry's per-tool `emoji` — e.g. `read_file` 📖, `write_file` ✍️, `terminal` 💻). Omitted when the tool is unknown, so the app falls back to its own default glyph. ### `todo.update` The agent's **full current todo list** for a chat/thread lane (last-write-wins). Emitted whenever the `todo` tool completes — the tool *result* is the authoritative list, which also covers `merge` writes (whose args carry only the changed items) and read-only calls — and, as a **snapshot**, right after `hello` when a device opens its event stream. ```json {"type":"todo.update","chat_id":"default","thread_id":null,"payload":{ "todos":[ {"id":"1","content":"Scaffold the module","status":"completed"}, {"id":"2","content":"Wire the store","status":"in_progress"}, {"id":"3","content":"Add tests","status":"pending"} ]}} ``` `status` is one of `pending | in_progress | completed | cancelled`. The frame is **ephemeral**: it is never outboxed, so a reconnecting device learns the current list from the snapshot (not a replay), and a gateway restart drops it (the agent re-emits on the next `todo` call). The app renders it as a compact scrollable strip above the composer (max 3 lines; auto-scrolls to the item whose status just changed) and hides it once the list is empty or fully resolved. ### `typing` / `typing.stop` ```json {"type":"typing","chat_id":"…","payload":{"on":true}} ``` ### `notification` In-app banner (foreground) and/or push mirror (background). ```json {"type":"notification","chat_id":"…","payload":{ "kind":"channel_renamed","title":"ARIA","body":"Renamed topic to …","ts":1724000000000}} ``` `kind` ∈ `channel_renamed | channel_created | cron | approval | clarify | generic`. ### `picker.model` / `picker.choice` / `picker.clarify` / `picker.approval` / `picker.confirm` Interactive prompts. App renders a native picker; answers via `picker.select`. `picker.choice` is implemented (the generic finite-choice menu used by `/reasoning`, `/fast`, and any future finite-choice slash command — hermes calls the adapter's `send_choice_picker` when the platform supports it). The server runs the command's selection callback on `picker.select` and delivers its reply as a normal `message` in the picker's chat. The frame is outboxed (a reconnecting device re-renders a still-pending picker); pending state is in-memory only, so a gateway restart expires it (a stale `picker.select` is a no-op). With no live device the adapter reports failure and hermes falls back to the text status card. ```json {"type":"picker.model","chat_id":"…","payload":{ "picker_id":"pm_1","current_model":"qwen3-27b","current_provider":"local", "providers":[{"id":"local","label":"Local","models":[{"id":"qwen3-27b","label":"Qwen3 27B"}]}]}} {"type":"picker.choice","chat_id":"…","payload":{ "picker_id":"pc_1","title":"Reasoning effort","choices":[ {"value":"low","label":"Low"},{"value":"high","label":"High","is_current":true}]}} ``` ### `channel.list` / `channel.created` / `channel.renamed` / `channel.deleted` Channel directory updates. **Broadcast to all connected devices** (no explicit subscribe; the server pushes to every open WS). ```json {"type":"channel.created","payload":{"chat_id":"chan_7","name":"Cron Reports", "kind":"channel","parent_chat_id":null}} ``` `channel.created` may carry `"auto":true` for a thread the gateway minted itself for an incoming message (auto-threading): the name is an instant derived title, and a follow-up `channel.renamed` upgrades it to the model's title. ### `history` Response to a `history` request. Returns a page of messages for a chat/thread. ```json {"type":"history","id":20,"chat_id":"default","thread_id":null, "payload":{ "messages":[ {"message_id":"m_8990","role":"user","text":"Hi","ts":1723990000000}, {"message_id":"m_8991","role":"assistant","text":"Hello!","reasoning":"…", "model":"qwen3-27b","tokens":8,"ts":1723990001000} ], "has_more":true, "oldest_message_id":"m_8990" }} ``` `messages` are ordered oldest → newest. Paginate with `before_message_id` in the request. The app uses this to **populate the initial view** when a channel is opened (complements `sync`, which only replays undelivered outbox frames). ### `commands.catalog` Request (app → server, empty payload) and response: the gateway's slash-command catalog for the app's `/` drawer. Derived from hermes' central `COMMAND_REGISTRY` (the same source the gateway help and the Telegram command menu use), restricted to commands available on gateway surfaces, plus plugin-registered commands. The app fuzzy-matches the typed prefix client-side (no `commands.complete` round-trip). ```json {"type":"commands.catalog","id":21,"payload":{ "commands":[ {"name":"/new","description":"Start a new session (fresh session ID + history)","args_hint":"[name]","category":"Session","aliases":["/reset"]}, {"name":"/model","description":"Switch model","args_hint":"","category":"Configuration","aliases":[]}, {"name":"/status","description":"Show session status","args_hint":"","category":"Info","aliases":[]} ]}} ``` `name`/`aliases` carry the leading slash; `args_hint` is the registry's argument placeholder (empty when the command takes none). ### `commands.complete` Response to a `commands.complete` request. Autocomplete matches for a typed prefix. ```json {"type":"commands.complete","id":22,"payload":{ "prefix":"/mod", "matches":[ {"name":"/model","description":"Switch model","args_hint":""} ]}} ``` ### `agent.busy` / `agent.idle` Agent lifecycle for a chat/thread. App shows a "thinking…" indicator on `busy`. ```json {"type":"agent.busy","chat_id":"default","thread_id":null, "payload":{"reason":"processing"}} {"type":"agent.idle","chat_id":"default","thread_id":null,"payload":{}} ``` `reason` ∈ `processing | tool | waiting_input | cron`. ### `search.results` ```json {"type":"search.results","id":7,"payload":{ "query":"deploy","scope":"all","hits":[ {"message_id":"m_123","chat_id":"chan_7","thread_id":null, "role":"assistant","snippet":"…deploy the service…","ts":1723900000000}]}} ``` ### `media.offer` Agent-sent media is available; app pulls bytes. ```json {"type":"media.offer","chat_id":"…","payload":{ "media_id":"md_5","kind":"video","mime":"video/mp4","size":123456,"filename":"clip.mp4"}} ``` ### `read.receipt` The gateway acknowledges that the agent has received and started processing the user's message. The app uses it to show ✓✓ on user bubbles. ```json {"type":"read.receipt","chat_id":"default","payload":{"message_id":"m_9001"}} ``` Emitted to the originating connection when a `message.send` is accepted for processing (at the moment it is handed to the agent), for user-originated messages only. ### `status` Gateway health state. Broadcast to all connected clients at startup (`state: "online"`) and to late joiners on `hello.ack`. The gateway also broadcasts `state: "restarting"` on its shutdown path (restart/stop), right before closing the sockets — the app posts the "Gateway restarting" chat notice immediately on that frame (the socket can take up to the ~20 s ping timeout to actually drop, so the notice must not wait for the disconnect); a plain network drop shows just the reconnect banner. `degraded` is reserved for future use. ```json {"type":"status","payload":{"state":"online"}} ``` `state` ∈ `online | restarting | degraded`. ### `error` ```json {"type":"error","id":7,"payload":{"code":"not_found","message":"chat_id unknown"}} ``` `code` ∈ `auth | not_found | rate_limited | media_too_large | unsupported | internal`. ### `pong` Keepalive reply to `ping`. ## App → Server (requests / actions) ### `hello` First frame; auth + caps. ```json {"type":"hello","payload":{ "token":"","device_id":"dev_a1b2","device_name":"MIX 2S", "caps":{"min_protocol":1,"media":true,"push":"fcm"}, "fcm_token":"","ntfy_topic":""}} ``` ### `message.send` Send text (or a `/slash-command`). ```json {"type":"message.send","id":10,"chat_id":"default","thread_id":null, "payload":{"text":"/model qwen3-27b","reply_to":"m_9001","media_refs":["mu_1"], "auto_thread":false}} ``` `media_refs` reference completed `media.upload`s to attach. `auto_thread` (optional, default false) asks the gateway to mint a fresh thread for the message (auto-threading, `06-channels-cron-search.md` §6.3): honored only in a channel's flat lane (`thread_id` null) with non-empty text that is not a slash command. The gateway then broadcasts `channel.created {auto:true}` and the echo / agent turn carry the new `thread_id`. ### `media.upload.start` / (binary) / `media.upload.end` See `07-media.md`. ```json {"type":"media.upload.start","id":11,"payload":{ "media_ref":"mu_1","kind":"image","mime":"image/jpeg","size":204800,"filename":"a.jpg"}} // … binary frames … {"type":"media.upload.end","id":11,"payload":{"media_ref":"mu_1","sha256":"…"}} ``` ### `media.upload.ack` Server → App response to `media.upload.end`: the ref is cached and may now be referenced in a `message.send` `media_refs`. Failures use `error` frames instead. ```json {"type":"media.upload.ack","id":11,"payload":{"ok":true,"media_ref":"mu_1"}} ``` ### `media.pull` Request agent-sent media bytes. ```json {"type":"media.pull","id":12,"payload":{"media_id":"md_5"}} // server replies: binary frames, then {"type":"media.pull.end","id":12,"payload":{"ok":true}} ``` ### `picker.select` Answer an interactive picker. ```json {"type":"picker.select","id":13,"payload":{"picker_id":"pm_1","value":"local/qwen3-27b"}} ``` ### `channel.create` / `channel.rename` / `channel.set_default` / `channel.delete` ```json {"type":"channel.create","id":14,"payload":{"name":"Cron Reports","kind":"channel"}} {"type":"channel.rename","id":15,"chat_id":"chan_7","payload":{"name":"Reports"}} {"type":"channel.set_default","id":16,"chat_id":"chan_7","payload":{}} ``` `channel.delete` is a **hard delete**: the channel/thread row is removed from the directory and the lane's history is wiped from the outbox and the hermes session store (no search trace, not recoverable). Deleting a channel also removes its threads. The default channel cannot be deleted. ### `search` ```json {"type":"search","id":17,"payload":{"query":"deploy","scope":"all"}} {"type":"search","id":18,"payload":{"query":"deploy","scope":"chat","chat_id":"chan_7","thread_id":null}} ``` `scope` ∈ `all | chat`. ### `read.receipt` App → server: "user has viewed this message." Server stores the read state and broadcasts to other devices (for multi-device ✓✓ sync). The app uses it to mark messages as read locally (✓✓ on user bubbles). ```json {"type":"read.receipt","payload":{"chat_id":"default","message_id":"m_9001"}} ``` ### `history` Load a page of messages for a chat/thread (initial open, scroll-up pagination). ```json {"type":"history","id":20,"chat_id":"default","thread_id":null, "payload":{"before_message_id":"m_8990","limit":50}} ``` `before_message_id` — return messages older than this (omit for newest page). `limit` — max messages (default 50, max 200). ### `message.delete` Completely delete the given message(s) from a chat/thread. The server removes them from the outbox (so `history`/`sync` no longer return them) **and** from the hermes session store (so no search trace survives and they are not recoverable), then broadcasts `message.deleted` to every device. Idempotent: a message already gone (pruned by retention) still yields a `message.deleted` broadcast so live caches drop it. ```json {"type":"message.delete","id":30,"chat_id":"default","thread_id":null, "payload":{"message_ids":["m_9001","m_9002"]}} ``` ### `commands.catalog` Fetch the full slash-command catalog (for the `Menü` bottom sheet). ```json {"type":"commands.catalog","id":21,"payload":{}} ``` ### `commands.complete` Autocomplete for a typed `/prefix`. ```json {"type":"commands.complete","id":22,"payload":{"prefix":"/mod"}} ``` ### `agent.stop` Stop the current agent turn (abort generation / tool execution). ```json {"type":"agent.stop","id":23,"chat_id":"default","thread_id":null,"payload":{}} ``` ### `agent.steer` Inject a steering message mid-turn (redirects the agent without a new turn). ```json {"type":"agent.steer","id":24,"chat_id":"default","thread_id":null, "payload":{"text":"Actually, focus on the error case."}} ``` ### `sync` Reconnect catch-up. Replays **undelivered outbox frames** (frames sent while this device was offline). Does NOT load full history — use `history` for that. ```json {"type":"sync","id":19,"payload":{"cursor":1042}} // server replays outbox frames with cursor > 1042, then {"type":"sync.done","id":19,"payload":{"cursor":1099}} ``` ### `fcm.register` Update push token. ```json {"type":"fcm.register","payload":{"fcm_token":"","ntfy_topic":""}} ``` ### `ping` Keepalive. `{"type":"ping","payload":{"ts":1724000000000}}` → `pong`. ## Ordering & reliability - Frames are ordered per connection (TCP/WS). Streaming `message.update` frames for a `message_id` are monotonic; the app may coalesce to the latest. - Terminal frames (`message`, `message.stop`, `message.deleted`, `tool.end`, `notification`, `picker.*`, `channel.*`, `agent.busy`, `agent.idle`, `history`, `commands.catalog`, `commands.complete`) are **never dropped** under backpressure; only intermediate `message.update`/`tool.progress` are coalesced. - Anything not delivered live goes to the **outbox** and is replayed by `sync`. - **Broadcast:** channel directory events (`channel.*`) and read-receipts are pushed to **all** connected devices for that gateway (no subscribe step). - Requests get exactly one response or `error` (matched by `id`).