HTTP transport: drop WS server, offline send queue + dead-stream watchdog
Gateway (docs/19): - Remove ws_server.py; frame dispatch factored into dispatch.py - http_server: media upload/pull, pairing over HTTP - protocol: media frames mirrored; tests + ws_probe updated for HTTP App: - HttpGateway: postFrame/uploadMedia/pullMedia no longer throw on network failure (PostResult ok=false / Result.failure) — uncaught SocketTimeoutException on Dispatchers.Default crashed the app - GatewayClient: dead-stream watchdog (health probe every 10s, 2 failures -> redial in ~20s instead of the 45s SSE read timeout); state flips to Reconnecting when the stream dies, restored from the last hello.ack on long-poll success; poke() + backoff reset on app resume (MainActivity.onResume) - Offline sends: composer enabled while disconnected; a send with no response (status 0) stays queued (Pending) and is auto-resent on the next (re)connect after a 2s outbox-replay grace; gateway 4xx rejections fail the bubble (tap to retry, no auto-loop) - ChatStore: echo-replace and thread-relocate also match Failed bubbles (POST response lost in a network drop); loadHistory dedupes local failed bubbles the server already has; failMessage() - MainActivity: poke() on resume so a backgrounded app reconnects promptly instead of waiting out the backoff
This commit is contained in:
1 parent
2349a95dd4
commit
e6015033b6
22 files changed
+2804
-2439
No files matched your search
+32
-28
@@ -1,12 +1,15 @@
|
||||
# 07 — Media (upload, download, playback)
|
||||
|
||||
Media travels **over the WebSocket** as chunked binary frames (decision: no
|
||||
separate HTTP server; keeps the plugin to `websockets` only). Both directions
|
||||
use the same chunking.
|
||||
Media travels **over HTTP** (`POST /v1/media` for upload,
|
||||
`GET /v1/media/{id}` for pull; see `19-http-fallback-transport.md` §19.15).
|
||||
HTTP is the only transport — the WebSocket leg (chunked binary frames) was
|
||||
removed entirely. The contracts below (kinds, sha256, re-sniffing, delivery
|
||||
validation) apply to both directions.
|
||||
|
||||
## 7.1 Kinds & MIME
|
||||
|
||||
`kind` ∈ `image | audio | video | document | voice`.
|
||||
|
||||
- `image` — `image/*` (jpg/png/webp/gif/heic).
|
||||
- `audio` — `audio/*` (mp3/m4a/ogg/…) — music.
|
||||
- `video` — `video/*` (mp4/webm/mov).
|
||||
@@ -20,32 +23,36 @@ receipt (don't trust the client) using hermes helpers
|
||||
## 7.2 Inbound (app → agent) — `media.upload`
|
||||
|
||||
**Flow:**
|
||||
1. App picks a file (SAF) → reads size + MIME.
|
||||
2. App sends `media.upload.start {media_ref, kind, mime, size, filename}`.
|
||||
3. App streams the file as **binary WS frames** (e.g. 256 KiB chunks).
|
||||
4. App sends `media.upload.end {media_ref, sha256}`.
|
||||
5. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
|
||||
|
||||
1. App picks a file (SAF) → reads size + MIME, computes sha256.
|
||||
2. App `POST /v1/media` with the raw file body; metadata in
|
||||
`X-Iris-Media-*` headers (`media_ref`, `kind`, `mime`, `filename`,
|
||||
`sha256`).
|
||||
3. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
|
||||
cache via hermes `cache_*_from_bytes`:
|
||||
- image → `cache_image_from_bytes`
|
||||
- audio/voice → `cache_audio_from_bytes`
|
||||
- video → `cache_video_from_bytes`
|
||||
- document → `cache_document_from_bytes`
|
||||
→ returns a local path.
|
||||
5b. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
|
||||
6. The path is attached to the next `message.send` via `media_refs`, becoming
|
||||
- document → `cache_document_from_bytes`
|
||||
→ returns a local path.
|
||||
4. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
|
||||
5. The path is attached to the next `message.send` via `media_refs`, becoming
|
||||
`MessageEvent.media_urls` + `media_types`
|
||||
(`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then
|
||||
read the file.
|
||||
|
||||
**Limits:** `get_inbound_media_max_bytes()` / `validate_inbound_media_size`
|
||||
(`base.py:758/779`) enforce the cap; over-limit → `error {code:"media_too_large"}`.
|
||||
(`base.py:758/779`) enforce the cap; over-limit → 413 +
|
||||
`error {code:"media_too_large"}`. (The 1 MiB `MAX_BODY_BYTES` cap applies to
|
||||
JSON *frame* bodies only, not media uploads.)
|
||||
|
||||
**Backpressure:** large uploads use the WS flow control; the plugin reads
|
||||
binary frames into a temp file (not memory) to bound RAM.
|
||||
**Single-shot:** no chunking/resumability — HTTP carries the body; single-user
|
||||
scale makes a one-shot upload sufficient.
|
||||
|
||||
## 7.3 Outbound (agent → app) — `media.offer` / `media.pull`
|
||||
|
||||
**Flow:**
|
||||
|
||||
1. Agent produces/references media (e.g. generates an image, or replies with a
|
||||
`MEDIA:` tag / image URL). hermes base `extract_media` / `extract_images`
|
||||
(`base.py:4439/4884`) pull these out and call the adapter's
|
||||
@@ -54,14 +61,13 @@ binary frames into a temp file (not memory) to bound RAM.
|
||||
2. Adapter stages the file in the media cache, mints a `media_id`, and emits
|
||||
`media.offer {media_id, kind, mime, size, filename}` (inside/with the
|
||||
`message` frame's `media[]`).
|
||||
3. App sends `media.pull {media_id}`.
|
||||
4. Plugin streams the file as **binary WS frames**; ends with
|
||||
`media.pull.end {ok:true}`.
|
||||
5. App writes to its cache dir and hands the path to the player/viewer.
|
||||
3. App `GET /v1/media/{id}` — the full file body.
|
||||
4. App writes to its cache dir and hands the path to the player/viewer.
|
||||
|
||||
**Security:** `validate_media_delivery_path` (`base.py:1684`) + the media
|
||||
delivery root/recency/denied-path checks (`base.py:1312-1480`) ensure the plugin
|
||||
only serves files hermes is allowed to deliver (no arbitrary file read).
|
||||
only serves files hermes is allowed to deliver (no arbitrary file read). The
|
||||
delivery-path check is re-run **at pull time**, not just at offer time.
|
||||
|
||||
## 7.4 Live playback (AI-sent music/video)
|
||||
|
||||
@@ -79,14 +85,12 @@ only serves files hermes is allowed to deliver (no arbitrary file read).
|
||||
a desktop backend (see `11-desktop-app.md`): a `libmpv`/`mpv`-backed surface
|
||||
or a WebView fallback for video, and a desktop audio player for music.
|
||||
|
||||
## 7.5 Chunking parameters
|
||||
## 7.5 Integrity
|
||||
|
||||
- Chunk size: **256 KiB** (tunable).
|
||||
- Binary frames carry raw bytes only; framing/metadata is in the JSON header +
|
||||
end frames.
|
||||
- Reassembly is ordered (WS preserves order); a gap/corruption → abort +
|
||||
`error {code:"internal"}` + retry the whole transfer.
|
||||
- `sha256` in `media.upload.end` / a size check on pull verify integrity.
|
||||
- Upload: `sha256` (precomputed by the app, sent in `X-Iris-Media-Sha256`)
|
||||
is verified by the plugin; mismatch → `media.upload.ack {ok:false}`.
|
||||
- Pull: the app checks the received size against the offered `size`.
|
||||
- A failed transfer → retry the whole upload (single-shot, no resume).
|
||||
|
||||
## 7.6 App-side storage
|
||||
|
||||
@@ -94,4 +98,4 @@ only serves files hermes is allowed to deliver (no arbitrary file read).
|
||||
- LRU eviction by size (configurable, default 500 MB) so old media doesn't fill
|
||||
the device.
|
||||
- A `MediaRepository` tracks `{media_id, local_path, kind, size, ts}` in Room so
|
||||
bubbles can re-render players after process death.
|
||||
bubbles can re-render players after process death.
|
||||
@@ -6,11 +6,24 @@ by the gateway. When the WS is down (flaky network, NAT timeout, app just
|
||||
relaunched), the app **sends over `POST` and receives over SSE** instead of
|
||||
waiting 2–20 s for a WS redial.
|
||||
|
||||
Status: **implemented** (gateway leg: `gateway-plugin/http_server.py`; app
|
||||
leg: `app/shared/src/commonMain/kotlin/iris/net/HttpGateway.kt` +
|
||||
`GatewayClient.State.HttpFallback`). Complements — does not replace —
|
||||
`04-wire-protocol.md` (frames), `08-push.md` (outbox/sync/push), and
|
||||
`09-pairing-security.md` (auth model).
|
||||
Status: **implemented — and now the ONLY transport.** The WebSocket leg has
|
||||
been removed entirely from the codebase (gateway `ws_server.py` deleted;
|
||||
WS-only frames `hello`/`ping`/`pong`/`media.upload.*`/`media.pull*` dropped
|
||||
from `protocol.py` and `Protocol.kt`; `GatewayClient` is HTTP-only with no
|
||||
`HttpFallback` state — it *is* the connected state). HTTP is the primary and
|
||||
sole transport: v1 (JSON frames over POST/SSE/long-poll) and v2 (media over
|
||||
`POST /v1/media` + `GET /v1/media/{id}`, §19.15). Gateway leg:
|
||||
`gateway-plugin/http_server.py` (+ `dispatch.py` for frame dispatch);
|
||||
app leg: `app/shared/src/commonMain/kotlin/iris/net/HttpGateway.kt` +
|
||||
`GatewayClient.kt`. Legacy `ws(s)://` URLs entered by users are still
|
||||
accepted and rewritten to `http(s)://` (`HttpGateway.deriveHttpUrl`).
|
||||
Complements — does not replace — `04-wire-protocol.md` (frames),
|
||||
`08-push.md` (outbox/sync/push), and `09-pairing-security.md` (auth model).
|
||||
|
||||
> **Note:** the rest of this document describes the original design, in
|
||||
> which HTTP was a *fallback* next to a WS primary. That framing is
|
||||
> historical; where it says "WS (primary)" / "HTTP (fallback)", read
|
||||
> "HTTP (the only transport)".
|
||||
|
||||
## 19.1 Problem
|
||||
|
||||
@@ -78,17 +91,17 @@ acceptable alternative if preferred).
|
||||
|
||||
**v1 scope**
|
||||
|
||||
| Over HTTP (v1) | WS-only (v1) |
|
||||
| Over HTTP | WS-only |
|
||||
| --- | --- |
|
||||
| All JSON request frames (`message.send`, `search`, `channel.*`, `commands.catalog`, `agent.stop`/`agent.steer`, …) via one generic endpoint | Binary media upload (chunked binary frames) |
|
||||
| All event/response frames via SSE (or long-poll) | Binary media pull stream |
|
||||
| All JSON request frames (`message.send`, `search`, `channel.*`, `commands.catalog`, `agent.stop`/`agent.steer`, …) via one generic endpoint | — |
|
||||
| All event/response frames via SSE (or long-poll) | — |
|
||||
| `sync` catch-up (same outbox, same cursor) | — |
|
||||
| Media upload + pull (`POST /v1/media`, `GET /v1/media/{id}`, v2 — §19.15) | — |
|
||||
|
||||
Media stays WS-only in v1: it is the one part of the protocol that is
|
||||
inherently binary/streaming, and attachments are a rarer action than sending
|
||||
text. While in HTTP-fallback mode the composer disables the attach button
|
||||
("media needs the live connection"). HTTP media endpoints are a v2 item
|
||||
(§19.13).
|
||||
With v2 the HTTP leg is feature-complete: media no longer needs the WS
|
||||
(the composer's attach button is enabled in `HTTP_FALLBACK` too). The WS
|
||||
binary media frames remain accepted for WS clients, but the app routes media
|
||||
over HTTP whenever the WS is down.
|
||||
|
||||
## 19.4 Gateway: `gateway-plugin/http_server.py`
|
||||
|
||||
@@ -117,6 +130,8 @@ to the WS server.
|
||||
| `POST /v1/frame` | Bearer token | Accept **any** JSON frame the WS accepts (except binary media). Body = one frame envelope (`04-wire-protocol.md`). Dispatched through the *same* adapter handlers as WS (`on_message_send`, `on_search`, …). |
|
||||
| `GET /v1/events?cursor=N` | Bearer token | **SSE** stream: catch-up from the outbox, then live frames (§19.5). |
|
||||
| `GET /v1/poll?cursor=N` | Bearer token | **Long-poll** fallback where SSE is blocked (§19.6). |
|
||||
| `POST /v1/media` | Bearer token | **Media upload** (v2, §19.15): whole file as the body, metadata in `X-Iris-Media-*` headers. |
|
||||
| `GET /v1/media/{media_id}` | Bearer token | **Media pull** (v2, §19.15): streams an outbound offer as the response body. |
|
||||
|
||||
### Auth & limits
|
||||
|
||||
@@ -243,7 +258,9 @@ New `iris/net/HttpGateway.kt` (OkHttp) + a transport state machine inside
|
||||
- **Send path:** `sendMessage()` builds the same `message.send` frame JSON and
|
||||
writes it to WS or POST depending on state. The `State.Connected` gate in
|
||||
`ChatScreen.doSend()` becomes `state is Connected || state is HttpFallback`.
|
||||
- **Media:** disabled in the composer while in `HTTP_FALLBACK` (v1).
|
||||
- **Media:** works in `HTTP_FALLBACK` too (v2, §19.15) — uploads go via
|
||||
`POST /v1/media`, pulls via `GET /v1/media/{id}`; the composer's attach
|
||||
button is enabled in both connected states.
|
||||
- **UI:** status pill shows "connected" (WS) or "connected · http" (fallback)
|
||||
— both green; the fallback is a healthy state, not an error.
|
||||
|
||||
@@ -283,8 +300,13 @@ New `iris/net/HttpGateway.kt` (OkHttp) + a transport state machine inside
|
||||
- long-poll: returns on new frame; empty 200 at timeout with advanced cursor.
|
||||
- **delivery counting:** frame with only an SSE subscriber → `delivered ≥ 1`
|
||||
→ **no push fired** (the critical regression test for §19.8).
|
||||
- **media (v2, §19.15):** `POST /v1/media` happy path (201 ack + cached
|
||||
entry), sha256 mismatch, oversize → 413, missing ref / bad kind → 400,
|
||||
auth → 401, magic-byte reclassification; `GET /v1/media/{id}` happy path
|
||||
(bytes + content-type), unknown id → 404, denied path → 404.
|
||||
- **Probe:** `ws_probe.py` gains an `--http` mode (health, post, SSE read with
|
||||
assertion flags, per `gateway-plugin/tests/README.md`).
|
||||
assertion flags, per `gateway-plugin/tests/README.md`) + `--http-media FILE`
|
||||
(v2: upload round-trip via `POST /v1/media`, exit 23 on rejection).
|
||||
- **Kotlin** (`:shared` commonTest): SSE parser (multi-line data, comments,
|
||||
`Last-Event-ID` bookkeeping); transport state machine transitions (fake
|
||||
clock: WS-loss → immediate fallback; startup race → fallback in < 1 s).
|
||||
@@ -295,9 +317,9 @@ New `iris/net/HttpGateway.kt` (OkHttp) + a transport state machine inside
|
||||
|
||||
## 19.13 Non-goals (v1) / future
|
||||
|
||||
- **Media over HTTP** (v2): `POST /v1/media` (chunked, same sha256 contract as
|
||||
`07-media.md`) + `GET /v1/media/{id}` for pull/playback. Unblocks
|
||||
attachments in fallback mode.
|
||||
- ~~**Media over HTTP** (v2)~~ — **done** (§19.15): `POST /v1/media`
|
||||
(whole-file body, sha256 contract per `07-media.md`) +
|
||||
`GET /v1/media/{id}` for pull/playback. Attachments work in fallback mode.
|
||||
- **App-side send outbox** (companion work, separate doc): queue sends locally
|
||||
when *both* legs are down; drains over whichever leg recovers. This doc
|
||||
removes the 2–20 s wait; the outbox removes the last "gateway was down for
|
||||
@@ -307,6 +329,69 @@ New `iris/net/HttpGateway.kt` (OkHttp) + a transport state machine inside
|
||||
- Per-device tokens (`16-open-questions.md` #3) apply to both legs identically
|
||||
when implemented.
|
||||
|
||||
## 19.15 Media over HTTP (v2)
|
||||
|
||||
The last WS-only feature, closed out so the HTTP leg is feature-complete.
|
||||
Same contracts as `07-media.md` — only the transport changes.
|
||||
|
||||
### Upload — `POST /v1/media`
|
||||
|
||||
One request per file (no chunked/resumable protocol — HTTP handles the
|
||||
body; single-user scale makes resume unnecessary):
|
||||
|
||||
```
|
||||
POST /v1/media
|
||||
Authorization: Bearer <token>
|
||||
X-Iris-Device: <device_id>
|
||||
X-Iris-Media-Ref: up_123456 # app-chosen ref (mu_*/up_*), ≤ 64 chars
|
||||
X-Iris-Media-Kind: image|audio|video|document|voice
|
||||
X-Iris-Media-Filename: photo.jpg
|
||||
X-Iris-Media-Sha256: <64 hex> # precomputed (headers precede the body)
|
||||
Content-Type: <media mime> # doubles as the declared MIME
|
||||
Content-Length: <size>
|
||||
|
||||
<raw file bytes>
|
||||
```
|
||||
|
||||
- **Response:** `201` with the `media.upload.ack` frame as the body
|
||||
(`{ok, media_ref}`); validation failures return the `error` frame as the
|
||||
4xx body with the same codes as the WS path (`media_too_large` → 413,
|
||||
`unsupported` → 400, `internal` → 500, `not_found` → 404).
|
||||
- **Server flow:** the body is streamed to a temp file in 256 KiB reads
|
||||
(bounded RAM, same `UploadSession` as the WS path), then
|
||||
`complete_upload` verifies size + sha256, re-sniffs the kind from magic
|
||||
bytes (the client's declared kind is not trusted), and caches via the
|
||||
hermes `cache_*_from_bytes` helpers. Runs entirely on the handler thread
|
||||
— no asyncio bridge (plain file IO).
|
||||
- **Limits:** `Content-Length` is checked against `max_upload_bytes`
|
||||
*before* reading the body (early 413); the 64 KiB `/v1/frame` body cap
|
||||
does not apply. Same per-device rate limit as the other endpoints.
|
||||
- **Abort:** a client that disconnects mid-body leaves a short read → the
|
||||
upload session (temp file) is discarded; nothing is cached.
|
||||
- The ref then travels in `message.send`'s `media_refs` exactly as on the WS
|
||||
path (single-use, resolved to `MessageEvent.media_urls`).
|
||||
|
||||
### Pull — `GET /v1/media/{media_id}`
|
||||
|
||||
- `media.offer` is a plain JSON event frame — it arrives on the SSE stream
|
||||
unchanged; only the byte transfer moves to HTTP.
|
||||
- **Response:** `200` with the file as the body,
|
||||
`Content-Type: <mime>`, `Content-Length: <size>`,
|
||||
`Content-Disposition: attachment; filename="<name>"`. Unknown id or a
|
||||
path that fails delivery validation → `404` with the `error` frame
|
||||
(`not_found`) — the delivery-path check is re-run at pull time, exactly
|
||||
as the WS `media.pull` handler does.
|
||||
- The app streams the body into its media cache (same
|
||||
`MediaCache.openWriter` path as the WS pull); playback is unchanged
|
||||
(`07-media.md` §7.4).
|
||||
|
||||
### What stays WS-only
|
||||
|
||||
Nothing feature-wise. The WS binary media frames (`media.upload.start/end`,
|
||||
`media.pull` + binary chunks) remain accepted for WS clients, and
|
||||
`POST /v1/frame` still rejects those frame types (they have HTTP endpoint
|
||||
equivalents now, not a WS dependency).
|
||||
|
||||
## 19.14 Effort & change list
|
||||
|
||||
| Slice | Files | Est. |
|
||||
|
||||
Reference in new issue
Block a user