HTTP transport: drop WS server, offline send queue + dead-stream watchdog
CI / Gateway plugin tests (push) Successful in 5m9s
CI / Kotlin tests (android host + desktop) (push) Successful in 6m55s

Gateway (docs/19):
- Remove ws_server.py; frame dispatch factored into dispatch.py
- http_server: media upload/pull, pairing over HTTP
- protocol: media frames mirrored; tests + ws_probe updated for HTTP

App:
- HttpGateway: postFrame/uploadMedia/pullMedia no longer throw on
  network failure (PostResult ok=false / Result.failure) — uncaught
  SocketTimeoutException on Dispatchers.Default crashed the app
- GatewayClient: dead-stream watchdog (health probe every 10s, 2
  failures -> redial in ~20s instead of the 45s SSE read timeout);
  state flips to Reconnecting when the stream dies, restored from the
  last hello.ack on long-poll success; poke() + backoff reset on app
  resume (MainActivity.onResume)
- Offline sends: composer enabled while disconnected; a send with no
  response (status 0) stays queued (Pending) and is auto-resent on the
  next (re)connect after a 2s outbox-replay grace; gateway 4xx
  rejections fail the bubble (tap to retry, no auto-loop)
- ChatStore: echo-replace and thread-relocate also match Failed
  bubbles (POST response lost in a network drop); loadHistory dedupes
  local failed bubbles the server already has; failMessage()
- MainActivity: poke() on resume so a backgrounded app reconnects
  promptly instead of waiting out the backoff
This commit is contained in:
ARIA committed 2026-08-22 20:10:05 +02:00
1 parent 2349a95dd4
commit e6015033b6
22 files changed
+2804 -2439

No files matched your search

+32 -28
View File
@@ -1,12 +1,15 @@
# 07 — Media (upload, download, playback)
Media travels **over the WebSocket** as chunked binary frames (decision: no
separate HTTP server; keeps the plugin to `websockets` only). Both directions
use the same chunking.
Media travels **over HTTP** (`POST /v1/media` for upload,
`GET /v1/media/{id}` for pull; see `19-http-fallback-transport.md` §19.15).
HTTP is the only transport — the WebSocket leg (chunked binary frames) was
removed entirely. The contracts below (kinds, sha256, re-sniffing, delivery
validation) apply to both directions.
## 7.1 Kinds & MIME
`kind` ∈ `image | audio | video | document | voice`.
- `image` — `image/*` (jpg/png/webp/gif/heic).
- `audio` — `audio/*` (mp3/m4a/ogg/…) — music.
- `video` — `video/*` (mp4/webm/mov).
@@ -20,32 +23,36 @@ receipt (don't trust the client) using hermes helpers
## 7.2 Inbound (app → agent) — `media.upload`
**Flow:**
1. App picks a file (SAF) → reads size + MIME.
2. App sends `media.upload.start {media_ref, kind, mime, size, filename}`.
3. App streams the file as **binary WS frames** (e.g. 256 KiB chunks).
4. App sends `media.upload.end {media_ref, sha256}`.
5. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
1. App picks a file (SAF) → reads size + MIME, computes sha256.
2. App `POST /v1/media` with the raw file body; metadata in
`X-Iris-Media-*` headers (`media_ref`, `kind`, `mime`, `filename`,
`sha256`).
3. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
cache via hermes `cache_*_from_bytes`:
- image → `cache_image_from_bytes`
- audio/voice → `cache_audio_from_bytes`
- video → `cache_video_from_bytes`
- document → `cache_document_from_bytes`
→ returns a local path.
5b. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
6. The path is attached to the next `message.send` via `media_refs`, becoming
- document → `cache_document_from_bytes`
→ returns a local path.
4. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
5. The path is attached to the next `message.send` via `media_refs`, becoming
`MessageEvent.media_urls` + `media_types`
(`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then
read the file.
**Limits:** `get_inbound_media_max_bytes()` / `validate_inbound_media_size`
(`base.py:758/779`) enforce the cap; over-limit → `error {code:"media_too_large"}`.
(`base.py:758/779`) enforce the cap; over-limit → 413 +
`error {code:"media_too_large"}`. (The 1 MiB `MAX_BODY_BYTES` cap applies to
JSON *frame* bodies only, not media uploads.)
**Backpressure:** large uploads use the WS flow control; the plugin reads
binary frames into a temp file (not memory) to bound RAM.
**Single-shot:** no chunking/resumability — HTTP carries the body; single-user
scale makes a one-shot upload sufficient.
## 7.3 Outbound (agent → app) — `media.offer` / `media.pull`
**Flow:**
1. Agent produces/references media (e.g. generates an image, or replies with a
`MEDIA:` tag / image URL). hermes base `extract_media` / `extract_images`
(`base.py:4439/4884`) pull these out and call the adapter's
@@ -54,14 +61,13 @@ binary frames into a temp file (not memory) to bound RAM.
2. Adapter stages the file in the media cache, mints a `media_id`, and emits
`media.offer {media_id, kind, mime, size, filename}` (inside/with the
`message` frame's `media[]`).
3. App sends `media.pull {media_id}`.
4. Plugin streams the file as **binary WS frames**; ends with
`media.pull.end {ok:true}`.
5. App writes to its cache dir and hands the path to the player/viewer.
3. App `GET /v1/media/{id}` — the full file body.
4. App writes to its cache dir and hands the path to the player/viewer.
**Security:** `validate_media_delivery_path` (`base.py:1684`) + the media
delivery root/recency/denied-path checks (`base.py:1312-1480`) ensure the plugin
only serves files hermes is allowed to deliver (no arbitrary file read).
only serves files hermes is allowed to deliver (no arbitrary file read). The
delivery-path check is re-run **at pull time**, not just at offer time.
## 7.4 Live playback (AI-sent music/video)
@@ -79,14 +85,12 @@ only serves files hermes is allowed to deliver (no arbitrary file read).
a desktop backend (see `11-desktop-app.md`): a `libmpv`/`mpv`-backed surface
or a WebView fallback for video, and a desktop audio player for music.
## 7.5 Chunking parameters
## 7.5 Integrity
- Chunk size: **256 KiB** (tunable).
- Binary frames carry raw bytes only; framing/metadata is in the JSON header +
end frames.
- Reassembly is ordered (WS preserves order); a gap/corruption → abort +
`error {code:"internal"}` + retry the whole transfer.
- `sha256` in `media.upload.end` / a size check on pull verify integrity.
- Upload: `sha256` (precomputed by the app, sent in `X-Iris-Media-Sha256`)
is verified by the plugin; mismatch → `media.upload.ack {ok:false}`.
- Pull: the app checks the received size against the offered `size`.
- A failed transfer → retry the whole upload (single-shot, no resume).
## 7.6 App-side storage
@@ -94,4 +98,4 @@ only serves files hermes is allowed to deliver (no arbitrary file read).
- LRU eviction by size (configurable, default 500 MB) so old media doesn't fill
the device.
- A `MediaRepository` tracks `{media_id, local_path, kind, size, ts}` in Room so
bubbles can re-render players after process death.
bubbles can re-render players after process death.