ARIA/iris_x_hermes

0 Fork 0

Files

T

ARIA e6015033b6

CI / Gateway plugin tests (push) Successful in 5m9s

Details

CI / Kotlin tests (android host + desktop) (push) Successful in 6m55s

Details

HTTP transport: drop WS server, offline send queue + dead-stream watchdog

Gateway (docs/19):
- Remove ws_server.py; frame dispatch factored into dispatch.py
- http_server: media upload/pull, pairing over HTTP
- protocol: media frames mirrored; tests + ws_probe updated for HTTP

App:
- HttpGateway: postFrame/uploadMedia/pullMedia no longer throw on
  network failure (PostResult ok=false / Result.failure) — uncaught
  SocketTimeoutException on Dispatchers.Default crashed the app
- GatewayClient: dead-stream watchdog (health probe every 10s, 2
  failures -> redial in ~20s instead of the 45s SSE read timeout);
  state flips to Reconnecting when the stream dies, restored from the
  last hello.ack on long-poll success; poke() + backoff reset on app
  resume (MainActivity.onResume)
- Offline sends: composer enabled while disconnected; a send with no
  response (status 0) stays queued (Pending) and is auto-resent on the
  next (re)connect after a 2s outbox-replay grace; gateway 4xx
  rejections fail the bubble (tap to retry, no auto-loop)
- ChatStore: echo-replace and thread-relocate also match Failed
  bubbles (POST response lost in a network drop); loadHistory dedupes
  local failed bubbles the server already has; failMessage()
- MainActivity: poke() on resume so a backgrounded app reconnects
  promptly instead of waiting out the backoff

2026-08-22 20:10:05 +02:00

4.6 KiB

Raw Blame History

07 — Media (upload, download, playback)

Media travels over HTTP (POST /v1/media for upload, GET /v1/media/{id} for pull; see 19-http-fallback-transport.md §19.15). HTTP is the only transport — the WebSocket leg (chunked binary frames) was removed entirely. The contracts below (kinds, sha256, re-sniffing, delivery validation) apply to both directions.

7.1 Kinds & MIME

kind ∈ image | audio | video | document | voice.

image — image/* (jpg/png/webp/gif/heic).
audio — audio/* (mp3/m4a/ogg/…) — music.
video — video/* (mp4/webm/mov).
document — anything else (pdf, docx, zip, txt, …).
voice — short voice note (audio/ogg; codecs=opus typical).

The app sniffs kind from the picked file's MIME; the plugin re-sniffs on receipt (don't trust the client) using hermes helpers (gateway/platforms/base.py _sniff_audio_ext, _looks_like_image).

7.2 Inbound (app → agent) — `media.upload`

Flow:

App picks a file (SAF) → reads size + MIME, computes sha256.
App POST /v1/media with the raw file body; metadata in X-Iris-Media-* headers (media_ref, kind, mime, filename, sha256).
Plugin verifies size ≤ max_upload_bytes and sha256, writes to the media cache via hermes cache_*_from_bytes:
- image → cache_image_from_bytes
- audio/voice → cache_audio_from_bytes
- video → cache_video_from_bytes
- document → cache_document_from_bytes → returns a local path.
Plugin replies media.upload.ack {ok, media_ref} (failures use error).
The path is attached to the next message.send via media_refs, becoming MessageEvent.media_urls + media_types (gateway/platforms/base.py:2337). The agent's vision/audio tools can then read the file.

Limits: get_inbound_media_max_bytes() / validate_inbound_media_size (base.py:758/779) enforce the cap; over-limit → 413 + error {code:"media_too_large"}. (The 1 MiB MAX_BODY_BYTES cap applies to JSON frame bodies only, not media uploads.)

Single-shot: no chunking/resumability — HTTP carries the body; single-user scale makes a one-shot upload sufficient.

7.3 Outbound (agent → app) — `media.offer` / `media.pull`

Flow:

Agent produces/references media (e.g. generates an image, or replies with a MEDIA: tag / image URL). hermes base extract_media / extract_images (base.py:4439/4884) pull these out and call the adapter's send_image / send_video / send_document / send_voice / send_image_file / send_multiple_images.
Adapter stages the file in the media cache, mints a media_id, and emits media.offer {media_id, kind, mime, size, filename} (inside/with the message frame's media[]).
App GET /v1/media/{id} — the full file body.
App writes to its cache dir and hands the path to the player/viewer.

Security: validate_media_delivery_path (base.py:1684) + the media delivery root/recency/denied-path checks (base.py:1312-1480) ensure the plugin only serves files hermes is allowed to deliver (no arbitrary file read). The delivery-path check is re-run at pull time, not just at offer time.

7.4 Live playback (AI-sent music/video)

Requirement: play AI-sent music and video in the app.

Audio (music/voice): ExoPlayer (Media3). Inline player in the bubble (play/pause, seek, duration); a persistent mini-player for music that survives navigation. Voice notes play inline with a waveform.
Video: ExoPlayer inline player (play/pause, seek, fullscreen, PiP on Android). Streams from the local cache file after media.pull.
Documents/images: image viewer (zoom) / open-with for documents (Android Intent.ACTION_VIEW with a FileProvider URI; Desktop opens with the system handler).
Desktop: ExoPlayer is Android-only → the MediaPlayer expect/actual uses a desktop backend (see 11-desktop-app.md): a libmpv/mpv-backed surface or a WebView fallback for video, and a desktop audio player for music.

7.5 Integrity

Upload: sha256 (precomputed by the app, sent in X-Iris-Media-Sha256) is verified by the plugin; mismatch → media.upload.ack {ok:false}.
Pull: the app checks the received size against the offered size.
A failed transfer → retry the whole upload (single-shot, no resume).

7.6 App-side storage

Cache dir: app-specific external cache (getExternalCacheDir()/media).
LRU eviction by size (configurable, default 500 MB) so old media doesn't fill the device.
A MediaRepository tracks {media_id, local_path, kind, size, ts} in Room so bubbles can re-render players after process death.

4.6 KiB Raw Blame History