ARIA
913ee91024
Gateway plugin: - media.upload (chunked binary) -> size/sha256 verify + MIME re-sniff -> cache_*_from_bytes -> media.upload.ack - media.offer / media.pull (chunked) for agent-sent media, delivery-path security re-checked at pull time - send_* overrides mint media_id and emit media.offer - message.send media_refs resolve to cached inbound media - per-send + per-chunk timeouts so a stalled peer can't starve the rest App (Kotlin CMP): - Protocol: media frame types/payloads/builders - GatewayClient: binary session, uploadMedia (chunked + streaming sha256), pullMedia serialized via Mutex so concurrent offers don't interleave - ChatStore/IrisController: MediaItem, attachments, auto-pull on offer - Platform media: SAF picker, ExoPlayer (audio mini-player + video), image loader, FileProvider document open (Android); AWT-free desktop actuals - ChatScreen: attach button + chips, media rendering, keyboard dismiss on send UI polish: - preserve image aspect ratio (no stretching), cap dominant dimension - adjustResize so only chat content squeezes for the keyboard - clear focus (hide keyboard) on send Docs: media.upload.ack in 04-wire-protocol.md + frames.schema.json + 07-media.md; M4 marked complete in 14-milestones.md. Tests: 17-test tests/gateway/test_android.py suite passes.
97 lines
4.5 KiB
Markdown
97 lines
4.5 KiB
Markdown
# 07 — Media (upload, download, playback)
|
|
|
|
Media travels **over the WebSocket** as chunked binary frames (decision: no
|
|
separate HTTP server; keeps the plugin to `websockets` only). Both directions
|
|
use the same chunking.
|
|
|
|
## 7.1 Kinds & MIME
|
|
|
|
`kind` ∈ `image | audio | video | document | voice`.
|
|
- `image` — `image/*` (jpg/png/webp/gif/heic).
|
|
- `audio` — `audio/*` (mp3/m4a/ogg/…) — music.
|
|
- `video` — `video/*` (mp4/webm/mov).
|
|
- `document` — anything else (pdf, docx, zip, txt, …).
|
|
- `voice` — short voice note (`audio/ogg; codecs=opus` typical).
|
|
|
|
The app sniffs `kind` from the picked file's MIME; the plugin re-sniffs on
|
|
receipt (don't trust the client) using hermes helpers
|
|
(`gateway/platforms/base.py` `_sniff_audio_ext`, `_looks_like_image`).
|
|
|
|
## 7.2 Inbound (app → agent) — `media.upload`
|
|
|
|
**Flow:**
|
|
1. App picks a file (SAF) → reads size + MIME.
|
|
2. App sends `media.upload.start {media_ref, kind, mime, size, filename}`.
|
|
3. App streams the file as **binary WS frames** (e.g. 256 KiB chunks).
|
|
4. App sends `media.upload.end {media_ref, sha256}`.
|
|
5. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
|
|
cache via hermes `cache_*_from_bytes`:
|
|
- image → `cache_image_from_bytes`
|
|
- audio/voice → `cache_audio_from_bytes`
|
|
- video → `cache_video_from_bytes`
|
|
- document → `cache_document_from_bytes`
|
|
→ returns a local path.
|
|
5b. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
|
|
6. The path is attached to the next `message.send` via `media_refs`, becoming
|
|
`MessageEvent.media_urls` + `media_types`
|
|
(`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then
|
|
read the file.
|
|
|
|
**Limits:** `get_inbound_media_max_bytes()` / `validate_inbound_media_size`
|
|
(`base.py:758/779`) enforce the cap; over-limit → `error {code:"media_too_large"}`.
|
|
|
|
**Backpressure:** large uploads use the WS flow control; the plugin reads
|
|
binary frames into a temp file (not memory) to bound RAM.
|
|
|
|
## 7.3 Outbound (agent → app) — `media.offer` / `media.pull`
|
|
|
|
**Flow:**
|
|
1. Agent produces/references media (e.g. generates an image, or replies with a
|
|
`MEDIA:` tag / image URL). hermes base `extract_media` / `extract_images`
|
|
(`base.py:4439/4884`) pull these out and call the adapter's
|
|
`send_image / send_video / send_document / send_voice / send_image_file /
|
|
send_multiple_images`.
|
|
2. Adapter stages the file in the media cache, mints a `media_id`, and emits
|
|
`media.offer {media_id, kind, mime, size, filename}` (inside/with the
|
|
`message` frame's `media[]`).
|
|
3. App sends `media.pull {media_id}`.
|
|
4. Plugin streams the file as **binary WS frames**; ends with
|
|
`media.pull.end {ok:true}`.
|
|
5. App writes to its cache dir and hands the path to the player/viewer.
|
|
|
|
**Security:** `validate_media_delivery_path` (`base.py:1684`) + the media
|
|
delivery root/recency/denied-path checks (`base.py:1312-1480`) ensure the plugin
|
|
only serves files hermes is allowed to deliver (no arbitrary file read).
|
|
|
|
## 7.4 Live playback (AI-sent music/video)
|
|
|
|
**Requirement:** play AI-sent music and video **in the app**.
|
|
|
|
- **Audio (music/voice):** `ExoPlayer` (Media3). Inline player in the bubble
|
|
(play/pause, seek, duration); a persistent **mini-player** for music that
|
|
survives navigation. Voice notes play inline with a waveform.
|
|
- **Video:** `ExoPlayer` inline player (play/pause, seek, fullscreen, PiP on
|
|
Android). Streams from the local cache file after `media.pull`.
|
|
- **Documents/images:** image viewer (zoom) / open-with for documents (Android
|
|
`Intent.ACTION_VIEW` with a `FileProvider` URI; Desktop opens with the system
|
|
handler).
|
|
- **Desktop:** ExoPlayer is Android-only → the `MediaPlayer` expect/actual uses
|
|
a desktop backend (see `11-desktop-app.md`): a `libmpv`/`mpv`-backed surface
|
|
or a WebView fallback for video, and a desktop audio player for music.
|
|
|
|
## 7.5 Chunking parameters
|
|
|
|
- Chunk size: **256 KiB** (tunable).
|
|
- Binary frames carry raw bytes only; framing/metadata is in the JSON header +
|
|
end frames.
|
|
- Reassembly is ordered (WS preserves order); a gap/corruption → abort +
|
|
`error {code:"internal"}` + retry the whole transfer.
|
|
- `sha256` in `media.upload.end` / a size check on pull verify integrity.
|
|
|
|
## 7.6 App-side storage
|
|
|
|
- Cache dir: app-specific external cache (`getExternalCacheDir()/media`).
|
|
- LRU eviction by size (configurable, default 500 MB) so old media doesn't fill
|
|
the device.
|
|
- A `MediaRepository` tracks `{media_id, local_path, kind, size, ts}` in Room so
|
|
bubbles can re-render players after process death. |