0 Fork 0

Files

T

ARIA 913ee91024 M4: media upload/download/playback (both directions)

Gateway plugin:
- media.upload (chunked binary) -> size/sha256 verify + MIME re-sniff ->
  cache_*_from_bytes -> media.upload.ack
- media.offer / media.pull (chunked) for agent-sent media, delivery-path
  security re-checked at pull time
- send_* overrides mint media_id and emit media.offer
- message.send media_refs resolve to cached inbound media
- per-send + per-chunk timeouts so a stalled peer can't starve the rest

App (Kotlin CMP):
- Protocol: media frame types/payloads/builders
- GatewayClient: binary session, uploadMedia (chunked + streaming sha256),
  pullMedia serialized via Mutex so concurrent offers don't interleave
- ChatStore/IrisController: MediaItem, attachments, auto-pull on offer
- Platform media: SAF picker, ExoPlayer (audio mini-player + video), image
  loader, FileProvider document open (Android); AWT-free desktop actuals
- ChatScreen: attach button + chips, media rendering, keyboard dismiss on send

UI polish:
- preserve image aspect ratio (no stretching), cap dominant dimension
- adjustResize so only chat content squeezes for the keyboard
- clear focus (hide keyboard) on send

Docs: media.upload.ack in 04-wire-protocol.md + frames.schema.json +
07-media.md; M4 marked complete in 14-milestones.md.

Tests: 17-test tests/gateway/test_android.py suite passes.

2026-08-19 17:29:39 +02:00

4.5 KiB

Raw Blame History

07 — Media (upload, download, playback)

Media travels over the WebSocket as chunked binary frames (decision: no separate HTTP server; keeps the plugin to websockets only). Both directions use the same chunking.

7.1 Kinds & MIME

kind ∈ image | audio | video | document | voice.

image — image/* (jpg/png/webp/gif/heic).
audio — audio/* (mp3/m4a/ogg/…) — music.
video — video/* (mp4/webm/mov).
document — anything else (pdf, docx, zip, txt, …).
voice — short voice note (audio/ogg; codecs=opus typical).

The app sniffs kind from the picked file's MIME; the plugin re-sniffs on receipt (don't trust the client) using hermes helpers (gateway/platforms/base.py _sniff_audio_ext, _looks_like_image).

7.2 Inbound (app → agent) — `media.upload`

Flow:

App picks a file (SAF) → reads size + MIME.
App sends media.upload.start {media_ref, kind, mime, size, filename}.
App streams the file as binary WS frames (e.g. 256 KiB chunks).
App sends media.upload.end {media_ref, sha256}.
Plugin verifies size ≤ max_upload_bytes and sha256, writes to the media cache via hermes cache_*_from_bytes:
- image → cache_image_from_bytes
- audio/voice → cache_audio_from_bytes
- video → cache_video_from_bytes

document → cache_document_from_bytes → returns a local path. 5b. Plugin replies media.upload.ack {ok, media_ref} (failures use error).

The path is attached to the next message.send via media_refs, becoming MessageEvent.media_urls + media_types (gateway/platforms/base.py:2337). The agent's vision/audio tools can then read the file.

Limits: get_inbound_media_max_bytes() / validate_inbound_media_size (base.py:758/779) enforce the cap; over-limit → error {code:"media_too_large"}.

Backpressure: large uploads use the WS flow control; the plugin reads binary frames into a temp file (not memory) to bound RAM.

7.3 Outbound (agent → app) — `media.offer` / `media.pull`

Flow:

Agent produces/references media (e.g. generates an image, or replies with a MEDIA: tag / image URL). hermes base extract_media / extract_images (base.py:4439/4884) pull these out and call the adapter's send_image / send_video / send_document / send_voice / send_image_file / send_multiple_images.
Adapter stages the file in the media cache, mints a media_id, and emits media.offer {media_id, kind, mime, size, filename} (inside/with the message frame's media[]).
App sends media.pull {media_id}.
Plugin streams the file as binary WS frames; ends with media.pull.end {ok:true}.
App writes to its cache dir and hands the path to the player/viewer.

Security: validate_media_delivery_path (base.py:1684) + the media delivery root/recency/denied-path checks (base.py:1312-1480) ensure the plugin only serves files hermes is allowed to deliver (no arbitrary file read).

7.4 Live playback (AI-sent music/video)

Requirement: play AI-sent music and video in the app.

Audio (music/voice): ExoPlayer (Media3). Inline player in the bubble (play/pause, seek, duration); a persistent mini-player for music that survives navigation. Voice notes play inline with a waveform.
Video: ExoPlayer inline player (play/pause, seek, fullscreen, PiP on Android). Streams from the local cache file after media.pull.
Documents/images: image viewer (zoom) / open-with for documents (Android Intent.ACTION_VIEW with a FileProvider URI; Desktop opens with the system handler).
Desktop: ExoPlayer is Android-only → the MediaPlayer expect/actual uses a desktop backend (see 11-desktop-app.md): a libmpv/mpv-backed surface or a WebView fallback for video, and a desktop audio player for music.

7.5 Chunking parameters

Chunk size: 256 KiB (tunable).
Binary frames carry raw bytes only; framing/metadata is in the JSON header + end frames.
Reassembly is ordered (WS preserves order); a gap/corruption → abort + error {code:"internal"} + retry the whole transfer.
sha256 in media.upload.end / a size check on pull verify integrity.

7.6 App-side storage

Cache dir: app-specific external cache (getExternalCacheDir()/media).
LRU eviction by size (configurable, default 500 MB) so old media doesn't fill the device.
A MediaRepository tracks {media_id, local_path, kind, size, ts} in Room so bubbles can re-render players after process death.

4.5 KiB Raw Blame History