07 — Media (upload, download, playback)

Media travels over HTTP (POST /v1/media for upload, GET /v1/media/{id} for pull; see 19-http-fallback-transport.md §19.15). HTTP is the only transport — the WebSocket leg (chunked binary frames) was removed entirely. The contracts below (kinds, sha256, re-sniffing, delivery validation) apply to both directions.

7.1 Kinds & MIME

kind ∈ image | audio | video | document | voice.

image — image/* (jpg/png/webp/gif/heic).
audio — audio/* (mp3/m4a/ogg/…) — music.
video — video/* (mp4/webm/mov).
document — anything else (pdf, docx, zip, txt, …).
voice — short voice note (audio/ogg; codecs=opus typical).

The app sniffs kind from the picked file's MIME; the plugin re-sniffs on receipt (don't trust the client) using hermes helpers (gateway/platforms/base.py _sniff_audio_ext, _looks_like_image).

7.2 Inbound (app → agent) — `media.upload`

Flow:

App picks a file (SAF) → reads size + MIME, computes sha256.
App POST /v1/media with the raw file body; metadata in X-Iris-Media-* headers (media_ref, kind, mime, filename, sha256).
Plugin verifies size ≤ max_upload_bytes and sha256, writes to the media cache via hermes cache_*_from_bytes:
- image → cache_image_from_bytes
- audio/voice → cache_audio_from_bytes
- video → cache_video_from_bytes
- document → cache_document_from_bytes → returns a local path.
Plugin replies media.upload.ack {ok, media_ref} (failures use error).
The path is attached to the next message.send via media_refs, becoming MessageEvent.media_urls + media_types (gateway/platforms/base.py:2337). The agent's vision/audio tools can then read the file.

Limits: two caps apply — the plugin's max_upload_bytes (default 100 MiB) and hermes's gateway.max_inbound_media_bytes (default 128 MiB, enforced by get_inbound_media_max_bytes() / validate_inbound_media_size, base.py:758/779); over-limit → 413 + error {code:"media_too_large"}. Effective limit is the min of both; see §7.7. (The 1 MiB MAX_BODY_BYTES cap applies to JSON frame bodies only, not media uploads.)

Single-shot: no chunking/resumability — HTTP carries the body; single-user scale makes a one-shot upload sufficient.

7.3 Outbound (agent → app) — `media.offer` / `media.pull`

Flow:

Agent produces/references media (e.g. generates an image, or replies with a MEDIA: tag / image URL). hermes base extract_media / extract_images (base.py:4439/4884) pull these out and call the adapter's send_image / send_video / send_document / send_voice / send_image_file / send_multiple_images.
Adapter stages the file in the media cache, mints a media_id, and emits media.offer {media_id, kind, mime, size, filename} (inside/with the message frame's media[]).
App GET /v1/media/{id} — the full file body.
App writes to its cache dir and hands the path to the player/viewer.

Security: validate_media_delivery_path (base.py:1684) + the media delivery root/recency/denied-path checks (base.py:1312-1480) ensure the plugin only serves files hermes is allowed to deliver (no arbitrary file read). The delivery-path check is re-run at pull time, not just at offer time.

7.4 Live playback (AI-sent music/video)

Requirement: play AI-sent music and video in the app.

Audio (music/voice): ExoPlayer (Media3). Inline player in the bubble (play/pause, seek, duration); a persistent mini-player for music that survives navigation. Voice notes play inline with a waveform.
Video: ExoPlayer inline player (play/pause, seek, fullscreen, PiP on Android). Streams from the local cache file after media.pull.
Documents/images: image viewer (zoom) / open-with for documents (Android Intent.ACTION_VIEW with a FileProvider URI; Desktop opens with the system handler).
Desktop: ExoPlayer is Android-only → the MediaPlayer expect/actual uses a desktop backend (see 11-desktop-app.md): a libmpv/mpv-backed surface or a WebView fallback for video, and a desktop audio player for music.

7.5 Integrity

Upload: sha256 (precomputed by the app, sent in X-Iris-Media-Sha256) is verified by the plugin; mismatch → media.upload.ack {ok:false}.
Pull: the app checks the received size against the offered size.
A failed transfer → retry the whole upload (single-shot, no resume).

7.6 App-side storage

Cache dir: app-specific external cache (getExternalCacheDir()/media, internal cacheDir fallback; desktop: ~/.iris/cache/media).
LRU eviction by size (hardcoded 500 MB, MediaCacheJvm.kt maxBytes) so old media doesn't fill the device.
A MediaRepository tracks {media_id, local_path, kind, size, ts} in Room so bubbles can re-render players after process death.

7.7 Size limits & where files live

Inbound (app → agent) — two caps, effective limit is the min:

Cap	Default	Set via	Enforced by
Plugin `max_upload_bytes`	100 MiB	`gateway.platforms.iris.extra.max_upload_bytes` (config.yaml)	`http_server.py` (Content-Length pre-check) + `media.py` `UploadSession.feed` (mid-stream) → `media_too_large`
Hermes `gateway.max_inbound_media_bytes`	128 MiB	`gateway.max_inbound_media_bytes` (config.yaml)	`validate_inbound_media_size` inside `cache_*_from_bytes` (`base.py:758/779`) → `media_too_large`

Both are pure config — raising the limit needs no code change in the plugin or hermes-agent. The app itself has no upload cap.

Why the caps exist: the upload is streamed to a temp file (disk, bounded RAM in flight), but at completion the plugin reads the entire blob into memory (UploadSession.read_bytes() → cache_*_from_bytes → write_bytes), so peak RAM ≈ file size per upload. Hermes's cap exists to prevent OOM-killing the gateway (comment at base.py:740-749).

Inbound storage (gateway host):

In flight: temp file upl_* under ~/.hermes/iris/media/tmp (removed after completion or failure).
After caching: hermes media cache — ~/.hermes/cache/images/img_<uuid12><ext>, cache/audio/audio_<uuid12><ext>, cache/videos/video_<uuid12><ext>, cache/documents/doc_<uuid12>_<original filename>. Video/image/audio lose their original filename; documents keep it.
Lifetime: hermes cleanup_video_cache(max_age_hours=24) deletes videos older than 24 h.

Outbound (agent → app) — no size cap at the gateway. media.offer carries metadata; GET /v1/media/{id} streams the full file in 256 KiB chunks regardless of size. The only constraints are the delivery-path re-validation at pull time and the 24 h offer TTL (MediaStore.prune_outbound).

The effective outbound limit is set by the app: the device media cache is LRU-capped at 500 MB (§7.6) and evict() runs right after each pull completes. A single file > 500 MB makes the eviction loop delete everything — including the file it just downloaded. So files > ~500 MB arrive but are immediately discarded. To retain large agent→app files, raise maxBytes in MediaCacheJvm.kt (or exempt files from eviction).

7.2 KiB Raw Blame History