146 lines
7.2 KiB
Markdown
146 lines
7.2 KiB
Markdown
# 07 — Media (upload, download, playback)
|
|
|
|
Media travels **over HTTP** (`POST /v1/media` for upload,
|
|
`GET /v1/media/{id}` for pull; see `19-http-fallback-transport.md` §19.15).
|
|
HTTP is the only transport — the WebSocket leg (chunked binary frames) was
|
|
removed entirely. The contracts below (kinds, sha256, re-sniffing, delivery
|
|
validation) apply to both directions.
|
|
|
|
## 7.1 Kinds & MIME
|
|
|
|
`kind` ∈ `image | audio | video | document | voice`.
|
|
|
|
- `image` — `image/*` (jpg/png/webp/gif/heic).
|
|
- `audio` — `audio/*` (mp3/m4a/ogg/…) — music.
|
|
- `video` — `video/*` (mp4/webm/mov).
|
|
- `document` — anything else (pdf, docx, zip, txt, …).
|
|
- `voice` — short voice note (`audio/ogg; codecs=opus` typical).
|
|
|
|
The app sniffs `kind` from the picked file's MIME; the plugin re-sniffs on
|
|
receipt (don't trust the client) using hermes helpers
|
|
(`gateway/platforms/base.py` `_sniff_audio_ext`, `_looks_like_image`).
|
|
|
|
## 7.2 Inbound (app → agent) — `media.upload`
|
|
|
|
**Flow:**
|
|
|
|
1. App picks a file (SAF) → reads size + MIME, computes sha256.
|
|
2. App `POST /v1/media` with the raw file body; metadata in
|
|
`X-Iris-Media-*` headers (`media_ref`, `kind`, `mime`, `filename`,
|
|
`sha256`).
|
|
3. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
|
|
cache via hermes `cache_*_from_bytes`:
|
|
- image → `cache_image_from_bytes`
|
|
- audio/voice → `cache_audio_from_bytes`
|
|
- video → `cache_video_from_bytes`
|
|
- document → `cache_document_from_bytes`
|
|
→ returns a local path.
|
|
4. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
|
|
5. The path is attached to the next `message.send` via `media_refs`, becoming
|
|
`MessageEvent.media_urls` + `media_types`
|
|
(`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then
|
|
read the file.
|
|
|
|
**Limits:** two caps apply — the plugin's `max_upload_bytes` (default
|
|
100 MiB) and hermes's `gateway.max_inbound_media_bytes` (default 128 MiB,
|
|
enforced by `get_inbound_media_max_bytes()` / `validate_inbound_media_size`,
|
|
`base.py:758/779`); over-limit → 413 + `error {code:"media_too_large"}`.
|
|
Effective limit is the **min** of both; see §7.7. (The 1 MiB
|
|
`MAX_BODY_BYTES` cap applies to JSON *frame* bodies only, not media uploads.)
|
|
|
|
**Single-shot:** no chunking/resumability — HTTP carries the body; single-user
|
|
scale makes a one-shot upload sufficient.
|
|
|
|
## 7.3 Outbound (agent → app) — `media.offer` / `media.pull`
|
|
|
|
**Flow:**
|
|
|
|
1. Agent produces/references media (e.g. generates an image, or replies with a
|
|
`MEDIA:` tag / image URL). hermes base `extract_media` / `extract_images`
|
|
(`base.py:4439/4884`) pull these out and call the adapter's
|
|
`send_image / send_video / send_document / send_voice / send_image_file /
|
|
send_multiple_images`.
|
|
2. Adapter stages the file in the media cache, mints a `media_id`, and emits
|
|
`media.offer {media_id, kind, mime, size, filename}` (inside/with the
|
|
`message` frame's `media[]`).
|
|
3. App `GET /v1/media/{id}` — the full file body.
|
|
4. App writes to its cache dir and hands the path to the player/viewer.
|
|
|
|
**Security:** `validate_media_delivery_path` (`base.py:1684`) + the media
|
|
delivery root/recency/denied-path checks (`base.py:1312-1480`) ensure the plugin
|
|
only serves files hermes is allowed to deliver (no arbitrary file read). The
|
|
delivery-path check is re-run **at pull time**, not just at offer time.
|
|
|
|
## 7.4 Live playback (AI-sent music/video)
|
|
|
|
**Requirement:** play AI-sent music and video **in the app**.
|
|
|
|
- **Audio (music/voice):** `ExoPlayer` (Media3). Inline player in the bubble
|
|
(play/pause, seek, duration); a persistent **mini-player** for music that
|
|
survives navigation. Voice notes play inline with a waveform.
|
|
- **Video:** `ExoPlayer` inline player (play/pause, seek, fullscreen, PiP on
|
|
Android). Streams from the local cache file after `media.pull`.
|
|
- **Documents/images:** image viewer (zoom) / open-with for documents (Android
|
|
`Intent.ACTION_VIEW` with a `FileProvider` URI; Desktop opens with the system
|
|
handler).
|
|
- **Desktop:** ExoPlayer is Android-only → the `MediaPlayer` expect/actual uses
|
|
a desktop backend (see `11-desktop-app.md`): a `libmpv`/`mpv`-backed surface
|
|
or a WebView fallback for video, and a desktop audio player for music.
|
|
|
|
## 7.5 Integrity
|
|
|
|
- Upload: `sha256` (precomputed by the app, sent in `X-Iris-Media-Sha256`)
|
|
is verified by the plugin; mismatch → `media.upload.ack {ok:false}`.
|
|
- Pull: the app checks the received size against the offered `size`.
|
|
- A failed transfer → retry the whole upload (single-shot, no resume).
|
|
|
|
## 7.6 App-side storage
|
|
|
|
- Cache dir: app-specific external cache (`getExternalCacheDir()/media`,
|
|
internal `cacheDir` fallback; desktop: `~/.iris/cache/media`).
|
|
- LRU eviction by size (hardcoded 500 MB, `MediaCacheJvm.kt` `maxBytes`) so old
|
|
media doesn't fill the device.
|
|
- A `MediaRepository` tracks `{media_id, local_path, kind, size, ts}` in Room so
|
|
bubbles can re-render players after process death.
|
|
|
|
## 7.7 Size limits & where files live
|
|
|
|
**Inbound (app → agent) — two caps, effective limit is the min:**
|
|
|
|
| Cap | Default | Set via | Enforced by |
|
|
| --- | --- | --- | --- |
|
|
| Plugin `max_upload_bytes` | 100 MiB | `gateway.platforms.iris.extra.max_upload_bytes` (config.yaml) | `http_server.py` (Content-Length pre-check) + `media.py` `UploadSession.feed` (mid-stream) → `media_too_large` |
|
|
| Hermes `gateway.max_inbound_media_bytes` | 128 MiB | `gateway.max_inbound_media_bytes` (config.yaml) | `validate_inbound_media_size` inside `cache_*_from_bytes` (`base.py:758/779`) → `media_too_large` |
|
|
|
|
Both are **pure config** — raising the limit needs no code change in the plugin
|
|
or hermes-agent. The app itself has no upload cap.
|
|
|
|
**Why the caps exist:** the upload is streamed to a temp file (disk, bounded
|
|
RAM in flight), but at completion the plugin reads the **entire blob into
|
|
memory** (`UploadSession.read_bytes()` → `cache_*_from_bytes` →
|
|
`write_bytes`), so peak RAM ≈ file size per upload. Hermes's cap exists to
|
|
prevent OOM-killing the gateway (comment at `base.py:740-749`).
|
|
|
|
**Inbound storage (gateway host):**
|
|
|
|
- In flight: temp file `upl_*` under `~/.hermes/iris/media/tmp` (removed after
|
|
completion or failure).
|
|
- After caching: hermes media cache — `~/.hermes/cache/images/img_<uuid12><ext>`,
|
|
`cache/audio/audio_<uuid12><ext>`, `cache/videos/video_<uuid12><ext>`,
|
|
`cache/documents/doc_<uuid12>_<original filename>`. Video/image/audio lose
|
|
their original filename; documents keep it.
|
|
- Lifetime: hermes `cleanup_video_cache(max_age_hours=24)` deletes videos older
|
|
than 24 h.
|
|
|
|
**Outbound (agent → app) — no size cap at the gateway.** `media.offer` carries
|
|
metadata; `GET /v1/media/{id}` streams the full file in 256 KiB chunks
|
|
regardless of size. The only constraints are the delivery-path re-validation at
|
|
pull time and the 24 h offer TTL (`MediaStore.prune_outbound`).
|
|
|
|
**The effective outbound limit is set by the app:** the device media cache is
|
|
LRU-capped at 500 MB (§7.6) and `evict()` runs right after each pull completes.
|
|
A single file > 500 MB makes the eviction loop delete everything — *including
|
|
the file it just downloaded*. So files > ~500 MB arrive but are immediately
|
|
discarded. To retain large agent→app files, raise `maxBytes` in
|
|
`MediaCacheJvm.kt` (or exempt files from eviction).
|