M4: media upload/download/playback (both directions)

Gateway plugin:
- media.upload (chunked binary) -> size/sha256 verify + MIME re-sniff ->
  cache_*_from_bytes -> media.upload.ack
- media.offer / media.pull (chunked) for agent-sent media, delivery-path
  security re-checked at pull time
- send_* overrides mint media_id and emit media.offer
- message.send media_refs resolve to cached inbound media
- per-send + per-chunk timeouts so a stalled peer can't starve the rest

App (Kotlin CMP):
- Protocol: media frame types/payloads/builders
- GatewayClient: binary session, uploadMedia (chunked + streaming sha256),
  pullMedia serialized via Mutex so concurrent offers don't interleave
- ChatStore/IrisController: MediaItem, attachments, auto-pull on offer
- Platform media: SAF picker, ExoPlayer (audio mini-player + video), image
  loader, FileProvider document open (Android); AWT-free desktop actuals
- ChatScreen: attach button + chips, media rendering, keyboard dismiss on send

UI polish:
- preserve image aspect ratio (no stretching), cap dominant dimension
- adjustResize so only chat content squeezes for the keyboard
- clear focus (hide keyboard) on send

Docs: media.upload.ack in 04-wire-protocol.md + frames.schema.json +
07-media.md; M4 marked complete in 14-milestones.md.

Tests: 17-test tests/gateway/test_android.py suite passes.
This commit is contained in:
ARIA committed 2026-08-19 17:29:39 +02:00
1 parent 60296b33fe
commit 913ee91024
28 files changed
+2234 -53

No files matched your search

+7
View File
@@ -227,6 +227,13 @@ See `07-media.md`.
{"type":"media.upload.end","id":11,"payload":{"media_ref":"mu_1","sha256":"…"}}
```
### `media.upload.ack`
Server → App response to `media.upload.end`: the ref is cached and may now be
referenced in a `message.send` `media_refs`. Failures use `error` frames instead.
```json
{"type":"media.upload.ack","id":11,"payload":{"ok":true,"media_ref":"mu_1"}}
```
### `media.pull`
Request agent-sent media bytes.
```json
+4 -3
View File
@@ -29,9 +29,10 @@ receipt (don't trust the client) using hermes helpers
- image → `cache_image_from_bytes`
- audio/voice → `cache_audio_from_bytes`
- video → `cache_video_from_bytes`
- document → `cache_document_from_bytes`
→ returns a local path.
6. The path is attached to the next `message.send` via `media_refs`, becoming
- document → `cache_document_from_bytes`
→ returns a local path.
5b. Plugin replies `media.upload.ack {ok, media_ref}` (failures use `error`).
6. The path is attached to the next `message.send` via `media_refs`, becoming
`MessageEvent.media_urls` + `media_types`
(`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then
read the file.
+17 -3
View File
@@ -90,17 +90,31 @@ has explicit **acceptance criteria**. Work top-to-bottom; don't skip M0/M1.
## M4 — Media
**Goal:** attach + receive + play media.
- [ ] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
- [X] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
size limit + sha256 + MIME re-sniff.
- [ ] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
- [X] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
security.
- [ ] App: SAF pickers + preview chips + upload; `media.pull` → cache;
- [X] App: SAF pickers + preview chips + upload; `media.pull` → cache;
**ExoPlayer** inline (audio mini-player, video fullscreen/PiP); image/doc
viewers.
- **Demo (on-device):** attach a photo + video (agent sees them); ask agent to
send an image/video → plays live in-app.
- **Accept:** both directions work; over-limit rejected; playback is live;
only allowed files are servable.
- **Status (complete):** Both directions verified end-to-end on the MIX 2S.
Inbound: SAF-picked photo → chunked binary `media.upload` (256 KiB) →
gateway size+sha256 verify + MIME re-sniff → `cache_image_from_bytes` →
`media.upload.ack` → `message.send` with `media_refs` → agent vision
described the image accurately. Outbound: agent `send_image` → `media.offer`
→ app auto-`media.pull` (chunked) → cache → image rendered inline. Live
playback verified on-device: agent offered a 2s MP4 + 2s MP3 → app pulled
both → ExoPlayer video player (blue frame, 00:02/00:02, controls) + audio
mini-player rendered and playable. Over-limit rejection, sha256 mismatch,
and delivery-path security are covered by the 17-test
`tests/gateway/test_android.py` suite (all pass). Note: the app serializes
pulls (single `binarySession` slot) via a `Mutex` so concurrent offers don't
interleave their byte streams. `media.upload.ack` documented in
`04-wire-protocol.md` + `frames.schema.json`.
## M5 — Push + offline (FCM + ntfy)
**Goal:** reach the phone when backgrounded; catch up on reconnect.
+2 -1
View File
@@ -68,7 +68,8 @@
"error": { "payload": { "code": { "type": "string", "enum": ["auth", "not_found", "rate_limited", "media_too_large", "unsupported", "internal"] }, "message": { "type": "string" } } },
"pong": { "payload": { "ts": { "type": "integer" } } },
"sync.done": { "payload": { "cursor": { "type": "integer" } } },
"media.pull.end": { "payload": { "ok": { "type": "boolean" } } }
"media.pull.end": { "payload": { "ok": { "type": "boolean" } } },
"media.upload.ack": { "description": "Response to media.upload.end; ref is cached and usable in message.send media_refs.", "payload": { "ok": { "type": "boolean" }, "media_ref": { "type": "string" } } }
},
"app_to_server": {
"hello": { "description": "First frame; auth + caps.", "payload": { "token": { "type": "string" }, "device_id": { "type": "string" }, "device_name": { "type": "string" }, "caps": { "type": "object", "properties": { "min_protocol": {"type":"integer"}, "media": {"type":"boolean"}, "push": {"type":"string"} } }, "fcm_token": { "type": "string" }, "ntfy_topic": { "type": "string" } } },