M4: media upload/download/playback (both directions)
Gateway plugin: - media.upload (chunked binary) -> size/sha256 verify + MIME re-sniff -> cache_*_from_bytes -> media.upload.ack - media.offer / media.pull (chunked) for agent-sent media, delivery-path security re-checked at pull time - send_* overrides mint media_id and emit media.offer - message.send media_refs resolve to cached inbound media - per-send + per-chunk timeouts so a stalled peer can't starve the rest App (Kotlin CMP): - Protocol: media frame types/payloads/builders - GatewayClient: binary session, uploadMedia (chunked + streaming sha256), pullMedia serialized via Mutex so concurrent offers don't interleave - ChatStore/IrisController: MediaItem, attachments, auto-pull on offer - Platform media: SAF picker, ExoPlayer (audio mini-player + video), image loader, FileProvider document open (Android); AWT-free desktop actuals - ChatScreen: attach button + chips, media rendering, keyboard dismiss on send UI polish: - preserve image aspect ratio (no stretching), cap dominant dimension - adjustResize so only chat content squeezes for the keyboard - clear focus (hide keyboard) on send Docs: media.upload.ack in 04-wire-protocol.md + frames.schema.json + 07-media.md; M4 marked complete in 14-milestones.md. Tests: 17-test tests/gateway/test_android.py suite passes.
This commit is contained in:
1 parent
60296b33fe
commit
913ee91024
28 files changed
+2234
-53
No files matched your search
+17
-3
@@ -90,17 +90,31 @@ has explicit **acceptance criteria**. Work top-to-bottom; don't skip M0/M1.
|
||||
|
||||
## M4 — Media
|
||||
**Goal:** attach + receive + play media.
|
||||
- [ ] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
|
||||
- [X] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
|
||||
size limit + sha256 + MIME re-sniff.
|
||||
- [ ] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
|
||||
- [X] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
|
||||
security.
|
||||
- [ ] App: SAF pickers + preview chips + upload; `media.pull` → cache;
|
||||
- [X] App: SAF pickers + preview chips + upload; `media.pull` → cache;
|
||||
**ExoPlayer** inline (audio mini-player, video fullscreen/PiP); image/doc
|
||||
viewers.
|
||||
- **Demo (on-device):** attach a photo + video (agent sees them); ask agent to
|
||||
send an image/video → plays live in-app.
|
||||
- **Accept:** both directions work; over-limit rejected; playback is live;
|
||||
only allowed files are servable.
|
||||
- **Status (complete):** Both directions verified end-to-end on the MIX 2S.
|
||||
Inbound: SAF-picked photo → chunked binary `media.upload` (256 KiB) →
|
||||
gateway size+sha256 verify + MIME re-sniff → `cache_image_from_bytes` →
|
||||
`media.upload.ack` → `message.send` with `media_refs` → agent vision
|
||||
described the image accurately. Outbound: agent `send_image` → `media.offer`
|
||||
→ app auto-`media.pull` (chunked) → cache → image rendered inline. Live
|
||||
playback verified on-device: agent offered a 2s MP4 + 2s MP3 → app pulled
|
||||
both → ExoPlayer video player (blue frame, 00:02/00:02, controls) + audio
|
||||
mini-player rendered and playable. Over-limit rejection, sha256 mismatch,
|
||||
and delivery-path security are covered by the 17-test
|
||||
`tests/gateway/test_android.py` suite (all pass). Note: the app serializes
|
||||
pulls (single `binarySession` slot) via a `Mutex` so concurrent offers don't
|
||||
interleave their byte streams. `media.upload.ack` documented in
|
||||
`04-wire-protocol.md` + `frames.schema.json`.
|
||||
|
||||
## M5 — Push + offline (FCM + ntfy)
|
||||
**Goal:** reach the phone when backgrounded; catch up on reconnect.
|
||||
|
||||
Reference in new issue
Block a user