ARIA/iris_x_hermes
1
0 Fork 0
Code Issues 2 Pull Requests Actions Packages Projects 1 Releases 4 Wiki Activity

M4: media upload/download/playback (both directions)

Browse Source 
Gateway plugin:
- media.upload (chunked binary) -> size/sha256 verify + MIME re-sniff ->
  cache_*_from_bytes -> media.upload.ack
- media.offer / media.pull (chunked) for agent-sent media, delivery-path
  security re-checked at pull time
- send_* overrides mint media_id and emit media.offer
- message.send media_refs resolve to cached inbound media
- per-send + per-chunk timeouts so a stalled peer can't starve the rest

App (Kotlin CMP):
- Protocol: media frame types/payloads/builders
- GatewayClient: binary session, uploadMedia (chunked + streaming sha256),
  pullMedia serialized via Mutex so concurrent offers don't interleave
- ChatStore/IrisController: MediaItem, attachments, auto-pull on offer
- Platform media: SAF picker, ExoPlayer (audio mini-player + video), image
  loader, FileProvider document open (Android); AWT-free desktop actuals
- ChatScreen: attach button + chips, media rendering, keyboard dismiss on send

UI polish:
- preserve image aspect ratio (no stretching), cap dominant dimension
- adjustResize so only chat content squeezes for the keyboard
- clear focus (hide keyboard) on send

Docs: media.upload.ack in 04-wire-protocol.md + frames.schema.json +
07-media.md; M4 marked complete in 14-milestones.md.

Tests: 17-test tests/gateway/test_android.py suite passes.
This commit is contained in:
ARIA committed 2026-08-19 17:29:39 +02:00
1 parent 60296b33fe
commit 913ee91024
28 files changed
+2234 -53
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

gateway-plugin/adapter.py
+318 -9
View File
@@ -18,8 +18,17 @@ Milestone M2: agent transparency. ``send()``/``edit_message()`` are mapped to
``message.start``/``message.update``/``message.stop`` (streaming), tool
progress is classified into structured ``tool.start``/``tool.end`` frames,
interim commentary becomes ``commentary`` frames, and the code-style
reasoning prefix is split into a ``reasoning`` field. Media, outbox, push,
and search land in later milestones (see ``docs/14-milestones.md``).
reasoning prefix is split into a ``reasoning`` field. Outbox and search land
in M3; media, push, and desktop land in later milestones (see
``docs/14-milestones.md``).
Milestone M4: media. Inbound ``media.upload`` (chunked binary frames) is
reassembled in a temp file, verified (size + sha256), re-sniffed, and cached
via hermes ``cache_*_from_bytes``; the resulting refs attach to the next
``message.send`` as ``MessageEvent.media_urls``. Outbound ``send_*`` calls
register the (delivery-validated) file in the media registry and emit
``media.offer``; ``media.pull`` streams the file back as chunked binary
frames, re-checking ``validate_media_delivery_path`` at pull time.
Configuration in config.yaml::
@@ -87,10 +96,12 @@ from gateway.platforms.base import ( # noqa: E402
SendResult,
MessageEvent,
MessageType,
validate_media_delivery_path,
)
from gateway.config import Platform # noqa: E402
from hermes_constants import get_hermes_home # noqa: E402
from . import media as media_bridge # noqa: E402
from . import protocol # noqa: E402
from . import search as search_bridge # noqa: E402
from .channels import get_directory # noqa: E402
@@ -678,6 +689,10 @@ class AndroidAdapter(BasePlatformAdapter):
get_hermes_home() / "android" / "outbox.db",
retention_hours=self.outbox_retention_hours,
)
# M4: media registry (inbound upload refs + outbound offers) and the
# last finalized assistant message id per chat (offer association).
self._media = media_bridge.MediaStore(get_hermes_home())
self._last_message_id: Dict[str, str] = {}
def _turn_state(self, chat_id: str) -> _TurnState:
st = self._turns.get(chat_id)
@@ -841,6 +856,8 @@ class AndroidAdapter(BasePlatformAdapter):
ts=int(time.time() * 1000),
),
)
# M4: media offers emitted after this final associate with it.
self._last_message_id[chat_id] = message_id
await self._close_open_tool(chat_id, state, thread_id)
self._reset_tool_state(state)
state.active = False
@@ -901,6 +918,8 @@ class AndroidAdapter(BasePlatformAdapter):
ts=int(time.time() * 1000),
),
)
# M4: media offers emitted after this final associate with it.
self._last_message_id[chat_id] = message_id
await self._close_open_tool(chat_id, state, thread_id)
self._reset_tool_state(state)
state.active = False
@@ -1049,6 +1068,49 @@ class AndroidAdapter(BasePlatformAdapter):
"""Clear the typing indicator (``typing`` frame, on=false)."""
await self._ws_server.broadcast(protocol.typing(chat_id, False))
# ── M4: outbound media (agent -> app) ─────────────────────────────────
#
# The gateway's dispatch partition (gateway/run.py) extracts MEDIA: tags /
# image URLs from the final response, filters them through
# ``filter_media_delivery_paths``, then calls these ``send_*`` overrides
# with local file paths. We re-validate each path (defense in depth),
# register it in the media registry, mint a ``media_id``, and emit
# ``media.offer``; the app fetches the bytes via ``media.pull``.
async def _offer_media(
self,
chat_id: str,
path: str,
kind: str,
filename: Optional[str],
metadata: Optional[Dict[str, Any]],
) -> SendResult:
safe = validate_media_delivery_path(path)
if safe is None:
logger.warning("android: media path failed delivery validation: %s", path)
return SendResult(success=False, error="android: media path not deliverable")
try:
size = os.path.getsize(safe)
except OSError as e:
logger.warning("android: media file unreadable %s: %s", safe, e)
return SendResult(success=False, error="android: media file unreadable")
entry = self._media.register_outbound(
safe, kind, media_bridge.mime_for_path(safe), filename or os.path.basename(safe), size
)
thread_id = _thread_id_from_metadata(metadata)
frame = protocol.media_offer(
entry.media_id,
entry.kind,
entry.mime,
entry.size,
entry.filename,
chat_id=chat_id,
thread_id=thread_id,
message_id=self._last_message_id.get(chat_id),
)
await self._broadcast_or_log(chat_id, frame)
return SendResult(success=True, message_id=entry.media_id)
async def send_image(
self,
chat_id: str,
@@ -1057,8 +1119,63 @@ class AndroidAdapter(BasePlatformAdapter):
reply_to: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
) -> SendResult:
"""Send an image. M1: not implemented (M4)."""
return SendResult(success=False, error="android: media not implemented yet (M4)")
"""Send an image (M4: local files offered over WS; remote URLs fall
back to the base text rendering)."""
if image_url.startswith("file://"):
from urllib.parse import unquote
return await self._offer_media(chat_id, unquote(image_url[7:]), "image", None, metadata)
return await super().send_image(
chat_id, image_url, caption=caption, reply_to=reply_to, metadata=metadata
)
async def send_image_file(
self,
chat_id: str,
image_path: str,
caption: Optional[str] = None,
reply_to: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
**kwargs: Any,
) -> SendResult:
"""Send a local image file (M4)."""
return await self._offer_media(chat_id, image_path, "image", None, metadata)
async def send_video(
self,
chat_id: str,
video_path: str,
caption: Optional[str] = None,
reply_to: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
**kwargs: Any,
) -> SendResult:
"""Send a video (M4)."""
return await self._offer_media(chat_id, video_path, "video", None, metadata)
async def send_voice(
self,
chat_id: str,
audio_path: str,
caption: Optional[str] = None,
reply_to: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
**kwargs: Any,
) -> SendResult:
"""Send a voice note / audio file (M4)."""
return await self._offer_media(chat_id, audio_path, "voice", None, metadata)
async def send_document(
self,
chat_id: str,
file_path: str,
caption: Optional[str] = None,
file_name: Optional[str] = None,
reply_to: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
**kwargs: Any,
) -> SendResult:
"""Send a document (M4)."""
return await self._offer_media(chat_id, file_path, "document", file_name, metadata)
# ── Inbound (app -> agent) ────────────────────────────────────────────
@@ -1068,10 +1185,24 @@ class AndroidAdapter(BasePlatformAdapter):
Echoes the user message to all devices (multi-device sync + ack),
then builds a ``MessageEvent`` and hands it to ``handle_message()``
(the gateway's command pipeline + agent turn).
M4: ``media_refs`` reference completed ``media.upload``s; they are
resolved to ``MessageEvent.media_urls``/``media_types`` (local paths
the agent's vision/audio tools can read) and echoed in the user
message's ``media[]`` so every device renders the attachments.
"""
payload = frame.payload
text = payload.get("text")
if not isinstance(text, str) or not text.strip():
text = text if isinstance(text, str) else ""
refs_raw = payload.get("media_refs")
media_refs = (
[r for r in refs_raw if isinstance(r, str) and r]
if isinstance(refs_raw, list)
else []
)
if not text.strip() and not media_refs:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_UNSUPPORTED, "message.send requires non-empty text", id=frame.id),
@@ -1091,6 +1222,30 @@ class AndroidAdapter(BasePlatformAdapter):
if not isinstance(reply_to, str) or not reply_to.strip():
reply_to = None
# M4: resolve media refs (single-use; unknown ref -> error).
media_urls: List[str] = []
media_types: List[str] = []
media_wire: List[Dict[str, Any]] = []
for ref in media_refs:
entry = self._media.get_inbound(ref)
if entry is None:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_UNSUPPORTED, f"unknown media_ref {ref}", id=frame.id),
)
return
media_urls.append(entry.path)
media_types.append(entry.mime)
media_wire.append(
{
"media_id": entry.media_id,
"kind": entry.kind,
"mime": entry.mime,
"size": entry.size,
"filename": entry.filename,
}
)
device = self._devices.get(device_id) or {}
user_name = device.get("name") or device_id
@@ -1103,10 +1258,31 @@ class AndroidAdapter(BasePlatformAdapter):
role=protocol.ROLE_USER,
text=text,
thread_id=thread_id,
media=media_wire or None,
reply_to=reply_to,
ts=int(time.time() * 1000),
)
await self._ws_server.broadcast(echo)
# Refs are consumed by this message (no replay).
for ref in media_refs:
self._media.pop_inbound(ref)
# M4: a new user turn starts -- stale offer association is dropped.
self._last_message_id.pop(chat_id, None)
kind = media_wire[0]["kind"] if media_wire else None
if kind == "image":
message_type = MessageType.PHOTO
elif kind == "video":
message_type = MessageType.VIDEO
elif kind == "audio":
message_type = MessageType.AUDIO
elif kind == "voice":
message_type = MessageType.VOICE
elif kind == "document":
message_type = MessageType.DOCUMENT
else:
message_type = MessageType.TEXT
source = self.build_source(
chat_id=chat_id,
@@ -1118,15 +1294,143 @@ class AndroidAdapter(BasePlatformAdapter):
)
event = MessageEvent(
text=text,
message_type=MessageType.TEXT,
message_type=message_type,
user_id=device_id,
user_name=user_name,
source=source,
message_id=message_id,
reply_to_message_id=reply_to,
media_urls=media_urls,
media_types=media_types,
)
await self.handle_message(event)
# ── M4: inbound media (app -> agent) ──────────────────────────────────
#
# ``media.upload.start`` -> raw binary frames (one at a time per
# connection) -> ``media.upload.end``. The session streams to a temp
# file (bounded RAM); on end we verify size + sha256, re-sniff the kind,
# and cache via hermes ``cache_*_from_bytes``. ``media.pull`` serves an
# outbound offer as chunked binary frames, re-checking the delivery-path
# validation at pull time.
async def on_media_upload_start(self, frame: protocol.Frame, device_id: str) -> None:
payload = frame.payload
media_ref = str(payload.get("media_ref") or "").strip()
if not media_ref or len(media_ref) > 64:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_UNSUPPORTED, "media.upload.start requires media_ref", id=frame.id),
)
return
kind = payload.get("kind")
if kind not in media_bridge.KINDS:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_UNSUPPORTED, f"unsupported media kind {kind!r}", id=frame.id),
)
return
mime = str(payload.get("mime") or "application/octet-stream")[:128]
filename = str(payload.get("filename") or "upload")[:255]
size = payload.get("size")
try:
size = int(size) if size is not None else -1
except (TypeError, ValueError):
size = -1
if size <= 0:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_UNSUPPORTED, "media.upload.start requires a positive size", id=frame.id),
)
return
if size > self.max_upload_bytes:
await self._ws_server.send_to(
device_id,
protocol.error(
protocol.ERR_MEDIA_TOO_LARGE,
f"upload of {size} bytes exceeds limit ({self.max_upload_bytes})",
id=frame.id,
),
)
return
try:
self._media.create_upload(
device_id, media_ref, kind, mime, filename, size, frame.id,
self.max_upload_bytes,
)
except media_bridge.MediaError as e:
await self._ws_server.send_to(device_id, protocol.error(e.code, e.message, id=frame.id))
return
# No ack: WS ordering guarantees the server processes this before the
# first binary chunk; failures arrive as ``error`` frames.
async def on_media_chunk(self, device_id: str, chunk: bytes) -> None:
session = self._media.get_upload(device_id)
if session is None:
return # stray binary frame: ignore (forward-compat)
session.feed(chunk)
if session.failed:
await self._ws_server.send_to(
device_id,
protocol.error(session.error_code, session.error_message, id=session.request_id),
)
self._media.discard_upload(device_id, session.media_ref)
async def on_media_upload_end(self, frame: protocol.Frame, device_id: str) -> None:
payload = frame.payload
media_ref = str(payload.get("media_ref") or "").strip()
sha256 = str(payload.get("sha256") or "").strip().lower()
if not media_ref:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_UNSUPPORTED, "media.upload.end requires media_ref", id=frame.id),
)
return
try:
entry = self._media.complete_upload(device_id, media_ref, sha256)
except media_bridge.MediaError as e:
await self._ws_server.send_to(device_id, protocol.error(e.code, e.message, id=frame.id))
return
await self._ws_server.send_to(
device_id, protocol.media_upload_ack(True, entry.media_id, id=frame.id)
)
async def on_media_pull(self, frame: protocol.Frame, device_id: str) -> None:
payload = frame.payload
media_id = str(payload.get("media_id") or "").strip()
entry = self._media.get_outbound(media_id) if media_id else None
if entry is None:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_NOT_FOUND, f"unknown media_id {media_id!r}", id=frame.id),
)
return
# Delivery-path security: re-validate at pull time (the file may have
# moved / been replaced since the offer).
safe = validate_media_delivery_path(entry.path)
if safe is None:
await self._ws_server.send_to(
device_id,
protocol.error(protocol.ERR_NOT_FOUND, "media no longer deliverable", id=frame.id),
)
return
conn = self._ws_server.connection(device_id)
if conn is None:
return
try:
await media_bridge.stream_file(conn.ws, safe, media_bridge.DEFAULT_CHUNK_BYTES)
except Exception as e:
logger.warning("android: media.pull stream failed for %s: %s", media_id, e)
await self._ws_server.send_to(
device_id, protocol.error(protocol.ERR_INTERNAL, f"pull failed: {e}", id=frame.id)
)
return
await self._ws_server.send_to(device_id, protocol.media_pull_end(True, id=frame.id))
def on_connection_closed(self, device_id: str) -> None:
"""M4: drop in-flight upload temp files for a disconnected device."""
self._media.discard_device(device_id)
# ── M3: channel directory management (app -> agent) ───────────────────
#
# Each request is answered by broadcasting the matching ``channel.*``
@@ -1318,12 +1622,12 @@ class AndroidAdapter(BasePlatformAdapter):
# ── hello.ack helpers ─────────────────────────────────────────────────
def server_caps(self) -> Dict[str, Any]:
"""Capability flags advertised in ``hello.ack`` (M3 surface)."""
"""Capability flags advertised in ``hello.ack`` (M4 surface)."""
return {
"streaming": True, # M2: message.start/update/stop
"reasoning": True, # M2: reasoning field on message / message.stop
"tools": True, # M2: tool.start/progress/end
"media": False, # M4
"media": True, # M4: media.upload/offer/pull
"search": True, # M3: search frame
"push": self.push_backend,
"pickers": False, # M2+
@@ -1431,6 +1735,11 @@ def register(ctx):
"(Android/Desktop). It renders Markdown, inline code, images, "
"audio and video, and shows your reasoning and tool activity. "
"Conversations are organized into channels and optional threads. "
"Keep formatting rich but readable."
"Keep formatting rich but readable. "
"You can send media files natively: to deliver a file to the user, "
"include MEDIA:/absolute/path/to/file in your response. Images "
"(.png, .jpg, .webp) appear as photos, audio (.ogg, .mp3, .m4a) "
"plays inline, videos (.mp4, .webm, .mov) play inline, and other "
"files arrive as downloadable documents."
),
)