M0: toolchain, monorepo scaffold, gateway plugin skeleton, CMP app
- gateway-plugin/: android platform plugin (plugin.yaml + adapter.py register(ctx) + no-op AndroidAdapter) + stub modules for M1-M5 - app/: Compose Multiplatform project (shared KMP + androidApp + desktopApp) with Gradle wrapper; builds :androidApp:assembleDebug and :desktopApp:compileKotlin - scripts/guard_hermes_agent.sh + pre-commit hook: fail if hermes-agent/ is staged (read-only reference, never committed) - .gitignore excludes hermes-agent/; docs/ reference library
This commit is contained in:
ARIA committed
commit
59acf66c89
49 files changed
+3950
No files matched your search
@@ -0,0 +1,93 @@
|
||||
# 00 — Overview
|
||||
|
||||
## Vision
|
||||
|
||||
A **native, Telegram-quality chat experience** for a personal hermes-agent:
|
||||
install an app on your phone (and a desktop app on your PC), pair it to your
|
||||
running `hermes gateway`, and talk to your agent with streaming replies,
|
||||
visible reasoning, structured tool activity, channels/threads, media, search,
|
||||
and push notifications — with cron jobs able to post into any channel you
|
||||
create.
|
||||
|
||||
## Goals
|
||||
|
||||
- **Native feel.** Real Android app (Kotlin/Compose), not a WebView. Desktop
|
||||
app that is the same app, resized for a big screen.
|
||||
- **First-class gateway citizen.** The app is a hermes *messaging platform*, so
|
||||
everything the gateway already does "just works": slash commands, cron
|
||||
delivery, `send_message` routing, coexistence with Telegram/Discord/etc.
|
||||
- **Full agent transparency.** Streaming text, reasoning shown *before* the
|
||||
answer, structured tool events (the app chooses how much to show), and
|
||||
intermediate assistant beats.
|
||||
- **Organized by default.** A default chat with optional threads, plus
|
||||
user-created channels that cron jobs can target.
|
||||
- **Reachable anywhere.** Live over a WebSocket; background push via FCM
|
||||
(primary) or ntfy (fallback).
|
||||
|
||||
## In scope (v1)
|
||||
|
||||
Everything in the feature checklist below.
|
||||
|
||||
## Out of scope / stretch (v1)
|
||||
|
||||
- **Standalone-cron delivery while the gateway process is fully down** —
|
||||
best-effort FCM/ntfy only (the outbox is served by the running gateway).
|
||||
- **Multi-user / group chat** — this is a *personal* 1-user agent.
|
||||
- **End-to-end encryption** — transport security (WSS) only.
|
||||
- **iOS** — Android + Desktop only (the protocol is transport-agnostic, so an
|
||||
iOS client is a future port, not a v1 goal).
|
||||
|
||||
## Feature checklist → where it's handled
|
||||
|
||||
| Requirement | Gateway plugin | App |
|
||||
|---|---|---|
|
||||
| Input box, auto-grow (max height) | — | Compose `TextField` + bounded `heightIn` |
|
||||
| Menu button → all slash commands | Dispatches `/…`; serves command catalog | Bottom-sheet menu + `/` autocomplete |
|
||||
| Tool output (app decides how much) | Emits **structured** tool events | App setting: everything / truncated / nothing |
|
||||
| Reasoning shown before message | Captures + splits reasoning | Collapsible "Reasoning" block above message |
|
||||
| Intermediate messages | Forwards `Commentary` events | Distinct dimmed bubble |
|
||||
| Threading + channels; default chat; user channels for cron | `chat_id`/`thread_id` model; cron `deliver=android:<chat>[:<thread>]` | Channel list, thread toggle, "new channel" |
|
||||
| Search ("everywhere" / "this chat/channel") | FTS5 session search bridge | Search UI + scope toggle |
|
||||
| Attach media (music/video/images/docs) | Inbound cache; outbound `send_*` | Pickers + chunked upload + preview |
|
||||
| Push notifications | FCM (primary) / ntfy (fallback) | FCM token / ntfy topic + notification service |
|
||||
| Live playback of AI-sent music/video | Serves media bytes over WS | ExoPlayer inline player |
|
||||
|
||||
## Locked decisions (from planning)
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Desktop app tech | **Compose Multiplatform** (shares Android code; "tweaked" for big screen) |
|
||||
| Push backend | **Both** — FCM primary, ntfy fallback (`ANDROID_PUSH_BACKEND`) |
|
||||
| Media transport | **Over the WebSocket** (chunked binary frames; no extra Python deps) |
|
||||
| Phone default layout | **User-toggleable, single-pane default** (auto two-pane on large screens) |
|
||||
|
||||
## Disclaimers (hard rules)
|
||||
|
||||
1. **`hermes-agent/` is a read-only research reference.** It lives next to this
|
||||
folder for study only. It is **git-ignored** and must **never** be committed,
|
||||
pushed, or included in any artifact. Our plugin is *installed* into a live
|
||||
hermes home (`~/.hermes/plugins/android`); we never edit hermes core files.
|
||||
2. **ADB is available and a device is connected** (`a5ca2a4b`, Xiaomi MIX 2S,
|
||||
Android 10 / API 29). Use `adb install` / `adb logcat` / `adb shell am start`
|
||||
to install, launch, and debug the app on-device throughout the build.
|
||||
|
||||
## Verified environment state (2026-08-19)
|
||||
|
||||
| Item | State |
|
||||
|---|---|
|
||||
| OS | CachyOS (Arch-based), `pacman` present |
|
||||
| JDK | **Not installed** → Milestone M0 (`pacman -S jdk17-openjdk`) |
|
||||
| Android SDK | **Not installed** → M0 (cmdline-tools + sdkmanager) |
|
||||
| Gradle | Via project wrapper (`gradlew`), no system install |
|
||||
| ADB | Installed; device `a5ca2a4b` (MIX 2S, API 29) connected |
|
||||
| Python | 3.14.7; `uv` 0.12.3 present |
|
||||
| hermes venv | **Not created** → M0 (`cd hermes-agent && uv sync`) |
|
||||
| hermes core deps | `websockets==15.0.1` and `httpx` are **core** deps → plugin needs **zero new Python deps** |
|
||||
| Disk / RAM | 522 GB free / 62 GB RAM — ample |
|
||||
|
||||
## Naming
|
||||
|
||||
- Product/effort name: **Iris × Hermes** (folder `iris_x_hermes`).
|
||||
- hermes platform name: **`android`** (the plugin registers `Platform("android")`).
|
||||
- WS default port: **8790** (configurable).
|
||||
- Default chat id: **`android:default`** (the home channel).
|
||||
@@ -0,0 +1,106 @@
|
||||
# 01 — Architecture
|
||||
|
||||
## System diagram
|
||||
|
||||
```
|
||||
┌────────────────────────────── User's machine (home server / PC) ─────────────────────────────┐
|
||||
│ │
|
||||
│ hermes gateway (ONE process) │
|
||||
│ ┌──────────────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Agent core (run_agent.py) ── sessions (SQLite + FTS5) ── cron scheduler │ │
|
||||
│ │ │ │ │
|
||||
│ │ ▼ legacy stream callbacks (delta / tool-progress / commentary) │ │
|
||||
│ │ ┌──────────────────────────────┐ ┌────────────────────────────────┐ │ │
|
||||
│ │ │ android PLATFORM PLUGIN │◄───────►│ WebSocket SERVER (websockets) │ │ │
|
||||
│ │ │ AndroidAdapter │ JSON │ ws://host:8790/ws │ │ │
|
||||
│ │ │ (BasePlatformAdapter) │ frames │ + media + FCM-token + pairing │ │ │
|
||||
│ │ │ • send / edit / stream │ └───────────────┬────────────────┘ │ │
|
||||
│ │ │ • media cache │ │ WSS │ │
|
||||
│ │ │ • outbox (SQLite) │ │ │ │
|
||||
│ │ │ • push (FCM / ntfy) │─────────────────────────┼──────────┐ │ │
|
||||
│ │ └──────────────────────────────┘ │ │ │ │
|
||||
│ └───────────────────────────────────────────────────────────┼──────────┼───────────┘ │
|
||||
└───────────────────────────────────────────────────────────────┼──────────┼───────────────────────┘
|
||||
│ │ push (FCM HTTP v1 / ntfy)
|
||||
┌──────────────────┼──────────▼─────────┐
|
||||
│ │ Google FCM cloud │
|
||||
▼ │ │ │
|
||||
┌────────────────────────┐ │ ▼ │
|
||||
│ ANDROID APP │◄──┴── (wake) ┌──────────┐
|
||||
│ (Kotlin / Compose) │ WSS │ PHONE │
|
||||
│ • WS client (OkHttp) │◄───────────►│ MIX 2S │
|
||||
│ • ExoPlayer │ │ (API 29) │
|
||||
│ • FCM service │ └──────────┘
|
||||
└────────────────────────┘
|
||||
DESKTOP APP (Compose Multiplatform)
|
||||
• same shared code, WSS to same server
|
||||
• tray + OS notifications (no FCM), big-screen two-pane
|
||||
```
|
||||
|
||||
## Process model
|
||||
|
||||
- **One `hermes gateway` process** hosts the agent core, the session store, the
|
||||
cron scheduler, *and* our `android` platform plugin. The plugin's WebSocket
|
||||
server runs on the gateway's asyncio loop (started in `AndroidAdapter.connect()`).
|
||||
- **The app is a client.** It *initiates* the WS connection to the gateway
|
||||
(outbound), so no inbound port is needed on the phone. For LAN/remote access
|
||||
the user points the app at the gateway's LAN IP / Tailscale name / a WSS
|
||||
tunnel (see `09-pairing-security.md`).
|
||||
- **Push is the only inbound path to a sleeping phone**, and it goes through a
|
||||
cloud relay (FCM or ntfy), not a direct connection.
|
||||
|
||||
## Why the *messaging gateway* is the connection point (not `tui_gateway`)
|
||||
|
||||
hermes has two "gateways": the **messaging gateway** (`hermes gateway`, which
|
||||
serves Telegram/Discord/… and cron) and the **`tui_gateway`** JSON-RPC backend
|
||||
(used by the TUI and the existing Electron desktop app). We deliberately use the
|
||||
**messaging gateway** because:
|
||||
|
||||
1. **Cron delivery is native.** Cron jobs resolve `deliver=android:<chat>[:<thread>]`
|
||||
through the platform registry and call our adapter's `send()`. No bridging.
|
||||
2. **`send_message` tool routing** works out of the box (plugin
|
||||
`parse_target_ref_fn`).
|
||||
3. **Slash commands** are dispatched by the gateway's command pipeline — the app
|
||||
just sends `/cmd args` as a message.
|
||||
4. **Coexistence.** The same agent is reachable via Telegram *and* the app at
|
||||
once; sessions/channels are shared.
|
||||
|
||||
The `tui_gateway` WS protocol is *not* reused; we define a clean, purpose-built
|
||||
protocol (`04-wire-protocol.md`) that borrows familiar names (`message.*`,
|
||||
`tool.*`, `reasoning`) but is owned by our plugin.
|
||||
|
||||
## Key architectural decisions + rationale
|
||||
|
||||
| Decision | Rationale |
|
||||
|---|---|
|
||||
| **Community-style platform plugin** (`register(ctx)` → `ctx.register_platform`) | Zero hermes-core changes; survives hermes updates; follows `ADDING_A_PLATFORM.md` "Plugin Path". |
|
||||
| **Single WS transport** for chat, streaming, tools, media, pairing, FCM-token | One connection, one auth, one dependency (`websockets`, already core). Media as chunked binary frames avoids adding an HTTP server. |
|
||||
| **`websockets` + `httpx` only** | Both are hermes *core* deps → the plugin adds **zero** new Python dependencies (respects hermes supply-chain pinning policy). |
|
||||
| **Structured tool events, app-side verbosity** | Per the requirement: the gateway sends full tool data; the *app* decides everything/truncated/nothing. |
|
||||
| **Reasoning split in the adapter** | The gateway prepends reasoning to the final text when `show_reasoning` is on; the adapter splits the stable prefix into a `reasoning` field so the app renders a clean collapsible block. |
|
||||
| **Channels/threads = `chat_id`/`thread_id`** | The gateway's `SessionSource` already models this; cron delivery already targets `platform:chat_id:thread_id`. We map app concepts onto existing primitives. |
|
||||
| **SQLite outbox + sync cursor** | Offline delivery + reconnect catch-up without re-reading full history. |
|
||||
| **Compose Multiplatform** | Desktop is "the Android app, tweaked" → share protocol/state/UI; only platform services + layout differ. |
|
||||
|
||||
## Data flow (one turn)
|
||||
|
||||
1. App sends `message.send {text}` (or `/cmd`).
|
||||
2. Plugin builds a `MessageEvent` (+ `media_urls` if attachments) →
|
||||
`AndroidAdapter.handle_message(event)`.
|
||||
3. Gateway resolves the session (`chat_id`/`thread_id`), runs the agent.
|
||||
4. Agent streams: `stream_delta_callback` → `GatewayStreamConsumer` →
|
||||
`adapter.send()` (first) / `adapter.edit_message()` (updates) →
|
||||
`message.start` / `message.update` frames.
|
||||
5. Tool calls: `tool_progress_callback` → progress queue → `adapter.send()` →
|
||||
`tool.*` frames (structured).
|
||||
6. Intermediate beats: `interim_assistant_callback` → consumer → `commentary` frames.
|
||||
7. Final answer: consumer finalizes → `adapter.send()` → `message` frame
|
||||
(reasoning split into its own field).
|
||||
8. If the app is disconnected at any point: frame is dropped to the **outbox**
|
||||
and a **push** is fired; on reconnect the app `sync`s the delta.
|
||||
|
||||
> Implementation note: the main gateway uses the **legacy callback path**
|
||||
> (not the ACP-only event-native `render_message_event` path). We therefore map
|
||||
> the legacy `send`/`edit_message`/progress calls to our frames. The exact
|
||||
> tool-progress vs commentary classification is verified empirically in M2 by
|
||||
> running the real gateway with a test WS client (see `13-testing.md`).
|
||||
@@ -0,0 +1,130 @@
|
||||
# 02 — Monorepo Structure
|
||||
|
||||
One repository, three artifacts. `hermes-agent/` is **not** part of the repo
|
||||
(git-ignored reference).
|
||||
|
||||
## Top-level tree
|
||||
|
||||
```
|
||||
iris_x_hermes/
|
||||
├── .gitignore # MUST exclude hermes-agent/ (see below)
|
||||
├── README.md # Repo root readme (short; points to docs/)
|
||||
├── docs/ # ← THIS reference library
|
||||
│
|
||||
├── hermes-agent/ # ⚠️ READ-ONLY REFERENCE — NEVER PUSHED (git-ignored)
|
||||
│
|
||||
├── gateway-plugin/ # ① Python plugin → installed to ~/.hermes/plugins/android
|
||||
│ ├── plugin.yaml # manifest (kind: platform, env vars, home channel)
|
||||
│ ├── __init__.py
|
||||
│ ├── adapter.py # AndroidAdapter(BasePlatformAdapter) + register(ctx)
|
||||
│ ├── ws_server.py # websockets server, connection registry, framing
|
||||
│ ├── protocol.py # frame schemas (source of truth, mirrored in Kotlin)
|
||||
│ ├── media.py # inbound cache + outbound chunked streaming
|
||||
│ ├── outbox.py # SQLite offline outbox + sync cursor
|
||||
│ ├── push.py # PushBackend: FcmBackend + NtfyBackend
|
||||
│ ├── pairing.py # token gen/verify, device registry
|
||||
│ ├── search.py # FTS5 session search bridge
|
||||
│ └── tests/ # pytest (run via hermes scripts/run_tests.sh)
|
||||
│
|
||||
├── app/ # ② + ③ Compose Multiplatform project (Kotlin)
|
||||
│ ├── settings.gradle.kts
|
||||
│ ├── build.gradle.kts
|
||||
│ ├── gradle.properties
|
||||
│ ├── gradle/ gradlew gradlew.bat
|
||||
│ ├── shared/ # KMP module — the bulk of the code
|
||||
│ │ ├── build.gradle.kts
|
||||
│ │ └── src/
|
||||
│ │ ├── commonMain/kotlin/iris/… # protocol, WS client, repo, state, Compose UI
|
||||
│ │ ├── androidMain/kotlin/… # FCM, ExoPlayer, SAF picker, notifications
|
||||
│ │ └── desktopMain/kotlin/… # tray, file dialog, player, window
|
||||
│ ├── androidApp/ # thin Android shell (Application, MainActivity)
|
||||
│ │ ├── build.gradle.kts
|
||||
│ │ └── src/main/… # AndroidManifest, res, Firebase options
|
||||
│ └── desktopApp/ # thin Desktop shell (main(), window)
|
||||
│ ├── build.gradle.kts
|
||||
│ └── src/main/kotlin/…
|
||||
│
|
||||
└── (no other top-level code)
|
||||
```
|
||||
|
||||
## Module responsibilities
|
||||
|
||||
### `gateway-plugin/` (Python)
|
||||
- **`plugin.yaml`** — manifest: `name: android-platform`, `kind: platform`,
|
||||
`requires_env` / `optional_env` (surfaced in `hermes config`/setup).
|
||||
- **`adapter.py`** — `AndroidAdapter(BasePlatformAdapter)` + `register(ctx)`.
|
||||
The heart of the plugin. See `03-gateway-plugin.md`.
|
||||
- **`ws_server.py`** — `websockets` server, per-device connection registry,
|
||||
frame encode/decode, heartbeat, broadcast routing to all connected devices.
|
||||
- **`protocol.py`** — dataclasses/constants for every frame (single source of
|
||||
truth; `docs/protocol/frames.schema.json` is generated/mirrored from it).
|
||||
- **`media.py`** — inbound chunked upload → `cache_*_from_bytes`; outbound
|
||||
`media.offer`/`media.pull` chunked streaming.
|
||||
- **`outbox.py`** — SQLite outbox per `chat_id` + monotonic sync cursor.
|
||||
- **`push.py`** — `PushBackend` interface; `FcmBackend` (httpx, FCM HTTP v1) and
|
||||
`NtfyBackend` (reuses hermes ntfy publish). Selected by `ANDROID_PUSH_BACKEND`.
|
||||
- **`pairing.py`** — token generation/verification (constant-time), device
|
||||
registry (SQLite), QR payload.
|
||||
- **`search.py`** — FTS5 query bridge over the hermes session store.
|
||||
|
||||
### `app/shared` (Kotlin KMP)
|
||||
- **`commonMain`** — protocol models (kotlinx-serialization), `GatewayClient`
|
||||
(OkHttp WS), repositories (Room), ViewModels (StateFlow), and the Compose UI
|
||||
(design system, screens). ~80% of app code.
|
||||
- **`androidMain`** — FCM service, ExoPlayer, SAF media picker, system
|
||||
notifications, `MediaPlayer` actual.
|
||||
- **`desktopMain`** — tray + OS notifications, desktop player, file dialog,
|
||||
window management, `MediaPlayer` actual.
|
||||
|
||||
### `app/androidApp` / `app/desktopApp`
|
||||
Thin shells: `Application`/`MainActivity` (Android) and `main()`/window
|
||||
(Desktop). They compose the `shared` UI and inject platform services.
|
||||
|
||||
## Build systems
|
||||
|
||||
- **Python plugin:** no build step (pure Python, stdlib + hermes core deps).
|
||||
Installed by copying/symlinking into `~/.hermes/plugins/android`. Tested with
|
||||
hermes's `scripts/run_tests.sh`.
|
||||
- **Kotlin/CMP:** Gradle (Kotlin DSL) with the Compose Multiplatform plugin.
|
||||
`./gradlew :androidApp:installDebug`, `./gradlew :desktopApp:run`,
|
||||
`./gradlew :shared:testDebugUnitTest`.
|
||||
|
||||
## `.gitignore` (root) — critical
|
||||
|
||||
```gitignore
|
||||
# hermes-agent is a read-only research reference — NEVER commit/push it
|
||||
/hermes-agent/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Kotlin / Gradle
|
||||
.gradle/
|
||||
build/
|
||||
local.properties
|
||||
*.iml
|
||||
.idea/
|
||||
|
||||
# Android / Firebase
|
||||
app/androidApp/src/main/res/values/secrets.xml
|
||||
google-services.json
|
||||
*.jks
|
||||
keystore.jks
|
||||
|
||||
# OS / misc
|
||||
.DS_Store
|
||||
*.log
|
||||
```
|
||||
|
||||
> The `/hermes-agent/` line is non-negotiable. Add a pre-commit guard (or CI
|
||||
> check) that fails if any path under `hermes-agent/` is staged.
|
||||
|
||||
## Install layout (runtime)
|
||||
|
||||
- **Plugin:** `~/.hermes/plugins/android/` ← copy of `gateway-plugin/`
|
||||
(or a symlink for dev). Discovered by hermes's `PluginManager`.
|
||||
- **App (dev):** installed on-device via `./gradlew :androidApp:installDebug`.
|
||||
- **App (desktop, dev):** `./gradlew :desktopApp:run`.
|
||||
@@ -0,0 +1,263 @@
|
||||
# 03 — Gateway Plugin (Python)
|
||||
|
||||
The plugin is a **community-style hermes platform plugin** named `android`.
|
||||
It follows the "Plugin Path" in `hermes-agent/gateway/platforms/ADDING_A_PLATFORM.md`
|
||||
and the canonical example `hermes-agent/plugins/platforms/irc/adapter.py`.
|
||||
**Zero hermes-core changes. Zero new Python dependencies** (`websockets` and
|
||||
`httpx` are already core deps).
|
||||
|
||||
> Source map for every hermes integration point: see `15-hermes-reference.md`.
|
||||
|
||||
## 3.1 `plugin.yaml` (manifest)
|
||||
|
||||
```yaml
|
||||
name: android-platform
|
||||
label: Android
|
||||
kind: platform
|
||||
version: 0.1.0
|
||||
description: >
|
||||
Native Android / Desktop client gateway adapter for Hermes Agent.
|
||||
Runs a WebSocket server inside the gateway; the app connects with a
|
||||
pairing token. Supports streaming, reasoning, structured tool events,
|
||||
channels/threads, media, FTS5 search, and FCM/ntfy push.
|
||||
author: <you>
|
||||
requires_env:
|
||||
- name: ANDROID_TOKEN
|
||||
description: "Shared pairing token the app presents on connect"
|
||||
prompt: "Android pairing token"
|
||||
password: true
|
||||
optional_env:
|
||||
- name: ANDROID_WS_HOST
|
||||
description: "WS bind host (default 127.0.0.1; use 0.0.0.0 for LAN)"
|
||||
prompt: "WS host"
|
||||
password: false
|
||||
- name: ANDROID_WS_PORT
|
||||
description: "WS port (default 8790)"
|
||||
prompt: "WS port"
|
||||
password: false
|
||||
- name: ANDROID_HOME_CHANNEL
|
||||
description: "Default chat id for cron/notification delivery (default android:default)"
|
||||
prompt: "Home channel"
|
||||
password: false
|
||||
- name: ANDROID_ALLOWED_USERS
|
||||
description: "Comma-separated allowed device_ids (empty = token-only auth)"
|
||||
prompt: "Allowed device ids"
|
||||
password: false
|
||||
- name: ANDROID_ALLOW_ALL_USERS
|
||||
description: "Allow any paired device (dev only)"
|
||||
prompt: "Allow all devices? (true/false)"
|
||||
password: false
|
||||
- name: ANDROID_PUSH_BACKEND
|
||||
description: "Push backend: fcm (default) or ntfy"
|
||||
prompt: "Push backend"
|
||||
password: false
|
||||
- name: ANDROID_FCM_SERVICE_ACCOUNT
|
||||
description: "Path to Firebase service-account JSON (FCM HTTP v1)"
|
||||
prompt: "FCM service account path"
|
||||
password: true
|
||||
- name: ANDROID_FCM_SERVER_KEY
|
||||
description: "Legacy FCM server key (fallback if no service account)"
|
||||
prompt: "FCM server key"
|
||||
password: true
|
||||
- name: NTFY_TOPIC
|
||||
description: "ntfy topic for push (when ANDROID_PUSH_BACKEND=ntfy)"
|
||||
prompt: "ntfy topic"
|
||||
password: false
|
||||
- name: NTFY_SERVER_URL
|
||||
description: "ntfy server URL (default https://ntfy.sh)"
|
||||
prompt: "ntfy server URL"
|
||||
password: false
|
||||
- name: ANDROID_WS_CERT
|
||||
description: "TLS cert path for WSS (optional)"
|
||||
prompt: "WSS cert"
|
||||
password: false
|
||||
- name: ANDROID_WS_KEY
|
||||
description: "TLS key path for WSS (optional)"
|
||||
prompt: "WSS key"
|
||||
password: false
|
||||
```
|
||||
|
||||
Behavioral (non-secret) settings live in `config.yaml` under
|
||||
`gateway.platforms.android.extra` (host, port, home_channel, outbox retention,
|
||||
max upload bytes, tls). Secrets live in `.env`. (hermes policy: `.env` = secrets
|
||||
only.)
|
||||
|
||||
## 3.2 `register(ctx)` entry point
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_platform(
|
||||
name="android",
|
||||
label="Android",
|
||||
adapter_factory=lambda cfg: AndroidAdapter(cfg),
|
||||
check_fn=check_requirements, # passive: websockets importable + token set
|
||||
validate_config=validate_config, # host/port/token present
|
||||
is_connected=is_connected,
|
||||
required_env=["ANDROID_TOKEN"],
|
||||
install_hint="No extra packages needed (websockets + httpx are core deps)",
|
||||
setup_fn=interactive_setup, # hermes gateway setup flow
|
||||
env_enablement_fn=_env_enablement, # seed extra + home_channel from env
|
||||
cron_deliver_env_var="ANDROID_HOME_CHANNEL",
|
||||
standalone_sender_fn=_standalone_send, # best-effort out-of-proc cron (stretch)
|
||||
parse_target_ref_fn=_parse_target_ref, # "android:<chat>[:<thread>]"
|
||||
allowed_users_env="ANDROID_ALLOWED_USERS",
|
||||
allow_all_env="ANDROID_ALLOW_ALL_USERS",
|
||||
max_message_length=0, # 0 = no limit (WS has none)
|
||||
emoji="📱",
|
||||
pii_safe=False,
|
||||
platform_hint=(
|
||||
"You are chatting with the user through their native Iris app "
|
||||
"(Android/Desktop). It renders Markdown, inline code, images, "
|
||||
"audio and video, and shows your reasoning and tool activity. "
|
||||
"Conversations are organized into channels and optional threads. "
|
||||
"Keep formatting rich but readable."
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
Field reference (all from `PlatformEntry`, `gateway/platform_registry.py:63`):
|
||||
`adapter_factory`, `check_fn`, `validate_config`, `is_connected`, `required_env`,
|
||||
`install_hint`, `setup_fn`, `env_enablement_fn`, `apply_yaml_config_fn`,
|
||||
`cron_deliver_env_var`, `parse_target_ref_fn`, `allowed_users_env`,
|
||||
`allow_all_env`, `max_message_length`, `pii_safe`, `platform_hint`, `emoji`,
|
||||
`ensure_deps_fn`.
|
||||
|
||||
- **`check_requirements()`** — passive probe: `import websockets` succeeds and
|
||||
`ANDROID_TOKEN` is set. Never installs.
|
||||
- **`_env_enablement()`** — returns a dict seeding `PlatformConfig.extra`
|
||||
(host/port/home_channel/push_backend) + a `home_channel` key
|
||||
`{"chat_id": "android:default", "name": "Default"}` so `hermes gateway status`
|
||||
and cron home-channel resolution work without instantiating the adapter.
|
||||
- **`_parse_target_ref(ref)`** — if `ref` starts with `android:`, return
|
||||
`(chat_id, thread_id)` parsed from `android:<chat>[:<thread>]`; else `None`.
|
||||
- **`interactive_setup()`** — prompts for token (or generates one), host/port,
|
||||
push backend + credentials, prints a QR code (pairing) and the app URL.
|
||||
|
||||
## 3.3 `AndroidAdapter(BasePlatformAdapter)`
|
||||
|
||||
Constructor: `super().__init__(config=config, platform=Platform("android"))`.
|
||||
Reads `config.extra` (env overrides win). Initializes: WS server (not started
|
||||
until `connect()`), connection registry, outbox (SQLite under
|
||||
`get_hermes_home()/"android"`), push backend, pairing store, channel directory.
|
||||
|
||||
### Lifecycle
|
||||
- **`connect(*, is_reconnect=False) -> bool`**
|
||||
- Acquire scoped lock (`gateway.status.acquire_scoped_lock("android", key)`)
|
||||
so two profiles can't bind the same port/identity.
|
||||
- Start the `websockets` server on `host:port` (TLS if cert/key set).
|
||||
- `_mark_connected()`; return True.
|
||||
- **`disconnect()`**
|
||||
- Stop server, close all device sockets, release lock, `_mark_disconnected()`.
|
||||
|
||||
### Inbound (app → agent)
|
||||
- WS `message.send {text, reply_to?, media_refs?}` → build `SessionSource` via
|
||||
`self.build_source(chat_id, chat_name, chat_type, user_id, user_name,
|
||||
thread_id)` → build `MessageEvent(text=…, message_type=TEXT, source=…,
|
||||
media_urls=[cached paths], media_types=[…], reply_to_message_id=…)` →
|
||||
`await self.handle_message(event)`.
|
||||
- Slash commands arrive as plain text starting with `/`; the gateway's command
|
||||
pipeline resolves + dispatches them (no special handling needed).
|
||||
- `picker.select {picker_id, value}` → route to the gateway-side resolvers
|
||||
(model picker / choice picker / clarify / approval / slash-confirm) using the
|
||||
shared callback-id conventions (`cl:<id>:<idx>`, `appr:<id>:<choice>`,
|
||||
`sc:<choice>:<id>`).
|
||||
- `channel.create` / `channel.rename` / `channel.set_default` → mutate the
|
||||
channel directory (SQLite) + emit `channel.*` frames to all devices.
|
||||
- `search {query, scope, chat_id?, thread_id?}` → `search.py` → `search.results`.
|
||||
- `media.upload` (chunked) → `media.py` → `cache_*_from_bytes` → `media_ref`.
|
||||
- `media.pull {media_id}` → stream cached bytes as binary frames.
|
||||
- `fcm.register {token}` / `hello` → update device registry.
|
||||
- `read.receipt {message_id}` → mark delivered/read (drives ✓✓), ack.
|
||||
- `sync {cursor}` → `outbox.py` → replay frames since cursor.
|
||||
|
||||
### Outbound (agent → app)
|
||||
- **`send(chat_id, content, reply_to=None, metadata=None) -> SendResult`**
|
||||
- Split reasoning prefix (see `05-streaming.md`) → `reasoning` field.
|
||||
- If **any** device is connected: broadcast `message` frame to all.
|
||||
- Else (no live devices): append to **outbox** + fire **push** (FCM/ntfy).
|
||||
- Return `SendResult(success=True, message_id=<id>)`.
|
||||
- **`edit_message(chat_id, message_id, content)`** → `message.update` frame
|
||||
(drives streaming). If no device, no-op (outbox holds the final `send`).
|
||||
- **`send_typing(chat_id, metadata=None)`** → `typing` frame.
|
||||
- **`get_chat_info(chat_id) -> dict`** → `{"name": <channel name>, "type": "dm"|"channel"}`
|
||||
from the channel directory.
|
||||
- **Media send** — `send_image / send_video / send_document / send_voice /
|
||||
send_image_file / send_multiple_images`: stage the file in the media cache,
|
||||
mint a `media_id`, emit `media.offer {media_id, mime, size, filename, kind}`,
|
||||
serve bytes on `media.pull`. (Base-class `extract_media`/`extract_images`
|
||||
already pull `MEDIA:`/image tags out of agent text and call these.)
|
||||
- **Interactive pickers** — `send_model_picker(...)`, `send_choice_picker(...)`,
|
||||
`send_clarify(...)`, `send_exec_approval(...)`, `send_slash_confirm(...)`:
|
||||
emit `picker.model` / `picker.choice` / `picker.clarify` / `picker.approval` /
|
||||
`picker.confirm` frames with options; store pending state keyed by
|
||||
`picker_id`; resolve on `picker.select`.
|
||||
- **`create_handoff_thread(chat_id, name)`** → create a thread id, register in
|
||||
channel directory, return it (used by cron "continuable" threads).
|
||||
|
||||
### Streaming hooks
|
||||
The main gateway drives delivery through the **legacy callback path**:
|
||||
- `stream_delta_callback` → `GatewayStreamConsumer` → `send()` (first) +
|
||||
`edit_message()` (updates) → `message.start` / `message.update`.
|
||||
- `tool_progress_callback` → progress queue → `send_progress_messages` →
|
||||
`send()` → `tool.*` frames (structured; classified in the adapter).
|
||||
- `interim_assistant_callback` → consumer `on_commentary` → `send()` →
|
||||
`commentary` frames.
|
||||
|
||||
The adapter tracks per-chat **turn state** (in-turn, current streaming
|
||||
`message_id`, last tool index) to classify outbound `send()` calls into
|
||||
`message` vs `tool.*` vs `commentary`. The exact classification markers are
|
||||
verified empirically in M2 (see `13-testing.md`).
|
||||
|
||||
## 3.4 WebSocket server (`ws_server.py`)
|
||||
|
||||
- Library: **`websockets`** (core dep, v15). `websockets.serve(handler, host,
|
||||
port, ssl=ctx)`.
|
||||
- **Handler** per connection:
|
||||
1. Await first frame; must be `hello {token, device_id, device_name, caps,
|
||||
fcm_token?}`. Verify token (constant-time) + allowlist. On failure: send
|
||||
`error {code:"auth"}` and close.
|
||||
2. On success: register in connection registry
|
||||
(`device_id → {ws, caps, fcm_token}`), send
|
||||
`hello.ack {server_caps, sync_cursor, channels[]}`.
|
||||
3. Loop: decode frames, dispatch to adapter inbound handlers.
|
||||
4. On close: deregister; if no devices remain, ensure pending outbox
|
||||
frames have push fired.
|
||||
- **Routing:** `emit(chat_id, frame)` → broadcast to **all** connected
|
||||
devices (no per-chat subscribe; single-user model). Global frames
|
||||
(`channel.*`, `status`) also broadcast to all.
|
||||
- **Heartbeat:** WS ping/pong + app-level `ping`/`pong`; dead peers reaped.
|
||||
- **Backpressure:** per-connection send queue with a bounded buffer; drop
|
||||
`message.update` (coalesce to latest) under pressure, never drop
|
||||
`message`/`tool.end`/`notification`.
|
||||
|
||||
## 3.5 State & storage (all under `get_hermes_home()/"android"`)
|
||||
|
||||
> Use `get_hermes_home()` from `hermes_constants` for **all** paths (profile-safe).
|
||||
> Never hardcode `~/.hermes`.
|
||||
|
||||
- `devices.db` — device registry (device_id, name, caps, fcm_token, ntfy_topic,
|
||||
last_seen, created).
|
||||
- `channels.db` — channel directory (chat_id, name, kind: default|channel|thread,
|
||||
parent_chat_id, created, is_default).
|
||||
- `outbox.db` — undelivered frames per chat_id + monotonic cursor.
|
||||
- `media/` — inbound + outbound media cache (reuse hermes `cache_*_from_bytes`
|
||||
dirs where possible).
|
||||
|
||||
## 3.6 Config resolution
|
||||
|
||||
- **Secrets (`.env`):** `ANDROID_TOKEN`, `ANDROID_FCM_SERVICE_ACCOUNT`,
|
||||
`ANDROID_FCM_SERVER_KEY`, `ANDROID_WS_CERT/KEY`, `NTFY_TOPIC` (if secret).
|
||||
- **Behavioral (`config.yaml` → `gateway.platforms.android.extra`):** `host`,
|
||||
`port`, `home_channel`, `allowed_users`, `push_backend`, `outbox_retention_hours`,
|
||||
`max_upload_bytes`, `tls`.
|
||||
- Env vars override `config.yaml` (hermes convention). Read secrets with the
|
||||
scope-aware `_get_scoped_secret` pattern (see `plugins/platforms/irc/adapter.py:42`)
|
||||
so multiplexed profiles don't leak each other's tokens.
|
||||
|
||||
## 3.7 Failure & lifecycle safety
|
||||
|
||||
- WS server bind failure → `_set_fatal_error("bind_failed", …, retryable=True)`.
|
||||
- All outbound sends are best-effort; a dead socket latches and the frame falls
|
||||
to the outbox.
|
||||
- `disconnect()` cancels the server task and closes sockets cleanly.
|
||||
- Token/PII redaction in all logs (hermes PII policy).
|
||||
@@ -0,0 +1,327 @@
|
||||
# 04 — Wire Protocol
|
||||
|
||||
JSON frames over a single WebSocket. One connection per device. Text frames are
|
||||
JSON; media travels as **binary frames** (chunked) referenced by a header frame.
|
||||
|
||||
## Envelope
|
||||
|
||||
Every frame:
|
||||
|
||||
```json
|
||||
{
|
||||
"v": 1,
|
||||
"id": 42, // optional; present on requests + their responses
|
||||
"type": "message", // frame type (below)
|
||||
"chat_id": "android:default", // optional; scope for chat-scoped frames
|
||||
"thread_id": "t_123", // optional
|
||||
"payload": { } // type-specific object
|
||||
}
|
||||
```
|
||||
|
||||
- `v` — protocol version (currently `1`). Server rejects unknown major versions.
|
||||
- `id` — request id (client-chosen). Responses/acks echo it. Events have no `id`.
|
||||
- `chat_id` / `thread_id` — top-level for convenience; may also be in `payload`.
|
||||
- Unknown `type`s are ignored (forward-compat); unknown `payload` fields ignored.
|
||||
|
||||
**Binary media frames** are not JSON. A media transfer is: one JSON header frame
|
||||
(`media.upload.start` / `media.pull` ack) followed by raw binary frames, then a
|
||||
JSON `media.upload.end` / final ack. See `07-media.md`.
|
||||
|
||||
## Server → App (events / responses)
|
||||
|
||||
### `hello.ack`
|
||||
Pairing succeeded.
|
||||
```json
|
||||
{"type":"hello.ack","payload":{
|
||||
"server_caps":{"streaming":true,"reasoning":true,"tools":true,"media":true,
|
||||
"search":true,"push":"fcm","pickers":true},
|
||||
"sync_cursor":1042,
|
||||
"channels":[{"chat_id":"android:default","name":"Default","kind":"default","is_default":true}]
|
||||
}}
|
||||
```
|
||||
|
||||
### `message`
|
||||
A final / standalone message.
|
||||
```json
|
||||
{"type":"message","chat_id":"android:default","thread_id":null,
|
||||
"payload":{
|
||||
"message_id":"m_9001","role":"assistant",
|
||||
"text":"Here is the answer…",
|
||||
"reasoning":"The user asked… so I will…", // optional; render ABOVE text
|
||||
"media":[{"media_id":"md_5","kind":"video","mime":"video/mp4","size":123456,
|
||||
"filename":"clip.mp4"}], // optional
|
||||
"reply_to":"m_8999", // optional
|
||||
"model":"qwen3-27b","tokens":11,"ts":1724000000000
|
||||
}}
|
||||
```
|
||||
`role` ∈ `user | assistant | system | cron`. `reasoning` present only when the
|
||||
agent produced reasoning and `show_reasoning` is on.
|
||||
|
||||
### `message.start` / `message.update` / `message.stop`
|
||||
Streaming a bubble. `update` carries the **full** current text (app replaces).
|
||||
```json
|
||||
{"type":"message.start","chat_id":"…","payload":{"message_id":"m_9002","role":"assistant"}}
|
||||
{"type":"message.update","chat_id":"…","payload":{"message_id":"m_9002","text":"partial…"}}
|
||||
{"type":"message.stop","chat_id":"…","payload":{"message_id":"m_9002","final_text":"full…",
|
||||
"reasoning":"…","model":"…","tokens":11}}
|
||||
```
|
||||
|
||||
### `commentary`
|
||||
Intermediate assistant beat (between tool iterations).
|
||||
```json
|
||||
{"type":"commentary","chat_id":"…","payload":{"message_id":"m_9003","text":"Let me inspect the repo first."}}
|
||||
```
|
||||
|
||||
### `tool.start` / `tool.progress` / `tool.end`
|
||||
**Structured** tool events. The app decides how much to show (everything /
|
||||
truncated / nothing).
|
||||
```json
|
||||
{"type":"tool.start","chat_id":"…","payload":{
|
||||
"index":3,"name":"terminal","preview":"pytest -q","args":{"command":"pytest -q"}}}
|
||||
{"type":"tool.progress","chat_id":"…","payload":{"index":3,"name":"terminal","note":"running…"}}
|
||||
{"type":"tool.end","chat_id":"…","payload":{"index":3,"name":"terminal","ok":true,"duration":12.4,
|
||||
"output_preview":"12 passed"}}
|
||||
```
|
||||
`args` may be large; the app truncates per its setting. `output_preview` is a
|
||||
short tail (full output is not streamed — it lives in agent history).
|
||||
|
||||
### `typing` / `typing.stop`
|
||||
```json
|
||||
{"type":"typing","chat_id":"…","payload":{"on":true}}
|
||||
```
|
||||
|
||||
### `notification`
|
||||
In-app banner (foreground) and/or push mirror (background).
|
||||
```json
|
||||
{"type":"notification","chat_id":"…","payload":{
|
||||
"kind":"channel_renamed","title":"ARIA","body":"Renamed topic to …","ts":1724000000000}}
|
||||
```
|
||||
`kind` ∈ `channel_renamed | channel_created | cron | approval | clarify | generic`.
|
||||
|
||||
### `picker.model` / `picker.choice` / `picker.clarify` / `picker.approval` / `picker.confirm`
|
||||
Interactive prompts. App renders a native picker; answers via `picker.select`.
|
||||
```json
|
||||
{"type":"picker.model","chat_id":"…","payload":{
|
||||
"picker_id":"pm_1","current_model":"qwen3-27b","current_provider":"local",
|
||||
"providers":[{"id":"local","label":"Local","models":[{"id":"qwen3-27b","label":"Qwen3 27B"}]}]}}
|
||||
{"type":"picker.choice","chat_id":"…","payload":{
|
||||
"picker_id":"pc_1","title":"Reasoning effort","choices":[
|
||||
{"value":"low","label":"Low"},{"value":"high","label":"High","is_current":true}]}}
|
||||
```
|
||||
|
||||
### `channel.list` / `channel.created` / `channel.renamed` / `channel.deleted`
|
||||
Channel directory updates. **Broadcast to all connected devices** (no explicit
|
||||
subscribe; the server pushes to every open WS).
|
||||
```json
|
||||
{"type":"channel.created","payload":{"chat_id":"android:chan_7","name":"Cron Reports",
|
||||
"kind":"channel","parent_chat_id":null}}
|
||||
```
|
||||
|
||||
### `history`
|
||||
Response to a `history` request. Returns a page of messages for a chat/thread.
|
||||
```json
|
||||
{"type":"history","id":20,"chat_id":"android:default","thread_id":null,
|
||||
"payload":{
|
||||
"messages":[
|
||||
{"message_id":"m_8990","role":"user","text":"Hi","ts":1723990000000},
|
||||
{"message_id":"m_8991","role":"assistant","text":"Hello!","reasoning":"…",
|
||||
"model":"qwen3-27b","tokens":8,"ts":1723990001000}
|
||||
],
|
||||
"has_more":true,
|
||||
"oldest_message_id":"m_8990"
|
||||
}}
|
||||
```
|
||||
`messages` are ordered oldest → newest. Paginate with `before_message_id` in the
|
||||
request. The app uses this to **populate the initial view** when a channel is
|
||||
opened (complements `sync`, which only replays undelivered outbox frames).
|
||||
|
||||
### `commands.catalog`
|
||||
Response to a `commands.catalog` request. Full slash-command list.
|
||||
```json
|
||||
{"type":"commands.catalog","id":21,"payload":{
|
||||
"commands":[
|
||||
{"name":"/new","description":"Start a new session","args_hint":"","category":"session"},
|
||||
{"name":"/model","description":"Switch model","args_hint":"<provider/model>","category":"config"},
|
||||
{"name":"/reasoning","description":"Toggle reasoning effort","args_hint":"[low|medium|high]","category":"config"},
|
||||
{"name":"/status","description":"Show session status","args_hint":"","category":"info"},
|
||||
{"name":"/cron","description":"Manage cron jobs","args_hint":"<list|add|rm>","category":"automation"},
|
||||
{"name":"/tools","description":"List available tools","args_hint":"","category":"info"}
|
||||
]}}
|
||||
```
|
||||
|
||||
### `commands.complete`
|
||||
Response to a `commands.complete` request. Autocomplete matches for a typed prefix.
|
||||
```json
|
||||
{"type":"commands.complete","id":22,"payload":{
|
||||
"prefix":"/mod",
|
||||
"matches":[
|
||||
{"name":"/model","description":"Switch model","args_hint":"<provider/model>"}
|
||||
]}}
|
||||
```
|
||||
|
||||
### `agent.busy` / `agent.idle`
|
||||
Agent lifecycle for a chat/thread. App shows a "thinking…" indicator on `busy`.
|
||||
```json
|
||||
{"type":"agent.busy","chat_id":"android:default","thread_id":null,
|
||||
"payload":{"reason":"processing"}}
|
||||
{"type":"agent.idle","chat_id":"android:default","thread_id":null,"payload":{}}
|
||||
```
|
||||
`reason` ∈ `processing | tool | waiting_input | cron`.
|
||||
|
||||
### `search.results`
|
||||
```json
|
||||
{"type":"search.results","id":7,"payload":{
|
||||
"query":"deploy","scope":"all","hits":[
|
||||
{"message_id":"m_123","chat_id":"android:chan_7","thread_id":null,
|
||||
"role":"assistant","snippet":"…deploy the service…","ts":1723900000000}]}}
|
||||
```
|
||||
|
||||
### `media.offer`
|
||||
Agent-sent media is available; app pulls bytes.
|
||||
```json
|
||||
{"type":"media.offer","chat_id":"…","payload":{
|
||||
"media_id":"md_5","kind":"video","mime":"video/mp4","size":123456,"filename":"clip.mp4"}}
|
||||
```
|
||||
|
||||
### `status`
|
||||
Gateway lifecycle / session info.
|
||||
```json
|
||||
{"type":"status","payload":{"state":"online","session":{"chat_id":"…","model":"…","tokens":11}}}
|
||||
```
|
||||
`state` ∈ `online | restarting | degraded`.
|
||||
|
||||
### `error`
|
||||
```json
|
||||
{"type":"error","id":7,"payload":{"code":"not_found","message":"chat_id unknown"}}
|
||||
```
|
||||
`code` ∈ `auth | not_found | rate_limited | media_too_large | unsupported | internal`.
|
||||
|
||||
### `pong`
|
||||
Keepalive reply to `ping`.
|
||||
|
||||
## App → Server (requests / actions)
|
||||
|
||||
### `hello`
|
||||
First frame; auth + caps.
|
||||
```json
|
||||
{"type":"hello","payload":{
|
||||
"token":"<ANDROID_TOKEN>","device_id":"dev_a1b2","device_name":"MIX 2S",
|
||||
"caps":{"min_protocol":1,"media":true,"push":"fcm"},
|
||||
"fcm_token":"<FCM token>","ntfy_topic":"<topic, if ntfy>"}}
|
||||
```
|
||||
|
||||
### `message.send`
|
||||
Send text (or a `/slash-command`).
|
||||
```json
|
||||
{"type":"message.send","id":10,"chat_id":"android:default","thread_id":null,
|
||||
"payload":{"text":"/model qwen3-27b","reply_to":"m_9001","media_refs":["mu_1"]}}
|
||||
```
|
||||
`media_refs` reference completed `media.upload`s to attach.
|
||||
|
||||
### `media.upload.start` / (binary) / `media.upload.end`
|
||||
See `07-media.md`.
|
||||
```json
|
||||
{"type":"media.upload.start","id":11,"payload":{
|
||||
"media_ref":"mu_1","kind":"image","mime":"image/jpeg","size":204800,"filename":"a.jpg"}}
|
||||
// … binary frames …
|
||||
{"type":"media.upload.end","id":11,"payload":{"media_ref":"mu_1","sha256":"…"}}
|
||||
```
|
||||
|
||||
### `media.pull`
|
||||
Request agent-sent media bytes.
|
||||
```json
|
||||
{"type":"media.pull","id":12,"payload":{"media_id":"md_5"}}
|
||||
// server replies: binary frames, then {"type":"media.pull.end","id":12,"payload":{"ok":true}}
|
||||
```
|
||||
|
||||
### `picker.select`
|
||||
Answer an interactive picker.
|
||||
```json
|
||||
{"type":"picker.select","id":13,"payload":{"picker_id":"pm_1","value":"local/qwen3-27b"}}
|
||||
```
|
||||
|
||||
### `channel.create` / `channel.rename` / `channel.set_default` / `channel.delete`
|
||||
```json
|
||||
{"type":"channel.create","id":14,"payload":{"name":"Cron Reports","kind":"channel"}}
|
||||
{"type":"channel.rename","id":15,"chat_id":"android:chan_7","payload":{"name":"Reports"}}
|
||||
{"type":"channel.set_default","id":16,"chat_id":"android:chan_7","payload":{}}
|
||||
```
|
||||
|
||||
### `search`
|
||||
```json
|
||||
{"type":"search","id":17,"payload":{"query":"deploy","scope":"all"}}
|
||||
{"type":"search","id":18,"payload":{"query":"deploy","scope":"chat","chat_id":"android:chan_7","thread_id":null}}
|
||||
```
|
||||
`scope` ∈ `all | chat`.
|
||||
|
||||
### `read.receipt`
|
||||
App → server: "user has viewed this message." Server stores the read state and
|
||||
broadcasts to other devices (for multi-device ✓✓ sync). The app uses it to
|
||||
mark messages as read locally (✓✓ on user bubbles).
|
||||
```json
|
||||
{"type":"read.receipt","payload":{"chat_id":"android:default","message_id":"m_9001"}}
|
||||
```
|
||||
|
||||
### `history`
|
||||
Load a page of messages for a chat/thread (initial open, scroll-up pagination).
|
||||
```json
|
||||
{"type":"history","id":20,"chat_id":"android:default","thread_id":null,
|
||||
"payload":{"before_message_id":"m_8990","limit":50}}
|
||||
```
|
||||
`before_message_id` — return messages older than this (omit for newest page).
|
||||
`limit` — max messages (default 50, max 200).
|
||||
|
||||
### `commands.catalog`
|
||||
Fetch the full slash-command catalog (for the `Menü` bottom sheet).
|
||||
```json
|
||||
{"type":"commands.catalog","id":21,"payload":{}}
|
||||
```
|
||||
|
||||
### `commands.complete`
|
||||
Autocomplete for a typed `/prefix`.
|
||||
```json
|
||||
{"type":"commands.complete","id":22,"payload":{"prefix":"/mod"}}
|
||||
```
|
||||
|
||||
### `agent.stop`
|
||||
Stop the current agent turn (abort generation / tool execution).
|
||||
```json
|
||||
{"type":"agent.stop","id":23,"chat_id":"android:default","thread_id":null,"payload":{}}
|
||||
```
|
||||
|
||||
### `agent.steer`
|
||||
Inject a steering message mid-turn (redirects the agent without a new turn).
|
||||
```json
|
||||
{"type":"agent.steer","id":24,"chat_id":"android:default","thread_id":null,
|
||||
"payload":{"text":"Actually, focus on the error case."}}
|
||||
```
|
||||
|
||||
### `sync`
|
||||
Reconnect catch-up. Replays **undelivered outbox frames** (frames sent while
|
||||
this device was offline). Does NOT load full history — use `history` for that.
|
||||
```json
|
||||
{"type":"sync","id":19,"payload":{"cursor":1042}}
|
||||
// server replays outbox frames with cursor > 1042, then {"type":"sync.done","id":19,"payload":{"cursor":1099}}
|
||||
```
|
||||
|
||||
### `fcm.register`
|
||||
Update push token.
|
||||
```json
|
||||
{"type":"fcm.register","payload":{"fcm_token":"<new>","ntfy_topic":"<topic>"}}
|
||||
```
|
||||
|
||||
### `ping`
|
||||
Keepalive. `{"type":"ping","payload":{"ts":1724000000000}}` → `pong`.
|
||||
|
||||
## Ordering & reliability
|
||||
|
||||
- Frames are ordered per connection (TCP/WS). Streaming `message.update` frames
|
||||
for a `message_id` are monotonic; the app may coalesce to the latest.
|
||||
- Terminal frames (`message`, `message.stop`, `tool.end`, `notification`,
|
||||
`picker.*`, `channel.*`, `agent.busy`, `agent.idle`, `history`,
|
||||
`commands.catalog`, `commands.complete`) are **never dropped** under
|
||||
backpressure; only intermediate `message.update`/`tool.progress` are coalesced.
|
||||
- Anything not delivered live goes to the **outbox** and is replayed by `sync`.
|
||||
- **Broadcast:** channel directory events (`channel.*`) and read-receipts are
|
||||
pushed to **all** connected devices for that gateway (no subscribe step).
|
||||
- Requests get exactly one response or `error` (matched by `id`).
|
||||
@@ -0,0 +1,143 @@
|
||||
# 05 — Streaming, Reasoning, Tools, Intermediate Messages
|
||||
|
||||
This doc covers the four "agent transparency" features and exactly how each is
|
||||
produced by the gateway and rendered by the app.
|
||||
|
||||
> The main hermes gateway delivers via the **legacy callback path** (not the
|
||||
> ACP-only event-native `render_message_event` path). We map the legacy
|
||||
> `send`/`edit_message`/progress calls to our WS frames.
|
||||
|
||||
## 5.1 Text streaming
|
||||
|
||||
**Gateway side.** The agent's `stream_delta_callback` feeds a
|
||||
`GatewayStreamConsumer` (`gateway/stream_consumer.py:156`). The consumer
|
||||
accumulates text and, at intervals / thresholds, calls:
|
||||
- `adapter.send(chat_id, text)` — first time a bubble is created.
|
||||
- `adapter.edit_message(chat_id, message_id, text)` — subsequent updates
|
||||
(each carries the **full** accumulated text).
|
||||
|
||||
**Adapter → frames.**
|
||||
- First `send()` of a turn segment → `message.start {message_id, role}`.
|
||||
- Each `edit_message()` → `message.update {message_id, text}` (full text).
|
||||
- Segment/turn finalization → `message.stop {message_id, final_text, reasoning?,
|
||||
model?, tokens?}` (or a final `message` frame when not streaming).
|
||||
|
||||
**App side.** Maintain a live bubble per `message_id`. On `message.update`,
|
||||
replace the bubble text (cheap: it's a full snapshot). On `message.stop`,
|
||||
finalize (attach reasoning/model/tokens footer, stop the cursor). Auto-scroll
|
||||
while the user is at the bottom.
|
||||
|
||||
**Streaming on/off.** Controlled by hermes `display.platforms.android.streaming`
|
||||
(default follows global). When off, the app just gets one final `message` frame.
|
||||
|
||||
## 5.2 Reasoning (shown *before* the message)
|
||||
|
||||
**Requirement:** reasoning appears as a block **above** the answer (like the
|
||||
reference screenshot's "Reasoning:" panel with a copy button).
|
||||
|
||||
**Gateway side.** hermes prepends reasoning to the final response when
|
||||
`show_reasoning` is enabled (`gateway/run.py:20089`). The format is stable and
|
||||
chosen by `reasoning_style` (`gateway/display_config.py:37`):
|
||||
- `code` (default): `💭 **Reasoning:**\n```\n<reasoning>\n```\n\n<response>`
|
||||
- `blockquote`: `> 💭 **Reasoning:**\n> …\n\n<response>`
|
||||
- `subtext`: `-# 💭 Reasoning\n-# …\n\n<response>` (Discord-style)
|
||||
|
||||
**Plugin config.** Set for the `android` platform:
|
||||
```yaml
|
||||
display:
|
||||
platforms:
|
||||
android:
|
||||
show_reasoning: true
|
||||
reasoning_style: code # we split on the code-fence form
|
||||
```
|
||||
|
||||
**Adapter split.** In `send()`, detect the `code`-style prefix and split:
|
||||
```
|
||||
prefix = "💭 **Reasoning:**\n```\n"
|
||||
# find the closing "\n```\n\n" after the prefix
|
||||
reasoning = text[len(prefix):close_idx]
|
||||
body = text[close_idx + len("\n```\n\n"):]
|
||||
```
|
||||
Emit `message {reasoning: <reasoning>, text: <body>, …}`. If no prefix is found
|
||||
(reasoning off / no reasoning), emit `message {text: …}` with no `reasoning`.
|
||||
|
||||
> Robustness: the split is a best-effort parse of a *stable, gateway-owned*
|
||||
> format. If the format ever changes, the fallback is "no reasoning field, full
|
||||
> text" — the app still shows the answer. Verified in M2 against the live
|
||||
> gateway.
|
||||
|
||||
**App side.** `ReasoningBlock` composable: collapsible, header "💭 Reasoning",
|
||||
monospace body, a **copy** button (matches reference). Rendered **above** the
|
||||
message body. Collapsed by default if long; expanded tap.
|
||||
|
||||
## 5.3 Tool output (app controls verbosity)
|
||||
|
||||
**Requirement:** hermes supports several tool-display modes; the **gateway sends
|
||||
full structured data**, and the **app** chooses how much to show
|
||||
(everything / truncated / nothing).
|
||||
|
||||
**Gateway side.** Tool activity flows through `tool_progress_callback` → the
|
||||
gateway progress queue → `send_progress_messages` (`gateway/run.py:4603`) →
|
||||
`adapter.send()`. The adapter receives tool lines during a turn.
|
||||
|
||||
**Adapter → frames.** The adapter classifies tool activity (via turn-state +
|
||||
line format) and emits **structured** frames — not pre-formatted strings:
|
||||
- `tool.start {index, name, preview, args}` — a tool call began.
|
||||
- `tool.progress {index, name, note}` — in-progress update (optional).
|
||||
- `tool.end {index, name, ok, duration, output_preview}` — completed.
|
||||
|
||||
`index` is a monotonic per-turn counter so the app correlates start→end.
|
||||
`args` is the full argument dict (the app truncates). `output_preview` is a
|
||||
short tail; full tool output is **not** streamed (it lives in agent history and
|
||||
is reachable via search).
|
||||
|
||||
**App side — the verbosity setting** (Settings → "Tool detail"):
|
||||
- **Everything** — show tool name, full args (collapsible), and output preview.
|
||||
- **Truncated** (default) — show `emoji name: "short preview"` one-liner,
|
||||
collapsible to expand.
|
||||
- **Nothing** — suppress `tool.*` frames entirely (clean chat).
|
||||
|
||||
Rendering: a `ToolCard` per tool, grouped under the message it belongs to, with
|
||||
a spinner while `tool.end` hasn't arrived, a ✓/✗ on completion, and duration.
|
||||
|
||||
> Classification detail: the exact way to distinguish a tool-progress `send()`
|
||||
> from a regular `send()`/commentary is confirmed empirically in M2 by running
|
||||
> the real gateway with a test WS client and observing the calls. The adapter
|
||||
> keeps a per-chat turn state machine (turn active, current streaming id, last
|
||||
> tool index) to make the classification deterministic.
|
||||
|
||||
## 5.4 Intermediate messages (commentary)
|
||||
|
||||
**Requirement:** show the agent's interim beats (e.g. "I'll inspect the repo
|
||||
first.") as distinct messages.
|
||||
|
||||
**Gateway side.** `interim_assistant_callback` → consumer `on_commentary`
|
||||
(`gateway/stream_consumer.py:518`) → delivered as its own message.
|
||||
|
||||
**Adapter → frames.** `commentary {message_id, text}`.
|
||||
|
||||
**App side.** Render as a **dimmed / smaller** bubble, visually distinct from
|
||||
final answers (e.g. reduced opacity, no model footer). It reads as a "beat" in
|
||||
the conversation, not a full reply.
|
||||
|
||||
## 5.5 Typing indicator
|
||||
|
||||
`send_typing` → `typing {on:true}`; the app shows an animated indicator in the
|
||||
chat header / above the composer until `typing.stop` or the first
|
||||
`message.start`.
|
||||
|
||||
## 5.6 Frame sequence for a typical turn
|
||||
|
||||
```
|
||||
app → message.send {text:"summarize the repo"}
|
||||
srv → typing {on:true}
|
||||
srv → message.start {message_id:m1}
|
||||
srv → message.update {m1, "Let me look…"} (streaming)
|
||||
srv → tool.start {index:1, name:terminal, args:{command:"ls -la"}}
|
||||
srv → tool.end {index:1, name:terminal, ok:true, duration:0.4}
|
||||
srv → commentary {m2, "Found 12 files."}
|
||||
srv → message.start {message_id:m3}
|
||||
srv → message.update {m3, "The repo has…"}
|
||||
srv → message.stop {m3, final_text:"…", reasoning:"…", model:"…", tokens:42}
|
||||
srv → typing {on:false}
|
||||
```
|
||||
@@ -0,0 +1,107 @@
|
||||
# 06 — Channels, Threads, Cron Delivery, Search
|
||||
|
||||
## 6.1 Concept model
|
||||
|
||||
The hermes gateway already models conversations as `SessionSource` with
|
||||
`chat_id` + `thread_id` + `chat_topic` (`gateway/platforms/base.py:7047`
|
||||
`build_source`). We map app concepts onto these existing primitives — **no new
|
||||
gateway identity concepts**.
|
||||
|
||||
| App concept | hermes primitive | Example |
|
||||
|---|---|---|
|
||||
| Default chat | home channel `chat_id` | `android:default` |
|
||||
| A thread (inside default chat) | `thread_id` under the default `chat_id` | `chat_id=android:default, thread_id=t_12` |
|
||||
| A user-created channel | a new `chat_id` | `android:chan_7` |
|
||||
| A thread inside a channel | `thread_id` under that `chat_id` | `chat_id=android:chan_7, thread_id=t_31` |
|
||||
|
||||
- **`chat_id`** = the conversation lane (a channel or the default chat).
|
||||
- **`thread_id`** = an optional sub-lane within a `chat_id` (topic-like).
|
||||
- The **channel directory** (plugin SQLite, `channels.db`) stores
|
||||
`{chat_id, name, kind: default|channel|thread, parent_chat_id, is_default,
|
||||
created}` and is the source of truth for the app's channel list.
|
||||
|
||||
## 6.2 Default chat
|
||||
|
||||
- On first connect, the plugin ensures a **default channel** exists:
|
||||
`chat_id = ANDROID_HOME_CHANNEL` (default `android:default`), `kind=default`,
|
||||
`is_default=true`, name "Default".
|
||||
- It is also the **cron home channel** (`cron_deliver_env_var=
|
||||
ANDROID_HOME_CHANNEL`), so `deliver=android` (bare) routes here.
|
||||
- The app opens the default chat on launch.
|
||||
|
||||
## 6.3 Threads (toggle for overview)
|
||||
|
||||
- **Requirement:** a default chat where the user can **activate threads** for a
|
||||
better overview, or not.
|
||||
- **Thread toggle** (per default chat, in the chat header menu):
|
||||
- **Threads OFF** — flat conversation; all messages use `thread_id=null`.
|
||||
- **Threads ON** — the app groups the conversation into topic-like lanes.
|
||||
Each new "topic" mints a `thread_id` (via `channel.create {kind:thread,
|
||||
parent_chat_id:android:default}` or an implicit thread). The UI shows a
|
||||
topic switcher (like Telegram topics) above the message list.
|
||||
- Threads are **app-organized** but **gateway-real**: each `thread_id` is a
|
||||
distinct hermes session lane, so context is isolated per thread and cron can
|
||||
target a specific thread.
|
||||
- The gateway's `create_handoff_thread` is used where hermes wants to open a
|
||||
named thread (e.g. continuable cron).
|
||||
|
||||
## 6.4 User-created channels (for cron delegation)
|
||||
|
||||
- **Requirement:** the user creates new channels so **cron job outputs can be
|
||||
delegated to them** instead of the default chat.
|
||||
- **`channel.create {name, kind:"channel"}`** → plugin mints
|
||||
`chat_id = android:chan_<n>`, stores in directory, broadcasts
|
||||
`channel.created` to all devices. The new channel appears in the channel list.
|
||||
- **`channel.rename` / `channel.set_default` / `channel.delete`** manage the
|
||||
directory (rename broadcasts `channel.renamed`; delete is soft — marks
|
||||
archived, keeps history for search).
|
||||
- **Cron targeting** (the key payoff): because the plugin registers
|
||||
`parse_target_ref_fn` and `cron_deliver_env_var`, cron jobs and the
|
||||
`send_message` tool can target any channel/thread:
|
||||
- `deliver="android"` → home (default) channel.
|
||||
- `deliver="android:android:chan_7"` → that channel.
|
||||
- `deliver="android:android:chan_7:t_31"` → that channel's thread.
|
||||
- In-chat: the agent's `cronjob` tool can be told "deliver to the *Cron
|
||||
Reports* channel"; the gateway resolves the name via the channel directory.
|
||||
- **In-app affordance:** each channel's menu has "Set as cron target" / shows a
|
||||
badge when a cron job points at it, and the channel name is offered in the
|
||||
`/cron` creation flow.
|
||||
|
||||
## 6.5 Cron delivery mechanics (how it works under the hood)
|
||||
|
||||
- Cron resolves delivery targets in `cron/scheduler.py:2148`
|
||||
(`_resolve_single_delivery_target`). For `platform:chat_id[:thread_id]` it
|
||||
calls `tools.send_message_tool.resolve_send_target`, which uses our
|
||||
`parse_target_ref_fn` to parse `android:<chat>[:<thread>]`.
|
||||
- Delivery then calls the **live adapter's `send(chat_id, text, …)`** (gateway
|
||||
running) → our WS `message` frame (or outbox+push if the app is offline).
|
||||
- Cron deliveries are framed with a `[Cron delivery: <name>]` header by hermes;
|
||||
the app can style cron messages distinctly (e.g. a small "⏰ <job name>"
|
||||
chip) using the `role:"cron"` / header.
|
||||
- **Mirror option:** hermes `cron.mirror_delivery` (default off) can also mirror
|
||||
a cron delivery into the origin session; we leave it off to keep channels
|
||||
clean.
|
||||
|
||||
## 6.6 Search
|
||||
|
||||
- **Requirement:** search with settings "search everywhere" / "search in this
|
||||
chat/channel".
|
||||
- **Backend:** hermes's session store is **SQLite + FTS5**
|
||||
(`hermes_state.py`, `hermes_state_search.py`). The plugin's `search.py` opens
|
||||
the session DB read-only and runs FTS5 queries.
|
||||
- **`search` frame** → `search.results`:
|
||||
- `scope:"all"` — search **everywhere** (all channels/threads/sessions).
|
||||
- `scope:"chat"` — restrict to the given `chat_id` (and optional `thread_id`).
|
||||
- **Result hit:** `{message_id, chat_id, thread_id, role, snippet, ts}`. The app
|
||||
renders a results list; tapping a hit navigates to that channel/thread and
|
||||
scrolls to + highlights the message.
|
||||
- **Query syntax:** plain text (FTS5). Optional `role:` / `channel:` qualifiers
|
||||
are a nice-to-have; v1 is plain-text + scope.
|
||||
- **Privacy:** search is local to the user's own hermes home; no data leaves the
|
||||
machine.
|
||||
|
||||
## 6.7 Channel list frame
|
||||
|
||||
`hello.ack` and `channel.*` frames carry the directory. App keeps a local copy
|
||||
(Room) and reconciles on `channel.*` events (merge, don't clobber — see
|
||||
`10-android-app.md` state rules).
|
||||
@@ -0,0 +1,96 @@
|
||||
# 07 — Media (upload, download, playback)
|
||||
|
||||
Media travels **over the WebSocket** as chunked binary frames (decision: no
|
||||
separate HTTP server; keeps the plugin to `websockets` only). Both directions
|
||||
use the same chunking.
|
||||
|
||||
## 7.1 Kinds & MIME
|
||||
|
||||
`kind` ∈ `image | audio | video | document | voice`.
|
||||
- `image` — `image/*` (jpg/png/webp/gif/heic).
|
||||
- `audio` — `audio/*` (mp3/m4a/ogg/…) — music.
|
||||
- `video` — `video/*` (mp4/webm/mov).
|
||||
- `document` — anything else (pdf, docx, zip, txt, …).
|
||||
- `voice` — short voice note (`audio/ogg; codecs=opus` typical).
|
||||
|
||||
The app sniffs `kind` from the picked file's MIME; the plugin re-sniffs on
|
||||
receipt (don't trust the client) using hermes helpers
|
||||
(`gateway/platforms/base.py` `_sniff_audio_ext`, `_looks_like_image`).
|
||||
|
||||
## 7.2 Inbound (app → agent) — `media.upload`
|
||||
|
||||
**Flow:**
|
||||
1. App picks a file (SAF) → reads size + MIME.
|
||||
2. App sends `media.upload.start {media_ref, kind, mime, size, filename}`.
|
||||
3. App streams the file as **binary WS frames** (e.g. 256 KiB chunks).
|
||||
4. App sends `media.upload.end {media_ref, sha256}`.
|
||||
5. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media
|
||||
cache via hermes `cache_*_from_bytes`:
|
||||
- image → `cache_image_from_bytes`
|
||||
- audio/voice → `cache_audio_from_bytes`
|
||||
- video → `cache_video_from_bytes`
|
||||
- document → `cache_document_from_bytes`
|
||||
→ returns a local path.
|
||||
6. The path is attached to the next `message.send` via `media_refs`, becoming
|
||||
`MessageEvent.media_urls` + `media_types`
|
||||
(`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then
|
||||
read the file.
|
||||
|
||||
**Limits:** `get_inbound_media_max_bytes()` / `validate_inbound_media_size`
|
||||
(`base.py:758/779`) enforce the cap; over-limit → `error {code:"media_too_large"}`.
|
||||
|
||||
**Backpressure:** large uploads use the WS flow control; the plugin reads
|
||||
binary frames into a temp file (not memory) to bound RAM.
|
||||
|
||||
## 7.3 Outbound (agent → app) — `media.offer` / `media.pull`
|
||||
|
||||
**Flow:**
|
||||
1. Agent produces/references media (e.g. generates an image, or replies with a
|
||||
`MEDIA:` tag / image URL). hermes base `extract_media` / `extract_images`
|
||||
(`base.py:4439/4884`) pull these out and call the adapter's
|
||||
`send_image / send_video / send_document / send_voice / send_image_file /
|
||||
send_multiple_images`.
|
||||
2. Adapter stages the file in the media cache, mints a `media_id`, and emits
|
||||
`media.offer {media_id, kind, mime, size, filename}` (inside/with the
|
||||
`message` frame's `media[]`).
|
||||
3. App sends `media.pull {media_id}`.
|
||||
4. Plugin streams the file as **binary WS frames**; ends with
|
||||
`media.pull.end {ok:true}`.
|
||||
5. App writes to its cache dir and hands the path to the player/viewer.
|
||||
|
||||
**Security:** `validate_media_delivery_path` (`base.py:1684`) + the media
|
||||
delivery root/recency/denied-path checks (`base.py:1312-1480`) ensure the plugin
|
||||
only serves files hermes is allowed to deliver (no arbitrary file read).
|
||||
|
||||
## 7.4 Live playback (AI-sent music/video)
|
||||
|
||||
**Requirement:** play AI-sent music and video **in the app**.
|
||||
|
||||
- **Audio (music/voice):** `ExoPlayer` (Media3). Inline player in the bubble
|
||||
(play/pause, seek, duration); a persistent **mini-player** for music that
|
||||
survives navigation. Voice notes play inline with a waveform.
|
||||
- **Video:** `ExoPlayer` inline player (play/pause, seek, fullscreen, PiP on
|
||||
Android). Streams from the local cache file after `media.pull`.
|
||||
- **Documents/images:** image viewer (zoom) / open-with for documents (Android
|
||||
`Intent.ACTION_VIEW` with a `FileProvider` URI; Desktop opens with the system
|
||||
handler).
|
||||
- **Desktop:** ExoPlayer is Android-only → the `MediaPlayer` expect/actual uses
|
||||
a desktop backend (see `11-desktop-app.md`): a `libmpv`/`mpv`-backed surface
|
||||
or a WebView fallback for video, and a desktop audio player for music.
|
||||
|
||||
## 7.5 Chunking parameters
|
||||
|
||||
- Chunk size: **256 KiB** (tunable).
|
||||
- Binary frames carry raw bytes only; framing/metadata is in the JSON header +
|
||||
end frames.
|
||||
- Reassembly is ordered (WS preserves order); a gap/corruption → abort +
|
||||
`error {code:"internal"}` + retry the whole transfer.
|
||||
- `sha256` in `media.upload.end` / a size check on pull verify integrity.
|
||||
|
||||
## 7.6 App-side storage
|
||||
|
||||
- Cache dir: app-specific external cache (`getExternalCacheDir()/media`).
|
||||
- LRU eviction by size (configurable, default 500 MB) so old media doesn't fill
|
||||
the device.
|
||||
- A `MediaRepository` tracks `{media_id, local_path, kind, size, ts}` in Room so
|
||||
bubbles can re-render players after process death.
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
# 08 — Push Notifications, Outbox & Sync
|
||||
|
||||
The gateway can't reach a sleeping phone directly. Push goes through a cloud
|
||||
relay. **Decision: FCM primary, ntfy fallback** (`ANDROID_PUSH_BACKEND`).
|
||||
|
||||
## 8.1 When push fires
|
||||
|
||||
- A frame targets a `chat_id` whose device is **disconnected** (WS closed) →
|
||||
drop to **outbox** + fire **push**.
|
||||
- Also fire push for high-priority foreground events the user should see even if
|
||||
the app is backgrounded (approvals, clarifies, cron completions) — the app
|
||||
decides whether to also show an in-app banner.
|
||||
- If the device is **connected**, no push (the live frame is enough).
|
||||
|
||||
## 8.2 `PushBackend` interface (`push.py`)
|
||||
|
||||
```python
|
||||
class PushBackend(Protocol):
|
||||
name: str
|
||||
async def send(self, *, device_id: str, chat_id: str,
|
||||
title: str, body: str,
|
||||
data: dict) -> bool: ...
|
||||
def configured(self) -> bool: ...
|
||||
```
|
||||
|
||||
Selected at adapter init by `ANDROID_PUSH_BACKEND` (`fcm` default, `ntfy`).
|
||||
|
||||
### 8.2.1 `FcmBackend` (primary)
|
||||
- **FCM HTTP v1 API** via `httpx` (core dep). Auth = Firebase **service
|
||||
account** (`ANDROID_FCM_SERVICE_ACCOUNT` JSON path) → mint a short-lived
|
||||
OAuth2 access token (cached, refreshed before expiry).
|
||||
- Fallback: legacy **server key** (`ANDROID_FCM_SERVER_KEY`) if no service
|
||||
account (simpler, but legacy).
|
||||
- Target = the device's **FCM token** (registered via `hello` /
|
||||
`fcm.register`, stored in `devices.db`).
|
||||
- Payload:
|
||||
- `notification {title, body}` → system notification (tap → open app).
|
||||
- `data {chat_id, thread_id, kind, message_id, cursor}` → app uses to `sync`.
|
||||
- **Data-only option:** for background catch-up, send a data message (silent) so
|
||||
the app's `FirebaseMessagingService` wakes a foreground service and syncs
|
||||
without a visible notification (used for non-urgent updates).
|
||||
- Batch: FCM allows up to 500 tokens/message; we send per-device (1 user, few
|
||||
devices).
|
||||
|
||||
### 8.2.2 `NtfyBackend` (fallback, self-host friendly)
|
||||
- Reuses hermes's existing ntfy publish path (hermes ships an ntfy adapter).
|
||||
- Publish to `NTFY_TOPIC` on `NTFY_SERVER_URL` (default `https://ntfy.sh`) via
|
||||
`httpx` POST, with an `X-Title` / `X-Message` / `X-Tag` / `X-Priority` and a
|
||||
JSON `data` attachment (`{chat_id, thread_id, kind, cursor}`).
|
||||
- The app subscribes to the topic (via an ntfy client lib or a lightweight
|
||||
listener) to receive pushes without Firebase.
|
||||
- Best for users who self-host ntfy and want zero Firebase.
|
||||
|
||||
## 8.3 Outbox (`outbox.py`)
|
||||
|
||||
- SQLite `outbox.db`: rows `{cursor (monotonic), chat_id, thread_id, frame_json,
|
||||
ts, delivered}`.
|
||||
- **Write** on every outbound frame that has no live subscriber (or always, then
|
||||
mark delivered on live send — simpler + crash-safe).
|
||||
- **Retention:** `outbox_retention_hours` (default 72h); prune on write.
|
||||
- **Cursor:** global monotonic int; `hello.ack` returns the current cursor;
|
||||
`sync {cursor}` replays rows with `cursor > given`.
|
||||
- Bounded: if the outbox grows past a cap (e.g. 5k rows), oldest are pruned and
|
||||
a `notification {kind:"generic", body:"Older messages pruned"}` is sent.
|
||||
|
||||
## 8.4 Sync (reconnect catch-up)
|
||||
|
||||
1. App reconnects → `hello` → `hello.ack {sync_cursor}`.
|
||||
2. App sends `sync {cursor: <last seen>}`.
|
||||
3. Server replays outbox frames (in cursor order) → app applies them (messages,
|
||||
tools, channels, media offers).
|
||||
4. Server sends `sync.done {cursor: <new>}`.
|
||||
5. App updates its local cursor (Room) — idempotent (dedupe by `message_id`).
|
||||
|
||||
This makes the app **eventually consistent** across disconnects, restarts, and
|
||||
gateway restarts.
|
||||
|
||||
## 8.5 App-side push handling (Android)
|
||||
|
||||
- **`FirebaseMessagingService.onMessageReceived`** (foreground): show in-app
|
||||
banner + optionally sync.
|
||||
- **`onNewToken`** → `fcm.register` the new token.
|
||||
- **Background data message** → start a **foreground service** (low-priority,
|
||||
notification channel "Sync") → open WS → `sync` → stop.
|
||||
- **Notification channels** (Android 8+): one per chat so cron channels can have
|
||||
their own style/priority (e.g. "Cron Reports" = high, "Default" = default).
|
||||
Tapping a notification deep-links to the chat/thread.
|
||||
- **ntfy mode:** a foreground service maintains the ntfy subscription; incoming
|
||||
events trigger the same sync path.
|
||||
|
||||
## 8.6 In-app banners (foreground)
|
||||
|
||||
`notification` frames (e.g. `channel_renamed`, `approval`, `clarify`, `cron`)
|
||||
render as a transient banner above the composer (like the reference screenshot's
|
||||
"ARIA hat Thema … umbenannt"). Dismissible; high-priority ones (approval/clarify)
|
||||
persist until acted on.
|
||||
|
||||
## 8.7 Security
|
||||
|
||||
- FCM tokens are per-device, revocable; stored only in `devices.db`.
|
||||
- Push payloads carry **no secrets** and no full message bodies larger than a
|
||||
short preview (privacy on lock screen). Full content is fetched via `sync`
|
||||
over the authenticated WS.
|
||||
- ntfy: use a **private topic + auth token** for any real trust boundary (hermes
|
||||
ntfy adapter guidance).
|
||||
@@ -0,0 +1,95 @@
|
||||
# 09 — Pairing, Auth & Security
|
||||
|
||||
## 9.1 Threat model
|
||||
|
||||
- **Trusted domain:** a personal agent on the user's own machine; one owner, a
|
||||
few of their own devices (phone + desktop).
|
||||
- **Primary risks:** (a) an unauthorized device connecting to the WS and
|
||||
reading/driving the agent; (b) eavesdropping on the WS in transit; (c) token
|
||||
leakage in logs; (d) arbitrary file read via media pull.
|
||||
- **Not addressing (v1):** multi-tenant isolation, adversarial multi-user abuse,
|
||||
E2E encryption.
|
||||
|
||||
## 9.2 Pairing flow
|
||||
|
||||
1. **Generate a token.** `hermes gateway setup` (our `interactive_setup`) either
|
||||
uses an existing `ANDROID_TOKEN` or generates a fresh high-entropy token
|
||||
(e.g. 32 bytes → 64 hex chars) and stores it in `.env`.
|
||||
2. **Present to the app.** Two options:
|
||||
- **QR code:** the setup prints a QR encoding
|
||||
`iris://pair?host=<lan-ip>&port=8790&token=<token>` (or a WSS URL). The
|
||||
phone scans it with the app's camera (or a system scanner) → pre-fills
|
||||
settings.
|
||||
- **Manual:** user types the server URL + token in the app's Connect screen.
|
||||
3. **App connects.** First WS frame is `hello {token, device_id, device_name,
|
||||
caps, fcm_token?}`.
|
||||
4. **Server verifies.** Constant-time compare of `token` vs `ANDROID_TOKEN`
|
||||
(`hmac.compare_digest`). Optionally check `device_id` against
|
||||
`ANDROID_ALLOWED_USERS` (if set) or `ANDROID_ALLOW_ALL_USERS`.
|
||||
5. **On success:** register the device in `devices.db`, send `hello.ack`.
|
||||
**On failure:** send `error {code:"auth"}` and close.
|
||||
|
||||
`device_id` is a stable, app-generated UUID (persisted in the app's
|
||||
secure storage). It identifies the device for routing + push, **not** as a
|
||||
security principal (the token is).
|
||||
|
||||
## 9.3 Auth model
|
||||
|
||||
- **Token = the security principal.** Any connection presenting the valid
|
||||
`ANDROID_TOKEN` is authorized (it's the user's own token).
|
||||
- **Allowlist (optional):** `ANDROID_ALLOWED_USERS` (comma-separated
|
||||
`device_id`s) restricts which *devices* may connect even with the token —
|
||||
useful if the token is shared. `ANDROID_ALLOW_ALL_USERS=true` disables the
|
||||
allowlist (dev only).
|
||||
- **Per-device tokens (stretch):** mint a unique token per device at pairing
|
||||
(revocable) instead of one shared token. v1 uses the shared token + optional
|
||||
device allowlist.
|
||||
- **Re-pairing:** rotating `ANDROID_TOKEN` invalidates all devices; they must
|
||||
re-pair. `hermes android pair` (stretch CLI) re-issues + prints a new QR.
|
||||
|
||||
## 9.4 Transport security
|
||||
|
||||
- **Default (LAN/dev):** plain `ws://` on the trusted LAN. Fine for a home
|
||||
network.
|
||||
- **WSS (recommended for remote):** set `ANDROID_WS_CERT` / `ANDROID_WS_KEY`
|
||||
(self-signed or CA-signed). The app pins/accepts the cert (self-signed → user
|
||||
confirms fingerprint on first pair, like a SSH host key).
|
||||
- **Remote reachability options** (documented, user's choice):
|
||||
- **Tailscale / WireGuard** (recommended): gateway gets a stable tailnet IP;
|
||||
app connects over the private mesh. No public exposure.
|
||||
- **Reverse proxy / tunnel** (Caddy, Cloudflare Tunnel, ngrok): terminate TLS
|
||||
at the edge, forward WS to `127.0.0.1:8790`.
|
||||
- **Public bind** (`0.0.0.0`) + WSS + strong token — last resort.
|
||||
- The app stores the server URL + (for self-signed) the pinned cert fingerprint
|
||||
in secure storage.
|
||||
|
||||
## 9.5 Secret & PII handling
|
||||
|
||||
- **Tokens/keys never logged.** Redact `ANDROID_TOKEN`, FCM tokens/keys, ntfy
|
||||
tokens in all log output (hermes PII policy; `agent/redact.py` patterns).
|
||||
- **`device_id`** is a random UUID (not PII). `device_name` is user-chosen.
|
||||
- **Media pull** is gated by hermes `validate_media_delivery_path` + delivery
|
||||
root/recency/denied-path checks (`gateway/platforms/base.py:1684`) — the
|
||||
plugin can only serve files hermes is allowed to deliver (no arbitrary file
|
||||
read).
|
||||
- **Search** is local-only (user's own hermes home); no data leaves the machine.
|
||||
|
||||
## 9.6 Profile safety
|
||||
|
||||
- All plugin state lives under `get_hermes_home()/"android"` (profile-aware).
|
||||
- Secrets are read with the scope-aware `_get_scoped_secret` pattern (see
|
||||
`plugins/platforms/irc/adapter.py:42`) so multiplexed profiles don't leak
|
||||
each other's tokens (fail-closed under `gateway.multiplex_profiles`).
|
||||
- The WS bind uses a scoped lock (`gateway.status.acquire_scoped_lock`) so two
|
||||
profiles can't bind the same port/identity.
|
||||
|
||||
## 9.7 Hardening checklist
|
||||
|
||||
- [ ] Constant-time token compare.
|
||||
- [ ] Bounded per-connection send buffer + rate limit on inbound frames.
|
||||
- [ ] Reject oversized frames / uploads (`max_upload_bytes`).
|
||||
- [ ] Verify media sha256 + re-sniff MIME (don't trust client).
|
||||
- [ ] Redact all secrets in logs.
|
||||
- [ ] WSS + cert pinning for remote.
|
||||
- [ ] Outbox retention cap + prune.
|
||||
- [ ] Fail-closed secret reads under multiplexing.
|
||||
@@ -0,0 +1,197 @@
|
||||
# 10 — Android App (Kotlin + Jetpack Compose)
|
||||
|
||||
Native client. Lives in the Compose Multiplatform project at `app/`; the bulk of
|
||||
the code is in `app/shared` (commonMain) so the Desktop app reuses it.
|
||||
|
||||
## 10.1 Tech stack
|
||||
|
||||
| Concern | Choice |
|
||||
|---|---|
|
||||
| Language | Kotlin |
|
||||
| UI | Jetpack Compose (Material 3), Compose Navigation |
|
||||
| Async | Kotlinx Coroutines + Flow |
|
||||
| WS client | OkHttp (`WebSocketListener`) |
|
||||
| JSON | kotlinx-serialization |
|
||||
| Local DB | **SQLDelight** (KMP; channels, messages cache, media index, sync cursor, settings) |
|
||||
| Media playback | Media3 **ExoPlayer** (audio + video) |
|
||||
| Push | Firebase Messaging (FCM) [primary] / ntfy listener [fallback] |
|
||||
| DI | Hilt |
|
||||
| Images | Coil |
|
||||
| minSdk / target | **26** / 34 (test device: MIX 2S, API 29) |
|
||||
|
||||
## 10.2 Module layout
|
||||
|
||||
```
|
||||
app/shared/src/
|
||||
├── commonMain/kotlin/iris/
|
||||
│ ├── protocol/ # frame data classes (mirror of gateway protocol.py)
|
||||
│ ├── net/ # GatewayClient (OkHttp WS), reconnect, heartbeat, dispatch
|
||||
│ ├── data/ # ChannelRepository, MessageRepository, MediaRepository,
|
||||
│ │ # SearchRepository, SettingsRepository (SQLDelight + WS)
|
||||
│ ├── state/ # ViewModels: ChatListVM, ChatVM, ComposerVM, SettingsVM
|
||||
│ ├── ui/
|
||||
│ │ ├── theme/ # Material 3 theme (dark default), type, color
|
||||
│ │ ├── components/ # MessageBubble, ReasoningBlock, ToolCard, MediaPlayer,
|
||||
│ │ │ # ChannelRow, Composer, PickerSheet, SearchBar, Banner
|
||||
│ │ └── screens/ # ConnectScreen, ChatListScreen, ChatScreen,
|
||||
│ │ # SearchScreen, SettingsScreen, ChannelMenu
|
||||
│ └── util/ # time formatting, markdown, id gen
|
||||
├── androidMain/kotlin/iris/
|
||||
│ ├── platform/ # MediaPlayer actual (ExoPlayer), MediaPicker (SAF),
|
||||
│ │ # Notifications, SecureStore (EncryptedSharedPreferences)
|
||||
│ ├── fcm/ # FirebaseMessagingService, foreground sync service
|
||||
│ └── IrisApp.kt # Application (Hilt), notification channels
|
||||
└── androidApp/ # MainActivity, AndroidManifest, res, google-services
|
||||
```
|
||||
|
||||
## 10.3 `GatewayClient` (net)
|
||||
|
||||
- OkHttp `WebSocket` to `ws(s)://host:port/ws`.
|
||||
- **Reconnect:** exponential backoff + jitter; on reconnect send `hello` then
|
||||
`sync {cursor}` (replays undelivered outbox only).
|
||||
- **Initial channel open:** on first open of a chat/thread, send `history`
|
||||
(newest page) to populate the view. `sync` does NOT load history.
|
||||
- **Heartbeat:** send `ping` every 20s; reap on missed `pong` (3×) → reconnect.
|
||||
- **Request/response:** map of `id → CompletableDeferred`; events → a
|
||||
`SharedFlow<Frame>` consumed by repositories.
|
||||
- **Backpressure:** collect frames on a bounded channel; coalesce
|
||||
`message.update` per `message_id` (keep latest) to avoid flooding the UI.
|
||||
- **Thread:** all frame emission on a single dispatcher; UI observes via Flow.
|
||||
|
||||
## 10.4 State rules (from hermes desktop AGENTS.md — apply here)
|
||||
|
||||
- **Server is authoritative** for channels/messages/cursor; the app's Room copy
|
||||
is a **cache**. Reconcile (merge, don't clobber) on `channel.*`/`sync`.
|
||||
- **Optimistic then honest:** send a message → show it immediately (pending),
|
||||
roll back visibly on `error`, authoritative `sync` gets the last word.
|
||||
- **Guard against the past:** generation counters / request tokens so a stale
|
||||
response never overwrites newer intent.
|
||||
- **Isolate the foreground:** only the visible chat publishes into the shared
|
||||
view; background chats update their own cache quietly.
|
||||
- **Coalesce noise, flush signal:** batch cosmetic updates; let terminal
|
||||
transitions (turn done, needs input, failed) reach the user immediately.
|
||||
|
||||
## 10.5 Feature implementation (your checklist)
|
||||
|
||||
### Input box, auto-grow (max height)
|
||||
- Compose `BasicTextField` inside a `Box` with
|
||||
`Modifier.heightIn(min = 1.line, max = 160.dp)`. Grows with lines, caps at
|
||||
160dp, then scrolls internally.
|
||||
- Bottom bar (matches reference): `Menü` button (left), emoji button, the
|
||||
auto-grow field, attach (paperclip), mic/send (right). Send on Enter
|
||||
(configurable: Enter=send vs Enter=newline).
|
||||
|
||||
### Menu button → all slash commands
|
||||
- `Menü` opens a **bottom sheet** listing the command catalog. The catalog is
|
||||
served by the gateway via `commands.catalog` request → response with
|
||||
`{name, description, args_hint, category}` per command, so it always matches
|
||||
hermes (`/new`, `/model`, `/reasoning`, `/status`, `/cron`, `/tools`, …).
|
||||
- Typing `/` in the field shows **autocomplete** via `commands.complete`
|
||||
request (gateway matches the typed prefix). Selecting inserts `/cmd `.
|
||||
- Selecting a command sends `message.send {text:"/cmd args"}`.
|
||||
- **Interactive commands** (`/model`, `/reasoning`, `/fast`, approvals,
|
||||
clarifies) render as **native pickers** from `picker.*` frames (a dialog /
|
||||
sheet with the options; answer via `picker.select`).
|
||||
|
||||
### Tool output (app-controlled verbosity)
|
||||
- `ToolCard` renders `tool.start/progress/end` frames.
|
||||
- **Settings → "Tool detail": Everything / Truncated / Nothing.**
|
||||
- Everything: name + full args (collapsible) + output preview.
|
||||
- Truncated (default): `emoji name: "short preview"` one-liner, expandable.
|
||||
- Nothing: suppress tool frames.
|
||||
- Spinner while running; ✓/✗ + duration on `tool.end`.
|
||||
|
||||
### Reasoning before message
|
||||
- `ReasoningBlock` (collapsible, "💭 Reasoning" header, monospace body, **copy**
|
||||
button) rendered **above** the message body from the `reasoning` field.
|
||||
Matches the reference screenshot.
|
||||
|
||||
### Intermediate messages
|
||||
- `commentary` frames → dimmed/smaller bubble, distinct from final answers.
|
||||
|
||||
### Agent busy / stop / steer
|
||||
- `agent.busy` → show "thinking…" indicator in chat header (animated dots).
|
||||
- `agent.idle` → clear indicator.
|
||||
- **Stop button** (appears in header while busy): sends `agent.stop`.
|
||||
- **Steer:** while busy, the composer accepts input; sending it calls
|
||||
`agent.steer` (injects mid-turn) rather than queuing a new `message.send`.
|
||||
|
||||
### Threading + channels
|
||||
- **Channel list** (drawer on single-pane; left rail on two-pane) = default chat
|
||||
+ user channels (avatar, name, last-message preview, unread badge, active
|
||||
highlight bar) — matches the reference left sidebar.
|
||||
- **Thread toggle** in the default chat header: "Threads on/off". On → topic
|
||||
switcher above the message list (each topic = a `thread_id`).
|
||||
- **New channel** (FAB / channel-list menu) → `channel.create` → appears in list;
|
||||
menu offers "Set as cron target".
|
||||
|
||||
### Search
|
||||
- Search bar (chat header or top) with a **scope toggle**: "Search everywhere" /
|
||||
"Search in this chat/channel". → `search` frame → results list → tap jumps to
|
||||
the message (navigate + highlight).
|
||||
|
||||
### Attach media
|
||||
- Paperclip → system pickers (Photos / Files / Audio / Video / Docs) via SAF.
|
||||
- Selected files show as **preview chips** in the composer (thumbnail + name +
|
||||
remove). On send: `media.upload` (chunked) for each, then `message.send` with
|
||||
`media_refs`.
|
||||
|
||||
### Voice input (mic button)
|
||||
- Mic button (right of composer, toggles to send when text is present).
|
||||
- Tap → request `RECORD_AUDIO` permission → start recording (MediaRecorder,
|
||||
OGG/Opus, 44.1 kHz mono).
|
||||
- While recording: timer + waveform; tap again to stop.
|
||||
- On stop: file becomes a **preview chip** (audio, kind=`voice`) in the
|
||||
composer, same as attached media. Send → `media.upload` (chunked) →
|
||||
`message.send` with `media_refs`.
|
||||
- Hermes receives it as an audio attachment; the agent's STT (if configured)
|
||||
transcribes it. No client-side STT.
|
||||
|
||||
### Push notifications
|
||||
- FCM service (see `08-push.md`): foreground banner + background foreground
|
||||
service → `sync`. Notification channel per chat. Tap → deep-link to chat.
|
||||
- ntfy fallback: foreground service maintains the subscription.
|
||||
|
||||
### Live playback
|
||||
- AI-sent audio/video → `media.pull` → cache file → **ExoPlayer** inline player
|
||||
(audio: mini-player; video: inline + fullscreen + PiP). Documents/images →
|
||||
viewer / open-with.
|
||||
|
||||
## 10.6 Layout (Telegram-style, per reference image)
|
||||
|
||||
**Two layout modes** (decision: user-toggleable, **single-pane default**):
|
||||
- **Single-pane (default on phones):** chat full-screen; channel list in a
|
||||
swipeable drawer (hamburger / edge swipe).
|
||||
- **Two-pane (Telegram-style, like the reference):** persistent left channel
|
||||
rail + chat. Auto-enabled on tablets / large screens; toggleable in Settings.
|
||||
|
||||
**Chat screen anatomy (matches reference):**
|
||||
- **Header:** back (single-pane), avatar, name + "Bot" subtitle, edit + overflow
|
||||
(⋮) menu (thread toggle, channel menu, set cron target, clear).
|
||||
- **Message list:** date separators ("7. August"); user bubbles **right**
|
||||
(accent color, ✓✓ read receipts); agent bubbles **left** (surface color) with
|
||||
optional ReasoningBlock + ToolCards + media + model/token footer
|
||||
("Qwen3-… · 11% · ~") + timestamp.
|
||||
- **In-app banner** above the composer (e.g. "renamed topic").
|
||||
- **Composer:** `Menü` / emoji / auto-grow input / attach / mic-send.
|
||||
|
||||
**Theme:** dark by default (reference is dark); Material 3; optional dynamic
|
||||
color. Accent = user's chosen brand color (default indigo, like the reference).
|
||||
|
||||
## 10.7 SQLDelight schema (cache)
|
||||
|
||||
- `channels(chat_id PK, name, kind, parent_chat_id, is_default, last_preview,
|
||||
last_ts, unread)`.
|
||||
- `messages(id PK, chat_id, thread_id, role, text, reasoning, model, tokens,
|
||||
ts, status[pending|sent|read], media_json)`.
|
||||
- `media(media_id PK, local_path, kind, mime, size, ts)`.
|
||||
- `meta(key PK, value)` — sync cursor, settings, device_id, server url, pinned
|
||||
cert fingerprint.
|
||||
|
||||
## 10.8 Onboarding / Connect screen
|
||||
|
||||
- First launch → **Connect**: server URL + token (or scan QR). "Test connection"
|
||||
does a real `hello` (not just a TCP probe — per hermes desktop guidance, the
|
||||
auth leg must be exercised). On success → save (secure storage) → main.
|
||||
- States: connecting / connected / reconnecting / degraded / auth-failed — each
|
||||
with honest copy and a way out.
|
||||
@@ -0,0 +1,75 @@
|
||||
# 11 — Desktop App (Kotlin + Compose Multiplatform)
|
||||
|
||||
The desktop app is **the Android app, tweaked for a big screen**. It reuses the
|
||||
entire `app/shared` module (protocol, network, repositories, state, most UI) and
|
||||
only adds desktop platform services + a wider default layout.
|
||||
|
||||
## 11.1 What's shared vs desktop-specific
|
||||
|
||||
| Layer | Shared (commonMain) | Desktop-specific (desktopMain) |
|
||||
|---|---|---|
|
||||
| Protocol / WS client | ✅ | — |
|
||||
| Repositories / state | ✅ | — |
|
||||
| Most Compose UI | ✅ | layout tweaks, keyboard shortcuts |
|
||||
| Push | — | **tray icon + OS notifications** (no FCM) |
|
||||
| Media playback | `MediaPlayer` interface | **desktop player** actual |
|
||||
| File picker | `MediaPicker` interface | **desktop file dialog** actual |
|
||||
| Window | — | resizable window, always-on-top, global hotkey |
|
||||
| Secure store | `SecureStore` interface | keyring / encrypted file actual |
|
||||
|
||||
## 11.2 `desktopMain` platform services
|
||||
|
||||
- **Push → tray + OS notifications.** Desktop is assumed reachable (persistent
|
||||
WS), so no FCM. A **system tray icon** shows connection state + unread count;
|
||||
OS notifications (Java Desktop / `SystemTray` + a cross-platform notifier)
|
||||
fire for background chats / approvals / cron. Clicking focuses the window and
|
||||
deep-links to the chat.
|
||||
- **Media playback → `MediaPlayer` actual.** ExoPlayer is Android-only. Desktop
|
||||
uses a `libmpv`/`mpv`-backed Compose surface for video (and audio), with a
|
||||
WebView-based fallback if `mpv` isn't available. Same `MediaPlayer` interface
|
||||
the Android `ExoPlayer` actual implements, so UI code is identical.
|
||||
- **File picker → `MediaPicker` actual.** A native file dialog (Compose
|
||||
Desktop `FileChooser` / AWT `JFileChooser`) returning local file paths, then
|
||||
the same `media.upload` chunked path.
|
||||
- **Window.** Resizable, remembers size/position, optional always-on-top, a
|
||||
global hotkey to focus/show. Minimize-to-tray option.
|
||||
- **Secure store → `SecureStore` actual.** OS keychain (macOS Keychain, Linux
|
||||
Secret Service / encrypted file, Windows Credential Manager) or an encrypted
|
||||
file for the token + pinned cert.
|
||||
|
||||
## 11.3 Layout (big-screen tweaks)
|
||||
|
||||
- **Two-pane is the default** (persistent left channel rail + chat), since there
|
||||
is width. Single-pane still available via the same toggle.
|
||||
- **Wider chat column** with comfortable max line length; optional **third
|
||||
pane** (inspector: session info, model, tokens, tool detail, channel settings).
|
||||
- **Larger type scale** and spacing for desktop; hover states; mouse + keyboard
|
||||
first.
|
||||
- **Keyboard shortcuts:**
|
||||
- `/` focus the composer and open the command menu.
|
||||
- `Ctrl/Cmd+K` command palette (all slash commands + actions).
|
||||
- `Ctrl/Cmd+N` new channel; `Ctrl/Cmd+T` new thread.
|
||||
- `Ctrl/Cmd+F` search (with the all/this-chat scope toggle).
|
||||
- `↑/↓` navigate channel list; `Enter` open.
|
||||
- `Esc` close sheet/dialog (one cancel gesture does exactly one thing).
|
||||
- **Multi-window (stretch):** open a channel in a separate window / pop-out.
|
||||
|
||||
## 11.4 Distribution
|
||||
|
||||
- **Compose Desktop** → native binaries via **jpackage** (or a plain app image):
|
||||
Linux (.deb/.AppImage), macOS (.dmg), Windows (.msi/.exe).
|
||||
- Bundles the JRE. Auto-update is a stretch goal (v1: manual download).
|
||||
- The desktop app connects to the **same** gateway WS server as the phone (the
|
||||
user's home server / Tailscale). It does **not** spawn its own backend (unlike
|
||||
hermes's existing Electron desktop, which spawns `hermes serve`) — our
|
||||
desktop is a pure client of the messaging gateway, matching the Android app.
|
||||
|
||||
## 11.5 Parity checklist (same functionality as Android)
|
||||
|
||||
- [ ] Streaming, reasoning block, tool cards (3-level verbosity), commentary.
|
||||
- [ ] Channels + threads + new channel + cron target.
|
||||
- [ ] Search (all / this-chat).
|
||||
- [ ] Media attach + live playback (via desktop player).
|
||||
- [ ] Slash command menu + autocomplete + interactive pickers.
|
||||
- [ ] Push (tray + OS notifications) + outbox/sync.
|
||||
- [ ] Pairing/Connect screen (URL + token, WSS cert pin).
|
||||
@@ -0,0 +1,138 @@
|
||||
# 12 — Toolchain Setup
|
||||
|
||||
First-time setup on a machine (verified baseline: CachyOS/Arch, `pacman`,
|
||||
`uv` present, ADB present, no JDK/SDK/Gradle).
|
||||
|
||||
## 12.1 JDK 17
|
||||
|
||||
```bash
|
||||
pacman -S jdk17-openjdk
|
||||
java -version # expect 17.x
|
||||
```
|
||||
(Compose Multiplatform + current AGP are happy on JDK 17. Use 17 to match the
|
||||
Android toolchain; 21 also works but 17 is the safe floor.)
|
||||
|
||||
## 12.2 Android SDK
|
||||
|
||||
```bash
|
||||
# cmdline-tools
|
||||
mkdir -p ~/android-sdk/cmdline-tools
|
||||
cd ~/android-sdk/cmdline-tools
|
||||
curl -O https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
|
||||
unzip commandlinetools-linux-*.zip && mv cmdline-tools latest
|
||||
rm commandlinetools-linux-*.zip
|
||||
|
||||
export ANDROID_HOME=$HOME/android-sdk
|
||||
export PATH=$PATH:$ANDROID_HOME/cmdline-tools/latest/bin:$ANDROID_HOME/platform-tools
|
||||
|
||||
sdkmanager --licenses
|
||||
sdkmanager "platform-tools" "platforms;android-34" "build-tools;34.0.0"
|
||||
```
|
||||
Persist `ANDROID_HOME`/`PATH` in `~/.bashrc`. ADB is already installed system-wide;
|
||||
`platform-tools` from the SDK is fine too (whichever is first on `PATH`).
|
||||
|
||||
Create `app/local.properties`:
|
||||
```
|
||||
sdk.dir=/home/<you>/android-sdk
|
||||
```
|
||||
|
||||
## 12.3 Gradle
|
||||
|
||||
No system install — use the project wrapper:
|
||||
```bash
|
||||
cd app
|
||||
./gradlew tasks # first run downloads the wrapper distribution
|
||||
```
|
||||
(The wrapper version is pinned in `app/gradle/wrapper/gradle-wrapper.properties`.)
|
||||
|
||||
## 12.4 hermes environment (for the plugin + running the gateway)
|
||||
|
||||
```bash
|
||||
cd hermes-agent
|
||||
uv sync # creates .venv with all core deps (websockets, httpx, …)
|
||||
source .venv/bin/activate
|
||||
hermes --version # sanity
|
||||
```
|
||||
- Run the gateway with the plugin:
|
||||
```bash
|
||||
# install the plugin (dev: symlink)
|
||||
mkdir -p ~/.hermes/plugins
|
||||
ln -s "$PWD/../gateway-plugin" ~/.hermes/plugins/android
|
||||
hermes gateway status # should list "android"
|
||||
hermes gateway # run
|
||||
```
|
||||
- Tests use hermes's hermetic runner (never bare `pytest`):
|
||||
```bash
|
||||
scripts/run_tests.sh tests/gateway/test_android.py
|
||||
```
|
||||
|
||||
## 12.5 Firebase (FCM) — primary push
|
||||
|
||||
1. Create a Firebase project (console.firebase.google.com).
|
||||
2. Add an **Android app** (package = `androidApp` applicationId, e.g.
|
||||
`dev.iris.app`). Download `google-services.json` → `app/androidApp/`.
|
||||
3. Create a **service account** (Project settings → Service accounts → Generate
|
||||
new private key) → download the JSON. Store its path in
|
||||
`ANDROID_FCM_SERVICE_ACCOUNT` (in `~/.hermes/.env`).
|
||||
4. The app's `FirebaseMessagingService` obtains the FCM token at runtime and
|
||||
registers it via `hello` / `fcm.register`.
|
||||
|
||||
> Skip Firebase → set `ANDROID_PUSH_BACKEND=ntfy` and configure `NTFY_TOPIC` /
|
||||
> `NTFY_SERVER_URL` (self-host ntfy or use ntfy.sh). See `08-push.md`.
|
||||
|
||||
## 12.6 Environment variables (summary)
|
||||
|
||||
**Secrets (`~/.hermes/.env`):**
|
||||
```
|
||||
ANDROID_TOKEN=<64-hex>
|
||||
ANDROID_PUSH_BACKEND=fcm # or ntfy
|
||||
ANDROID_FCM_SERVICE_ACCOUNT=/path/to/service-account.json
|
||||
# ANDROID_FCM_SERVER_KEY=<legacy key> # fallback if no service account
|
||||
# NTFY_TOPIC=iris-push # when ntfy
|
||||
# NTFY_SERVER_URL=https://ntfy.sh
|
||||
# ANDROID_WS_CERT=/path/cert.pem # WSS
|
||||
# ANDROID_WS_KEY=/path/key.pem
|
||||
```
|
||||
|
||||
**Behavioral (`~/.hermes/config.yaml`):**
|
||||
```yaml
|
||||
gateway:
|
||||
platforms:
|
||||
android:
|
||||
enabled: true
|
||||
extra:
|
||||
host: 127.0.0.1 # 0.0.0.0 for LAN
|
||||
port: 8790
|
||||
home_channel: android:default
|
||||
push_backend: fcm
|
||||
outbox_retention_hours: 72
|
||||
max_upload_bytes: 104857600 # 100 MB
|
||||
display:
|
||||
platforms:
|
||||
android:
|
||||
show_reasoning: true
|
||||
reasoning_style: code
|
||||
streaming: true
|
||||
tool_progress: all # gateway sends full data; app controls display
|
||||
```
|
||||
|
||||
## 12.7 Verify the stack (smoke test)
|
||||
|
||||
```bash
|
||||
# 1. gateway up with plugin
|
||||
hermes gateway status | grep -i android
|
||||
|
||||
# 2. a raw WS client can pair + echo
|
||||
python - <<'PY'
|
||||
import asyncio, json, websockets
|
||||
async def main():
|
||||
async with websockets.connect("ws://127.0.0.1:8790/ws") as ws:
|
||||
await ws.send(json.dumps({"v":1,"type":"hello","payload":{
|
||||
"token":"<ANDROID_TOKEN>","device_id":"test","device_name":"probe",
|
||||
"caps":{"min_protocol":1}}}))
|
||||
print("recv:", await ws.recv())
|
||||
asyncio.run(main())
|
||||
PY
|
||||
```
|
||||
Expect a `hello.ack`. If you get `error {code:"auth"}`, the token/host/port is
|
||||
wrong.
|
||||
@@ -0,0 +1,131 @@
|
||||
# 13 — Testing & Debugging (incl. ADB on-device)
|
||||
|
||||
Three layers: **Python plugin tests**, **Kotlin unit/UI tests**, and **on-device
|
||||
E2E via ADB**. Plus a **WS test-client harness** to drive the real gateway
|
||||
without the app (critical for verifying frame shapes early).
|
||||
|
||||
## 13.1 Python plugin tests
|
||||
|
||||
- Location: `gateway-plugin/tests/` (and, for hermes-integration tests, mirror
|
||||
into the hermes `tests/gateway/test_android.py` pattern when running under
|
||||
hermes's suite).
|
||||
- **Run with hermes's hermetic runner** (never bare `pytest`):
|
||||
```bash
|
||||
cd hermes-agent
|
||||
scripts/run_tests.sh tests/gateway/test_android.py
|
||||
scripts/run_tests.sh # full suite (CI parity)
|
||||
```
|
||||
- Coverage to write (behavioral, not change-detector — per hermes test policy):
|
||||
- `register(ctx)` produces a valid `PlatformEntry` (name, cron env var,
|
||||
parse_target_ref).
|
||||
- `check_requirements` / `validate_config` / `is_connected` truth table.
|
||||
- `_parse_target_ref`: `android:<chat>`, `android:<chat>:<thread>`, non-android
|
||||
→ None.
|
||||
- **Reasoning split:** given a `show_reasoning`-style final text, `send()`
|
||||
emits `message {reasoning, text}` correctly; no-prefix → no reasoning field.
|
||||
- **Outbox:** frame with no subscriber → written with a cursor; `sync` replays
|
||||
the right range; retention prunes.
|
||||
- **Pairing:** valid token → `hello.ack`; wrong token → `error auth` + close.
|
||||
Constant-time compare used.
|
||||
- **Media:** upload start→binary→end reassembles + sha256 verified; over-limit
|
||||
→ `media_too_large`; pull serves only allowed paths.
|
||||
- **Push backend selection:** fcm vs ntfy chosen by config; `configured()`
|
||||
reflects missing creds.
|
||||
- **Channel directory:** create/rename/set_default; cron target resolution.
|
||||
- **No `~/.hermes` writes in tests** — use the `_isolate_hermes_home` fixture
|
||||
pattern (temp `HERMES_HOME`). Profile tests also mock `Path.home()`.
|
||||
|
||||
## 13.2 WS test-client harness (do this FIRST, in M1/M2)
|
||||
|
||||
A small Python script (`gateway-plugin/tests/ws_probe.py`) that connects to the
|
||||
**real running gateway** and drives a turn, printing every frame. This is how we
|
||||
**empirically confirm** the exact frame shapes (especially tool-progress vs
|
||||
commentary classification and the reasoning prefix) before/while building the
|
||||
Kotlin client.
|
||||
|
||||
```bash
|
||||
hermes gateway & # with the android plugin
|
||||
python gateway-plugin/tests/ws_probe.py --token <ANDROID_TOKEN> \
|
||||
--send "list the files and summarize"
|
||||
# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end,
|
||||
# commentary, message.stop {reasoning,…}, …
|
||||
```
|
||||
Use it to lock `04-wire-protocol.md` against reality and to debug the adapter
|
||||
without waiting for the app.
|
||||
|
||||
## 13.3 Kotlin unit / UI tests
|
||||
|
||||
- **Protocol codec:** round-trip every frame type (serialize → deserialize →
|
||||
equal); unknown `type`/fields ignored (forward-compat).
|
||||
- **Repositories:** merge-don't-clobber on `channel.*`; optimistic send + rollback
|
||||
on `error`; sync dedupe by `message_id`; cursor monotonic.
|
||||
- **`GatewayClient`:** reconnect/backoff; `message.update` coalescing (latest
|
||||
wins); request/response correlation by `id`.
|
||||
- **ViewModels:** state transitions (connecting→connected→reconnecting); tool
|
||||
verbosity filtering (everything/truncated/nothing); reasoning present/absent.
|
||||
- **Compose UI tests:** ReasoningBlock collapse/expand + copy; ToolCard
|
||||
spinner→done; composer auto-grow cap; channel list active highlight.
|
||||
- Run: `./gradlew :shared:testDebugUnitTest` (and `:shared:testDesktopTest`).
|
||||
|
||||
## 13.4 On-device E2E via ADB (the MIX 2S, API 29)
|
||||
|
||||
Device: `a5ca2a4b` (Xiaomi MIX 2S). Workflow:
|
||||
|
||||
```bash
|
||||
# install + launch
|
||||
cd app
|
||||
./gradlew :androidApp:installDebug
|
||||
adb shell am start -n dev.iris.app/.MainActivity
|
||||
|
||||
# logs (filter our tags + WS + FCM + ExoPlayer)
|
||||
adb logcat -c
|
||||
adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec"
|
||||
|
||||
# screenshots for visual checks
|
||||
adb exec-out screencap -p > /tmp/shot.png
|
||||
|
||||
# clear app data between pairing attempts
|
||||
adb shell pm clear dev.iris.app
|
||||
|
||||
# push a file to the app's cache (for media tests) / pull logs
|
||||
adb shell run-as dev.iris.app ls files
|
||||
adb logcat -d > /tmp/logcat.txt
|
||||
```
|
||||
|
||||
**E2E scenarios (script where possible):**
|
||||
1. **Pair:** connect screen → enter URL+token → `hello.ack` → main. (Verify auth
|
||||
leg, not just TCP.)
|
||||
2. **Text round-trip:** send "hello" → streamed reply appears (message.start →
|
||||
updates → stop).
|
||||
3. **Reasoning:** ask a reasoning-model question → ReasoningBlock shows above the
|
||||
answer; copy button works.
|
||||
4. **Tools:** trigger a tool (e.g. "list files") → ToolCard shows; toggle
|
||||
verbosity in Settings → rendering changes.
|
||||
5. **Intermediate:** a multi-step prompt → commentary bubble appears dimmed.
|
||||
6. **Channels:** create "Cron Reports" → appears in list; set as cron target.
|
||||
7. **Cron delivery:** create a cron job `deliver=android:android:chan_<n>` → it
|
||||
fires → lands in that channel (not default).
|
||||
8. **Search:** "search everywhere" vs "this chat" → correct scoping; tap → jump.
|
||||
9. **Media (in):** attach a photo + a video → agent receives (vision) → reply.
|
||||
10. **Media (out):** ask the agent to send an image/video → `media.offer` →
|
||||
inline player plays it live.
|
||||
11. **Push:** background the app (`adb shell am start` another app / lock) →
|
||||
trigger a message → FCM notification appears → tap → syncs + deep-links.
|
||||
12. **Reconnect/sync:** kill the WS (stop gateway briefly) → restart → app
|
||||
reconnects → `sync` catches up (no lost/dup messages).
|
||||
|
||||
## 13.5 Debugging tips
|
||||
|
||||
- **Gateway side:** `~/.hermes/logs/gateway.log` (and `hermes logs --follow`).
|
||||
Our plugin logs under the `android` adapter name; secrets redacted.
|
||||
- **WS framing bugs:** use the `ws_probe.py` harness — it isolates the protocol
|
||||
from the app.
|
||||
- **Streaming jitter:** the consumer edits at intervals; if updates look chunky,
|
||||
check `display.platforms.android.streaming` and the consumer's edit interval.
|
||||
- **Media pull stalls:** check chunk size + backpressure; confirm the file is
|
||||
within hermes delivery roots (`validate_media_delivery_path`).
|
||||
- **FCM not arriving:** confirm the token registered (`devices.db`), the service
|
||||
account can mint a token, and the app's `onNewToken` re-registered after
|
||||
`pm clear`.
|
||||
- **Profile leaks:** if tokens look wrong under multiple profiles, verify the
|
||||
scope-aware secret read (`_get_scoped_secret`) is used.
|
||||
@@ -0,0 +1,138 @@
|
||||
# 14 — Milestones (M0–M7)
|
||||
|
||||
Phased delivery. Each milestone ends with a **demo** (on-device where noted) and
|
||||
has explicit **acceptance criteria**. Work top-to-bottom; don't skip M0/M1.
|
||||
|
||||
---
|
||||
|
||||
## M0 — Toolchain & scaffolding
|
||||
**Goal:** everything builds; the plugin is discoverable; the repo is safe.
|
||||
- [ ] Install JDK 17, Android SDK, set `ANDROID_HOME` (`12-toolchain.md`).
|
||||
- [ ] `cd hermes-agent && uv sync` (hermes venv works).
|
||||
- [ ] Create monorepo scaffold (`02-monorepo.md`): `gateway-plugin/`, `app/`
|
||||
(CMP: `shared`, `androidApp`, `desktopApp`), root `.gitignore`
|
||||
(**excludes `hermes-agent/`**), root `README.md`.
|
||||
- [ ] `git init` + a pre-commit/CI guard that fails if `hermes-agent/` is staged.
|
||||
- [ ] CMP project builds empty: `./gradlew :androidApp:assembleDebug`,
|
||||
`./gradlew :desktopApp:run` (blank window).
|
||||
- [ ] Plugin skeleton: `plugin.yaml` + `adapter.py` with `register(ctx)` + a
|
||||
no-op `AndroidAdapter` → `hermes gateway status` lists **android**.
|
||||
- **Demo:** `hermes gateway status` shows `android`; `./gradlew
|
||||
:androidApp:installDebug` installs a blank app on the MIX 2S.
|
||||
- **Accept:** blank app installs + launches on-device; plugin visible in
|
||||
`hermes gateway status`; `hermes-agent/` is git-ignored (verify with
|
||||
`git status --ignored`).
|
||||
|
||||
## M1 — Gateway core loop (text round-trip)
|
||||
**Goal:** pair + send a text message + get a (non-streaming) reply.
|
||||
- [ ] WS server (`ws_server.py`): bind, `hello` auth (constant-time),
|
||||
`hello.ack`, heartbeat, connection registry.
|
||||
- [ ] `AndroidAdapter.send()` → `message` frame; inbound `message.send` →
|
||||
`MessageEvent` → `handle_message`.
|
||||
- [ ] Pairing store + `ANDROID_TOKEN`; QR payload in `interactive_setup`.
|
||||
- [ ] App: Connect screen (URL+token, real `hello` test), `GatewayClient`
|
||||
(connect + reconnect), ChatScreen sends + renders `message`.
|
||||
- [ ] `ws_probe.py` harness drives a real turn.
|
||||
- **Demo (on-device):** pair the phone, send "hello", see the agent's reply.
|
||||
- **Accept:** text round-trip works on-device; wrong token is rejected;
|
||||
reconnect after gateway restart re-pairs.
|
||||
|
||||
## M2 — Streaming + reasoning + tools + commentary
|
||||
**Goal:** the "agent transparency" features.
|
||||
- [ ] Map consumer `send`/`edit_message` → `message.start/update/stop`.
|
||||
- [ ] Reasoning: set `show_reasoning` for android; adapter splits prefix →
|
||||
`reasoning` field. **Verify format with `ws_probe.py`.**
|
||||
- [ ] Tool events: classify tool-progress `send()`s → structured
|
||||
`tool.start/progress/end` (turn-state machine). **Verify with probe.**
|
||||
- [ ] Commentary → `commentary` frames. Typing → `typing`.
|
||||
- [ ] App: live bubble (coalesced updates), `ReasoningBlock` (collapse + copy),
|
||||
`ToolCard` with **Everything/Truncated/Nothing** setting, dimmed
|
||||
`commentary` bubble.
|
||||
- **Demo (on-device):** a multi-step prompt streams, shows reasoning above the
|
||||
answer, tool cards (toggle verbosity), and an intermediate beat.
|
||||
- **Accept:** all four render correctly; tool verbosity setting changes
|
||||
rendering; reasoning copy button works; frame shapes match `04-wire-protocol`.
|
||||
|
||||
## M3 — Channels/threads + cron + search
|
||||
**Goal:** organization + cron delegation + search.
|
||||
- [ ] Channel directory (SQLite): default channel ensured; `channel.create/
|
||||
rename/set_default/delete` + `channel.*` frames.
|
||||
- [ ] Threads: toggle in default chat; `thread_id` lanes; `create_handoff_thread`.
|
||||
- [ ] `parse_target_ref_fn` + `cron_deliver_env_var` → cron
|
||||
`deliver=android:<chat>[:<thread>]` works.
|
||||
- [ ] `search.py` FTS5 bridge; `search` frame (all / this-chat) → results.
|
||||
- [ ] App: channel list (drawer/rail), thread toggle + topic switcher, "new
|
||||
channel" + "set as cron target", SearchScreen with scope toggle + jump.
|
||||
- **Demo (on-device):** create "Cron Reports"; create a cron job delivering to
|
||||
it; it fires into that channel; search finds a message (both scopes).
|
||||
- **Accept:** cron output lands in the chosen channel (not default); threads
|
||||
isolate context; search scoping correct; channel list reconciles on events.
|
||||
|
||||
## M4 — Media
|
||||
**Goal:** attach + receive + play media.
|
||||
- [ ] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
|
||||
size limit + sha256 + MIME re-sniff.
|
||||
- [ ] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
|
||||
security.
|
||||
- [ ] App: SAF pickers + preview chips + upload; `media.pull` → cache;
|
||||
**ExoPlayer** inline (audio mini-player, video fullscreen/PiP); image/doc
|
||||
viewers.
|
||||
- **Demo (on-device):** attach a photo + video (agent sees them); ask agent to
|
||||
send an image/video → plays live in-app.
|
||||
- **Accept:** both directions work; over-limit rejected; playback is live;
|
||||
only allowed files are servable.
|
||||
|
||||
## M5 — Push + offline (FCM + ntfy)
|
||||
**Goal:** reach the phone when backgrounded; catch up on reconnect.
|
||||
- [ ] Outbox (SQLite) + sync cursor; `sync`/`sync.done`; retention prune.
|
||||
- [ ] `push.py`: `FcmBackend` (HTTP v1 + service account, httpx) +
|
||||
`NtfyBackend`; selected by `ANDROID_PUSH_BACKEND`.
|
||||
- [ ] Fire push on no-live-subscriber; data payload for silent sync.
|
||||
- [ ] App: FCM service (foreground banner + background foreground-service sync),
|
||||
`onNewToken` → `fcm.register`; per-chat notification channels; deep-link.
|
||||
ntfy listener fallback.
|
||||
- [ ] In-app `notification` banners (channel_renamed, approval, cron, …).
|
||||
- **Demo (on-device):** background the app → trigger a message → notification
|
||||
appears → tap → syncs + opens the chat. Repeat with ntfy backend.
|
||||
- **Accept:** push arrives when backgrounded (both backends); reconnect syncs
|
||||
with no loss/dup; banners show for foreground events.
|
||||
|
||||
## M6 — Desktop app
|
||||
**Goal:** the same app on a big screen.
|
||||
- [ ] `desktopMain`: tray + OS notifications; `MediaPlayer` actual (mpv/WebView);
|
||||
`MediaPicker` actual (file dialog); `SecureStore` actual; window mgmt.
|
||||
- [ ] Two-pane default layout; keyboard shortcuts; optional inspector pane.
|
||||
- [ ] Parity pass vs Android feature checklist (`11-desktop-app.md`).
|
||||
- [ ] jpackage builds (Linux first; macOS/Windows as available).
|
||||
- **Demo:** desktop app pairs to the same gateway; full feature parity; tray
|
||||
notifications; media plays.
|
||||
- **Accept:** all Android features work on desktop; tray + shortcuts work;
|
||||
native binary launches.
|
||||
|
||||
## M7 — Polish + E2E + docs
|
||||
**Goal:** ship-quality.
|
||||
- [ ] Telegram-style layout pass (per reference image): header, bubbles, date
|
||||
separators, ✓✓, model/token footer, banner, bottom bar.
|
||||
- [ ] Theming (dark default, accent), onboarding/pairing UX, empty/loading/
|
||||
reconnecting/degraded states with honest copy.
|
||||
- [ ] Full E2E suite (`13-testing.md` scenarios 1–12) automated where possible.
|
||||
- [ ] Docs: `docs/protocol/frames.schema.json` finalized; `docs/setup.md`
|
||||
(user-facing pairing + FCM/ntfy + remote access); root README.
|
||||
- [ ] Security hardening checklist (`09-pairing-security.md`) verified.
|
||||
- **Demo:** end-to-end on phone + desktop simultaneously; cron into a channel;
|
||||
push; media; search.
|
||||
- **Accept:** all feature-checklist items pass on-device; E2E green; docs
|
||||
complete; `hermes-agent/` still never committed.
|
||||
|
||||
---
|
||||
|
||||
## Sequencing notes
|
||||
- **M1/M2 depend on the `ws_probe.py` harness** to lock frame shapes early —
|
||||
build it in M1.
|
||||
- **M3 (cron) and M5 (push) both touch the outbox** — build the outbox in M3,
|
||||
extend for push in M5.
|
||||
- **M6 (desktop) reuses M1–M5 shared code** — do it after the Android features
|
||||
are stable so the shared module is settled.
|
||||
- Parallelizable: plugin (Python) and app (Kotlin) can be worked on
|
||||
concurrently once the protocol (`04-wire-protocol.md`) is agreed; the probe
|
||||
harness is the integration seam.
|
||||
@@ -0,0 +1,136 @@
|
||||
# 15 — hermes-agent Source Reference Map
|
||||
|
||||
A cheat-sheet of the **exact hermes-agent files/lines** to read for each
|
||||
integration point. Paths are relative to `hermes-agent/` (the read-only
|
||||
reference). This lets a coder jump straight to the right code instead of
|
||||
re-deriving the architecture.
|
||||
|
||||
> ⚠️ Read-only. We **install** our plugin into `~/.hermes/plugins/android`; we
|
||||
> never edit these files.
|
||||
|
||||
## Plugin / platform registration
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| How to add a platform (Plugin Path) | `gateway/platforms/ADDING_A_PLATFORM.md` |
|
||||
| Canonical plugin-platform example | `plugins/platforms/irc/adapter.py` (esp. `register(ctx)` at :953, `_env_enablement` :677, `_standalone_send` :743, `_get_scoped_secret` :42) |
|
||||
| ntfy plugin example (push-ish) | `plugins/platforms/ntfy/adapter.py`, `plugin.yaml` |
|
||||
| `register_platform()` (PluginContext) | `hermes_cli/plugins.py:2774` |
|
||||
| `PlatformEntry` dataclass (all fields) | `gateway/platform_registry.py:63` |
|
||||
| `Platform` enum | `gateway/config.py` |
|
||||
| Plugin discovery (`PluginManager`) | `hermes_cli/plugins.py` |
|
||||
|
||||
## Base adapter contract
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| `BasePlatformAdapter` (ABC) | `gateway/platforms/base.py:2890` |
|
||||
| `MessageEvent` (inbound) | `gateway/platforms/base.py:2300` |
|
||||
| `MessageType` | `gateway/platforms/base.py:2278` |
|
||||
| `SendResult` | `gateway/platforms/base.py:2466` |
|
||||
| `build_source(...)` (SessionSource) | `gateway/platforms/base.py:7047` |
|
||||
| `handle_message(event)` | `gateway/platforms/base.py:5981` |
|
||||
| `send()` / `edit_message()` / `delete_message()` | `base.py:3920 / 3976 / 4005` |
|
||||
| `send_typing` / `stop_typing` | `base.py:4298 / 4307` |
|
||||
| Media send: `send_image/video/document/voice/animation/image_file/multiple_images` | `base.py:4396 / 4632 / 4659 / 4486 / 4415 / 4339 / 7834(tg)` |
|
||||
| `extract_images` / `extract_media` / `extract_local_files` | `base.py:4439 / 4884 / 5020` |
|
||||
| Media cache helpers: `cache_image/audio/video/document_from_bytes` | `base.py:854 / 1005 / 1122 / 2126` |
|
||||
| Inbound media size: `get_inbound_media_max_bytes` / `validate_inbound_media_size` | `base.py:758 / 779` |
|
||||
| Media delivery security: `validate_media_delivery_path` + roots/recency/denied | `base.py:1684 / 1312-1480` |
|
||||
| Interactive: `send_slash_confirm` / `send_clarify` / `send_private_notice` | `base.py:4169 / 4204 / 4278` |
|
||||
| `create_handoff_thread` | `base.py:3949` |
|
||||
| Streaming hooks: `supports_draft_streaming` / `send_draft` / `render_message_event` / `format_tool_event` | `base.py:3215 / 3274 / 3316 / 3337` |
|
||||
| Scoped secret read pattern (profile-safe) | `plugins/platforms/irc/adapter.py:42` |
|
||||
| Scoped lock (profile-safe bind) | `gateway/status.py` (`acquire_scoped_lock`) |
|
||||
|
||||
## Streaming (legacy callback path — what the main gateway uses)
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| `GatewayStreamConsumer` (the sink) | `gateway/stream_consumer.py:156` |
|
||||
| `on_delta` / `on_commentary` / `on_segment_break` / `finish` | `stream_consumer.py:611 / 518 / 514 / 623` |
|
||||
| Consumer `run()` loop (edit cadence) | `stream_consumer.py:781` |
|
||||
| Structured events (ACP-only, NOT main gateway) | `gateway/stream_events.py`, `gateway/stream_dispatch.py:40` |
|
||||
| Callback wiring (agent → consumer) | `gateway/run.py:5642-5704` (`tool_progress_callback`, `stream_delta_callback`, `interim_assistant_callback`, `status_callback`, `event_callback`) |
|
||||
| `send_progress_messages` (tool progress → `send`) | `gateway/run.py:4603` |
|
||||
| Progress metadata | `gateway/run.py:28089` |
|
||||
|
||||
## Reasoning display
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| Reasoning prepended to final response | `gateway/run.py:20089-20127` |
|
||||
| `show_reasoning` / `reasoning_style` defaults + per-platform | `gateway/display_config.py:33-181` |
|
||||
| `resolve_display_setting()` | `gateway/display_config.py:187` |
|
||||
| `last_reasoning` in agent result | `gateway/run.py:6471` |
|
||||
| Think-tag filtering in consumer | `stream_consumer.py:175-185, 627+` |
|
||||
|
||||
## Display settings (tool progress, interim, streaming, etc.)
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| All overrideable display keys + defaults | `gateway/display_config.py:33` (`tool_progress`, `show_reasoning`, `reasoning_style`, `tool_preview_length`, `streaming`, `interim_assistant_messages`, `long_running_notifications`, `cleanup_progress`, `live_status`) |
|
||||
| Per-platform tiers | `gateway/display_config.py:81-181` |
|
||||
|
||||
## Slash commands
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| `COMMAND_REGISTRY` / `CommandDef` (all commands) | `hermes_cli/commands.py:144+` |
|
||||
| `GATEWAY_KNOWN_COMMANDS` / `is_gateway_known_command` / `resolve_command` | `hermes_cli/commands.py` |
|
||||
| Gateway command dispatch (alias, access, hooks) | `gateway/run.py:16974-17099` |
|
||||
| `send_model_picker` / `send_choice_picker` (Telegram ref) | `plugins/platforms/telegram/adapter.py:6350 / 6424` |
|
||||
| Picker invocation from gateway | `gateway/slash_commands.py:1859, 2157, 3622-3657` |
|
||||
| Button-callback id conventions (`cl:`, `appr:`, `sc:`) | `gateway/platforms/ADDING_A_PLATFORM.md` (Interactive UX) |
|
||||
|
||||
## Cron delivery
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| Resolve a delivery target (`platform:chat_id[:thread_id]`) | `cron/scheduler.py:2148` (`_resolve_single_delivery_target`) |
|
||||
| `_resolve_delivery_targets` / routing tokens (`all`) | `cron/scheduler.py:2296 / 2278` |
|
||||
| `cron_deliver_env_var` handling (home channel) | `cron/scheduler.py:1903+` |
|
||||
| `deliver` param normalization | `cron/scheduler.py:2251` |
|
||||
| `send_message` tool target resolution (`resolve_send_target`, `prepare_send_message_platforms`) | `tools/send_message_tool.py` |
|
||||
| `parse_target_ref_fn` usage | `tools/send_message_tool.py` (`_parse_target_ref`) |
|
||||
| Cron mirror delivery (default off) | `cron/scheduler.py:1520` |
|
||||
| `cronjob` tool schema (`deliver` description) | `tools/cronjob_tools.py` |
|
||||
|
||||
## Search (FTS5)
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| Session store (SQLite + FTS5) | `hermes_state.py` |
|
||||
| Search implementation | `hermes_state_search.py` |
|
||||
| Schema | `hermes_state_schema.py` |
|
||||
|
||||
## Config / env / profiles
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| `get_hermes_home()` / `display_hermes_home()` (profile-safe paths) | `hermes_constants.py` |
|
||||
| `DEFAULT_CONFIG` / `OPTIONAL_ENV_VARS` | `hermes_cli/config.py` |
|
||||
| Gateway config load (`load_gateway_config`, `_apply_env_overrides`) | `gateway/config.py` |
|
||||
| Profile override (`_apply_profile_override`) | `hermes_cli/main.py` |
|
||||
| Secret scope (multiplex fail-closed) | `agent/secret_scope.py` |
|
||||
| PII redaction | `agent/redact.py` |
|
||||
|
||||
## Dependencies (confirm zero new deps)
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| `websockets==15.0.1` (core) | `pyproject.toml:111` |
|
||||
| `httpx[socks]==0.28.1` (core) | `pyproject.toml:44` |
|
||||
| `aiohttp` (messaging extra, NOT core) | `pyproject.toml:185` |
|
||||
| Dependency pinning policy | `pyproject.toml:19-39`, root `AGENTS.md` |
|
||||
|
||||
## Testing
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| Hermetic test runner (use this, not bare pytest) | `scripts/run_tests.sh` |
|
||||
| `_isolate_hermes_home` fixture | `tests/conftest.py` |
|
||||
| Example platform tests | `tests/gateway/test_*.py` (e.g. `test_google_chat.py`, `test_line_plugin.py`) |
|
||||
| Stream-event tests (dispatcher) | `tests/gateway/test_stream_events.py` |
|
||||
|
||||
## Existing desktop app (for reference only — we do NOT reuse its backend)
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| Electron desktop app | `apps/desktop/` (its own `AGENTS.md`, `DESIGN.md`) |
|
||||
| Shared JSON-RPC WS client (tui_gateway protocol) | `apps/shared/src/json-rpc-gateway.ts` |
|
||||
| tui_gateway WS transport (mobile-client-ready) | `tui_gateway/ws.py` |
|
||||
| tui_gateway method catalog | `tui_gateway/server.py` |
|
||||
|
||||
> Note: the existing desktop app talks to the **`tui_gateway`** backend
|
||||
> (`hermes serve`), a *different* process from the messaging gateway. Our apps
|
||||
> talk to the **messaging gateway** via our own plugin + protocol. We borrow
|
||||
> naming conventions only.
|
||||
@@ -0,0 +1,75 @@
|
||||
# 16 — Decisions & Open Questions
|
||||
|
||||
## Locked decisions (from planning, 2026-08-19)
|
||||
|
||||
| # | Decision | Choice | Rationale |
|
||||
|---|---|---|---|
|
||||
| 1 | Desktop app tech | **Compose Multiplatform** | Desktop = "the Android app, tweaked"; share protocol/state/UI. |
|
||||
| 2 | Push backend | **Both — FCM primary, ntfy fallback** | FCM is standard/reliable; ntfy for self-hosters with no Firebase. `ANDROID_PUSH_BACKEND`. |
|
||||
| 3 | Media transport | **Over the WebSocket** | One transport, zero new Python deps; chunked binary frames. |
|
||||
| 4 | Phone default layout | **User-toggleable, single-pane default** | App-like on phones; auto two-pane on large screens; desktop defaults two-pane. |
|
||||
|
||||
## Additional decisions made during planning
|
||||
|
||||
| Decision | Choice | Note |
|
||||
|---|---|---|
|
||||
| Connection point | **Messaging gateway** (platform plugin `android`), not `tui_gateway` | Makes cron/`send_message`/slash/coexistence native. |
|
||||
| Plugin style | **Community plugin** (`register(ctx)`) | Zero hermes-core changes. |
|
||||
| Python deps | **None new** (`websockets` + `httpx` are core) | Respects hermes pinning policy. |
|
||||
| Tool events | **Structured frames; app controls verbosity** | Gateway sends full data; app = everything/truncated/nothing. |
|
||||
| Reasoning | **Adapter splits the `show_reasoning` prefix** | Clean `reasoning` field → collapsible block above message. |
|
||||
| Channels/threads | **Map onto `chat_id`/`thread_id`** | Existing gateway primitives; cron targets them. |
|
||||
| Offline | **SQLite outbox + sync cursor** | Catch-up on reconnect; push on disconnect. |
|
||||
| Default chat id | `android:default` | Home channel + cron default. |
|
||||
| WS port | `8790` (default) | Configurable. |
|
||||
| minSdk | 26 (test device API 29) | Broad coverage. |
|
||||
| Frame routing | **Broadcast to all connected devices** (no per-chat subscribe) | Single-user model; simpler. |
|
||||
| Initial channel load | **`history` frame** (paginated) | `sync` only replays outbox; `history` loads full messages. |
|
||||
| Slash menu | **`commands.catalog` + `commands.complete`** frames | Gateway serves catalog; app renders bottom sheet + autocomplete. |
|
||||
| Agent lifecycle | **`agent.busy`/`agent.idle`** events + **`agent.stop`/`agent.steer`** requests | App shows thinking indicator; user can abort or steer mid-turn. |
|
||||
| Local DB (KMP) | **SQLDelight** (not Room) | Room is Android-only; SQLDelight works in commonMain for both platforms. |
|
||||
| Voice input | **Record → upload as audio media** (no client-side STT) | Agent's STT (if configured) handles transcription. |
|
||||
|
||||
## Open questions (resolve during implementation)
|
||||
|
||||
These are **implementation-time** details, not blockers. Each has a default we
|
||||
will proceed with unless you say otherwise.
|
||||
|
||||
1. **Tool-progress vs commentary classification (M2).** The exact signal that
|
||||
distinguishes a tool-progress `send()` from a regular `send()`/commentary in
|
||||
the legacy path. *Default:* per-chat turn-state machine + line-format
|
||||
heuristic, **verified empirically** with the `ws_probe.py` harness against the
|
||||
live gateway. If a clean metadata marker exists, prefer it.
|
||||
2. **Reasoning prefix format stability (M2).** We split on the `code`-style
|
||||
`💭 **Reasoning:**\n```\n…\n```\n\n` prefix. *Default:* set
|
||||
`reasoning_style: code` for android and split on that; fallback = no
|
||||
reasoning field (full text) if the prefix isn't found. Verify in M2.
|
||||
3. **Per-device tokens vs shared token (M1/M5).** *Default (v1):* shared
|
||||
`ANDROID_TOKEN` + optional `ANDROID_ALLOWED_USERS` device allowlist.
|
||||
Per-device revocable tokens are a stretch.
|
||||
4. **Desktop video backend (M6).** *Default:* `libmpv`/`mpv`-backed Compose
|
||||
surface, WebView fallback. Confirm `mpv` availability on target OSes during
|
||||
M6.
|
||||
5. **Remote access default (M1).** *Default:* document Tailscale as the
|
||||
recommended remote path; WSS + reverse proxy as alternatives. No public bind
|
||||
by default.
|
||||
6. **Streaming cadence (M2).** If live updates look chunky, tune
|
||||
`display.platforms.android.streaming` / consumer edit interval. *Default:*
|
||||
follow global streaming config.
|
||||
7. **App package name / branding.** *Default:* applicationId `dev.iris.app`,
|
||||
app name "Iris". Confirm final product name + package + icon.
|
||||
8. **ntfy in-app listener (M5).** *Default:* a foreground service maintaining the
|
||||
ntfy subscription (no extra native lib) — or a lightweight ntfy client lib if
|
||||
one is acceptable. Decide in M5.
|
||||
9. **Auto-update for desktop (M7).** *Default:* out of scope (manual download).
|
||||
Revisit post-v1.
|
||||
10. **iOS port.** Out of scope for v1 (protocol is transport-agnostic, so it's a
|
||||
future port). No action now.
|
||||
|
||||
## Explicit non-goals (v1)
|
||||
|
||||
- Multi-user / group chat (personal 1-user agent).
|
||||
- End-to-end encryption (transport WSS only).
|
||||
- Standalone-cron delivery while the gateway process is fully down (best-effort
|
||||
push only).
|
||||
- iOS.
|
||||
@@ -0,0 +1,67 @@
|
||||
# Iris × Hermes — Implementation Reference Library
|
||||
|
||||
A coder-facing reference library for building a **native Android + Desktop
|
||||
experience** for [hermes-agent](https://github.com/NousResearch/hermes-agent),
|
||||
connected through a **gateway platform plugin**.
|
||||
|
||||
This folder is the single source of truth for *what to build and why*. Read it
|
||||
top-to-bottom once, then use the numbered docs as a lookup while implementing.
|
||||
|
||||
> ⚠️ **READ FIRST — two hard rules**
|
||||
> 1. **`hermes-agent/` (sibling of this folder) is a read-only research
|
||||
> reference. It must NEVER be committed, pushed, or shipped.** It is
|
||||
> git-ignored at the repo root. We only *install* our plugin into a live
|
||||
> hermes install (`~/.hermes/plugins/`); we never modify hermes core.
|
||||
> 2. **ADB is installed and a device is connected** (`a5ca2a4b`, Xiaomi MIX 2S,
|
||||
> Android 10 / API 29). Use it to install/launch/debug the app on-device.
|
||||
|
||||
---
|
||||
|
||||
## Reading order
|
||||
|
||||
| # | File | When to read |
|
||||
|---|------|--------------|
|
||||
| 0 | [`00-overview.md`](00-overview.md) | Always first. Vision, scope, disclaimers, locked decisions. |
|
||||
| 1 | [`01-architecture.md`](01-architecture.md) | Before touching code. System shape + rationale. |
|
||||
| 2 | [`02-monorepo.md`](02-monorepo.md) | When scaffolding the repo. |
|
||||
| 3 | [`03-gateway-plugin.md`](03-gateway-plugin.md) | When building the Python plugin. |
|
||||
| 4 | [`04-wire-protocol.md`](04-wire-protocol.md) | When implementing either side of the WS. |
|
||||
| 5 | [`05-streaming.md`](05-streaming.md) | Streaming / reasoning / tools / intermediate. |
|
||||
| 6 | [`06-channels-cron-search.md`](06-channels-cron-search.md) | Channels, threads, cron delivery, search. |
|
||||
| 7 | [`07-media.md`](07-media.md) | Media upload/download + playback. |
|
||||
| 8 | [`08-push.md`](08-push.md) | Push (FCM + ntfy), outbox, sync. |
|
||||
| 9 | [`09-pairing-security.md`](09-pairing-security.md) | Pairing, auth, security model. |
|
||||
| 10 | [`10-android-app.md`](10-android-app.md) | When building the Android app. |
|
||||
| 11 | [`11-desktop-app.md`](11-desktop-app.md) | When building the Desktop app. |
|
||||
| 12 | [`12-toolchain.md`](12-toolchain.md) | First time on a machine (JDK/SDK/uv/Firebase). |
|
||||
| 13 | [`13-testing.md`](13-testing.md) | Writing tests + on-device ADB workflow. |
|
||||
| 14 | [`14-milestones.md`](14-milestones.md) | Planning work / tracking progress. |
|
||||
| 15 | [`15-hermes-reference.md`](15-hermes-reference.md) | **Cheat-sheet** of hermes-agent source to read. |
|
||||
| 16 | [`16-open-questions.md`](16-open-questions.md) | Decisions made + open items. |
|
||||
|
||||
Machine-readable / diagrams:
|
||||
- [`protocol/frames.schema.json`](protocol/frames.schema.json) — wire-frame schema.
|
||||
- [`diagrams/architecture.mmd`](diagrams/architecture.mmd) — mermaid architecture.
|
||||
|
||||
---
|
||||
|
||||
## The three deliverables (one monorepo)
|
||||
|
||||
1. **`gateway-plugin/`** — a Python hermes **platform plugin** named `android`.
|
||||
Runs inside the `hermes gateway` process. Opens a WebSocket server the apps
|
||||
connect to. Implements the full `BasePlatformAdapter` contract. **Zero new
|
||||
Python dependencies, zero hermes-core changes.**
|
||||
2. **`app/androidApp`** — native Kotlin + Jetpack Compose client.
|
||||
3. **`app/desktopApp`** — Kotlin + Compose Multiplatform client that *shares*
|
||||
the Android app's code and is "tweaked" for a big screen.
|
||||
|
||||
The Android and Desktop clients live in **one Compose Multiplatform Gradle
|
||||
project** (`app/`) with a shared KMP module (`app/shared`).
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
- **Phase:** Planning complete → ready to implement (Milestone M0).
|
||||
- **Owner decisions locked:** see [`16-open-questions.md`](16-open-questions.md).
|
||||
- **Last updated:** 2026-08-19.
|
||||
@@ -0,0 +1,54 @@
|
||||
%% Iris x Hermes — architecture (mermaid)
|
||||
%% Render with any mermaid viewer (e.g. https://mermaid.live or `mmdc`).
|
||||
flowchart TB
|
||||
subgraph HOST["User's machine (home server / PC)"]
|
||||
subgraph GW["hermes gateway (ONE process)"]
|
||||
AGENT["Agent core<br/>(run_agent.py)"]
|
||||
SESS["Sessions<br/>(SQLite + FTS5)"]
|
||||
CRON["Cron scheduler"]
|
||||
subgraph PLUGIN["android PLATFORM PLUGIN"]
|
||||
ADAPTER["AndroidAdapter<br/>(BasePlatformAdapter)"]
|
||||
OUTBOX["Outbox (SQLite)<br/>+ sync cursor"]
|
||||
PUSH["push.py<br/>FcmBackend / NtfyBackend"]
|
||||
MEDIA["media.py<br/>cache + chunk stream"]
|
||||
PAIR["pairing.py<br/>token + device registry"]
|
||||
SEARCH["search.py<br/>FTS5 bridge"]
|
||||
end
|
||||
WSS["WebSocket SERVER<br/>(websockets) ws://host:8790/ws"]
|
||||
end
|
||||
AGENT -->|legacy stream callbacks| ADAPTER
|
||||
CRON -->|deliver=android:chat:thread| ADAPTER
|
||||
ADAPTER <--> WSS
|
||||
ADAPTER <--> OUTBOX
|
||||
ADAPTER <--> PUSH
|
||||
ADAPTER <--> MEDIA
|
||||
ADAPTER <--> PAIR
|
||||
ADAPTER <--> SEARCH
|
||||
ADAPTER <--> SESS
|
||||
end
|
||||
|
||||
subgraph CLOUD["Cloud relays"]
|
||||
FCM["Google FCM"]
|
||||
NTFY["ntfy (self-host / ntfy.sh)"]
|
||||
end
|
||||
|
||||
subgraph DEVICES["Clients"]
|
||||
PHONE["ANDROID APP<br/>(Kotlin / Compose)<br/>WS client + ExoPlayer + FCM"]
|
||||
DESKTOP["DESKTOP APP<br/>(Compose Multiplatform)<br/>WS client + tray + desktop player"]
|
||||
end
|
||||
|
||||
WSS <-->|WSS JSON + binary media| PHONE
|
||||
WSS <-->|WSS JSON + binary media| DESKTOP
|
||||
PUSH -->|FCM HTTP v1| FCM
|
||||
PUSH -->|ntfy publish| NTFY
|
||||
FCM -->|wake (data msg)| PHONE
|
||||
NTFY -->|subscribe| PHONE
|
||||
|
||||
classDef host fill:#eef,stroke:#333;
|
||||
classDef plugin fill:#efe,stroke:#333;
|
||||
classDef device fill:#fee,stroke:#333;
|
||||
classDef cloud fill:#ffe,stroke:#333;
|
||||
class GW,AGENT,SESS,CRON,OUTBOX,PUSH,MEDIA,PAIR,SEARCH,WSS host;
|
||||
class ADAPTER plugin;
|
||||
class PHONE,DESKTOP device;
|
||||
class FCM,NTFY cloud;
|
||||
@@ -0,0 +1,107 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Iris x Hermes wire protocol",
|
||||
"description": "Machine-readable description of the JSON frames exchanged over the gateway WebSocket. Mirrors gateway-plugin/protocol.py and docs/04-wire-protocol.md. Media travels as binary WS frames referenced by header/end frames.",
|
||||
"protocol_version": 1,
|
||||
"envelope": {
|
||||
"type": "object",
|
||||
"required": ["v", "type"],
|
||||
"properties": {
|
||||
"v": { "type": "integer", "const": 1, "description": "Protocol version." },
|
||||
"id": { "type": "integer", "description": "Request id; present on requests and their responses/acks. Absent on pure events." },
|
||||
"type": { "type": "string", "description": "Frame type (see frame_types)." },
|
||||
"chat_id": { "type": "string", "description": "Optional chat scope (e.g. android:default, android:chan_7)." },
|
||||
"thread_id": { "type": "string", "description": "Optional thread scope within a chat_id." },
|
||||
"payload": { "type": "object", "description": "Type-specific payload." }
|
||||
}
|
||||
},
|
||||
"frame_types": {
|
||||
"server_to_app": {
|
||||
"hello.ack": {
|
||||
"description": "Pairing succeeded.",
|
||||
"payload": {
|
||||
"server_caps": { "type": "object", "properties": { "streaming": {"type":"boolean"}, "reasoning": {"type":"boolean"}, "tools": {"type":"boolean"}, "media": {"type":"boolean"}, "search": {"type":"boolean"}, "push": {"type":"string","enum":["fcm","ntfy","none"]}, "pickers": {"type":"boolean"} } },
|
||||
"sync_cursor": { "type": "integer" },
|
||||
"channels": { "type": "array", "items": { "$ref": "#/definitions/channel" } }
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"description": "A final / standalone message.",
|
||||
"payload": {
|
||||
"message_id": { "type": "string" },
|
||||
"role": { "type": "string", "enum": ["user", "assistant", "system", "cron"] },
|
||||
"text": { "type": "string" },
|
||||
"reasoning": { "type": "string", "description": "Optional; render ABOVE text." },
|
||||
"media": { "type": "array", "items": { "$ref": "#/definitions/media_ref" } },
|
||||
"reply_to": { "type": "string" },
|
||||
"model": { "type": "string" },
|
||||
"tokens": { "type": "integer" },
|
||||
"ts": { "type": "integer", "description": "epoch millis" }
|
||||
}
|
||||
},
|
||||
"message.start": { "payload": { "message_id": { "type": "string" }, "role": { "type": "string" } } },
|
||||
"message.update": { "payload": { "message_id": { "type": "string" }, "text": { "type": "string", "description": "Full current text (app replaces)." } } },
|
||||
"message.stop": { "payload": { "message_id": { "type": "string" }, "final_text": { "type": "string" }, "reasoning": { "type": "string" }, "model": { "type": "string" }, "tokens": { "type": "integer" } } },
|
||||
"commentary": { "description": "Intermediate assistant beat.", "payload": { "message_id": { "type": "string" }, "text": { "type": "string" } } },
|
||||
"tool.start": { "payload": { "index": { "type": "integer" }, "name": { "type": "string" }, "preview": { "type": "string" }, "args": { "type": "object" } } },
|
||||
"tool.progress": { "payload": { "index": { "type": "integer" }, "name": { "type": "string" }, "note": { "type": "string" } } },
|
||||
"tool.end": { "payload": { "index": { "type": "integer" }, "name": { "type": "string" }, "ok": { "type": "boolean" }, "duration": { "type": "number" }, "output_preview": { "type": "string" } } },
|
||||
"typing": { "payload": { "on": { "type": "boolean" } } },
|
||||
"notification": { "payload": { "kind": { "type": "string", "enum": ["channel_renamed", "channel_created", "cron", "approval", "clarify", "generic"] }, "title": { "type": "string" }, "body": { "type": "string" }, "ts": { "type": "integer" } } },
|
||||
"picker.model": { "payload": { "picker_id": { "type": "string" }, "current_model": { "type": "string" }, "current_provider": { "type": "string" }, "providers": { "type": "array", "items": { "type": "object", "properties": { "id": {"type":"string"}, "label": {"type":"string"}, "models": { "type": "array", "items": { "type": "object", "properties": { "id": {"type":"string"}, "label": {"type":"string"} } } } } } } } },
|
||||
"picker.choice": { "payload": { "picker_id": { "type": "string" }, "title": { "type": "string" }, "choices": { "type": "array", "items": { "type": "object", "properties": { "value": {"type":"string"}, "label": {"type":"string"}, "is_current": {"type":"boolean"} } } } } },
|
||||
"picker.clarify": { "payload": { "picker_id": { "type": "string" }, "question": { "type": "string" }, "choices": { "type": "array", "items": { "type": "object" } } } },
|
||||
"picker.approval": { "payload": { "picker_id": { "type": "string" }, "command": { "type": "string" }, "description": { "type": "string" } } },
|
||||
"picker.confirm": { "payload": { "picker_id": { "type": "string" }, "title": { "type": "string" }, "message": { "type": "string" } } },
|
||||
"channel.list": { "payload": { "channels": { "type": "array", "items": { "$ref": "#/definitions/channel" } } } },
|
||||
"channel.created": { "payload": { "$ref": "#/definitions/channel" } },
|
||||
"channel.renamed": { "payload": { "chat_id": { "type": "string" }, "name": { "type": "string" } } },
|
||||
"channel.deleted": { "payload": { "chat_id": { "type": "string" } } },
|
||||
"history": { "description": "Response to history request; page of messages oldest→newest.", "payload": { "messages": { "type": "array", "items": { "type": "object", "properties": { "message_id": {"type":"string"}, "role": {"type":"string"}, "text": {"type":"string"}, "reasoning": {"type":"string"}, "model": {"type":"string"}, "tokens": {"type":"integer"}, "ts": {"type":"integer"} } } }, "has_more": { "type": "boolean" }, "oldest_message_id": { "type": "string" } } },
|
||||
"commands.catalog": { "description": "Full slash-command catalog.", "payload": { "commands": { "type": "array", "items": { "type": "object", "properties": { "name": {"type":"string"}, "description": {"type":"string"}, "args_hint": {"type":"string"}, "category": {"type":"string"} } } } } },
|
||||
"commands.complete": { "description": "Autocomplete matches for a typed prefix.", "payload": { "prefix": { "type": "string" }, "matches": { "type": "array", "items": { "type": "object", "properties": { "name": {"type":"string"}, "description": {"type":"string"}, "args_hint": {"type":"string"} } } } } },
|
||||
"agent.busy": { "description": "Agent is processing; app shows thinking indicator.", "payload": { "reason": { "type": "string", "enum": ["processing", "tool", "waiting_input", "cron"] } } },
|
||||
"agent.idle": { "description": "Agent turn complete; clear thinking indicator.", "payload": {} },
|
||||
"search.results": { "payload": { "query": { "type": "string" }, "scope": { "type": "string", "enum": ["all", "chat"] }, "hits": { "type": "array", "items": { "type": "object", "properties": { "message_id": {"type":"string"}, "chat_id": {"type":"string"}, "thread_id": {"type":["string","null"]}, "role": {"type":"string"}, "snippet": {"type":"string"}, "ts": {"type":"integer"} } } } } },
|
||||
"media.offer": { "description": "Agent-sent media available; app pulls bytes.", "payload": { "$ref": "#/definitions/media_ref" } },
|
||||
"status": { "payload": { "state": { "type": "string", "enum": ["online", "restarting", "degraded"] }, "session": { "type": "object" } } },
|
||||
"error": { "payload": { "code": { "type": "string", "enum": ["auth", "not_found", "rate_limited", "media_too_large", "unsupported", "internal"] }, "message": { "type": "string" } } },
|
||||
"pong": { "payload": { "ts": { "type": "integer" } } },
|
||||
"sync.done": { "payload": { "cursor": { "type": "integer" } } },
|
||||
"media.pull.end": { "payload": { "ok": { "type": "boolean" } } }
|
||||
},
|
||||
"app_to_server": {
|
||||
"hello": { "description": "First frame; auth + caps.", "payload": { "token": { "type": "string" }, "device_id": { "type": "string" }, "device_name": { "type": "string" }, "caps": { "type": "object", "properties": { "min_protocol": {"type":"integer"}, "media": {"type":"boolean"}, "push": {"type":"string"} } }, "fcm_token": { "type": "string" }, "ntfy_topic": { "type": "string" } } },
|
||||
"message.send": { "payload": { "text": { "type": "string" }, "reply_to": { "type": "string" }, "media_refs": { "type": "array", "items": { "type": "string" } } } },
|
||||
"media.upload.start": { "payload": { "media_ref": { "type": "string" }, "kind": { "$ref": "#/definitions/kind" }, "mime": { "type": "string" }, "size": { "type": "integer" }, "filename": { "type": "string" } } },
|
||||
"media.upload.end": { "payload": { "media_ref": { "type": "string" }, "sha256": { "type": "string" } } },
|
||||
"media.pull": { "payload": { "media_id": { "type": "string" } } },
|
||||
"picker.select": { "payload": { "picker_id": { "type": "string" }, "value": { "type": "string" } } },
|
||||
"channel.create": { "payload": { "name": { "type": "string" }, "kind": { "type": "string", "enum": ["channel", "thread"] }, "parent_chat_id": { "type": ["string", "null"] } } },
|
||||
"channel.rename": { "payload": { "name": { "type": "string" } } },
|
||||
"channel.set_default": { "payload": {} },
|
||||
"channel.delete": { "payload": {} },
|
||||
"search": { "payload": { "query": { "type": "string" }, "scope": { "type": "string", "enum": ["all", "chat"] }, "chat_id": { "type": "string" }, "thread_id": { "type": "string" } } },
|
||||
"history": { "description": "Load a page of messages (initial open / scroll-up).", "payload": { "before_message_id": { "type": "string" }, "limit": { "type": "integer", "minimum": 1, "maximum": 200 } } },
|
||||
"commands.catalog": { "description": "Fetch full slash-command catalog.", "payload": {} },
|
||||
"commands.complete": { "description": "Autocomplete for typed /prefix.", "payload": { "prefix": { "type": "string" } } },
|
||||
"agent.stop": { "description": "Abort current agent turn.", "payload": {} },
|
||||
"agent.steer": { "description": "Inject steering message mid-turn.", "payload": { "text": { "type": "string" } } },
|
||||
"read.receipt": { "description": "User viewed message; server stores + broadcasts to other devices.", "payload": { "chat_id": { "type": "string" }, "message_id": { "type": "string" } } },
|
||||
"sync": { "description": "Reconnect catch-up; replays undelivered outbox frames only (not full history).", "payload": { "cursor": { "type": "integer" } } },
|
||||
"fcm.register": { "payload": { "fcm_token": { "type": "string" }, "ntfy_topic": { "type": "string" } } },
|
||||
"ping": { "payload": { "ts": { "type": "integer" } } }
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"kind": { "type": "string", "enum": ["image", "audio", "video", "document", "voice"] },
|
||||
"channel": { "type": "object", "properties": { "chat_id": {"type":"string"}, "name": {"type":"string"}, "kind": {"type":"string","enum":["default","channel","thread"]}, "parent_chat_id": {"type":["string","null"]}, "is_default": {"type":"boolean"} } },
|
||||
"media_ref": { "type": "object", "properties": { "media_id": {"type":"string"}, "kind": { "$ref": "#/definitions/kind" }, "mime": {"type":"string"}, "size": {"type":"integer"}, "filename": {"type":"string"} } }
|
||||
},
|
||||
"reliability": {
|
||||
"ordering": "Per-connection (TCP/WS). message.update for a message_id is monotonic; app may coalesce to latest.",
|
||||
"never_dropped": ["message", "message.stop", "tool.end", "notification", "picker.*", "channel.*", "agent.busy", "agent.idle", "history", "commands.catalog", "commands.complete", "search.results", "error"],
|
||||
"coalescable_under_backpressure": ["message.update", "tool.progress"],
|
||||
"offline": "Undelivered frames go to the outbox; replayed by sync. Terminal frames always outboxed."
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user