14 — Milestones (M0–M7)

Phased delivery. Each milestone ends with a demo (on-device where noted) and has explicit acceptance criteria. Work top-to-bottom; don't skip M0/M1.

M0 — Toolchain & scaffolding

Goal: everything builds; the plugin is discoverable; the repo is safe.

Install JDK 17, Android SDK, set ANDROID_HOME (12-toolchain.md).
cd hermes-agent && uv sync (hermes venv works).
Create monorepo scaffold (02-monorepo.md): gateway-plugin/, app/ (CMP: shared, androidApp, desktopApp), root .gitignore (excludes hermes-agent/), root README.md.
git init + a pre-commit/CI guard that fails if hermes-agent/ is staged.
CMP project builds empty: ./gradlew :androidApp:assembleDebug, ./gradlew :desktopApp:run (blank window).
Plugin skeleton: plugin.yaml + adapter.py with register(ctx) + a no-op AndroidAdapter → hermes gateway status lists android.
Demo: hermes gateway status shows android; ./gradlew :androidApp:installDebug installs a blank app on the MIX 2S.
Accept: blank app installs + launches on-device; plugin visible in hermes gateway status; hermes-agent/ is git-ignored (verify with git status --ignored).

M1 — Gateway core loop (text round-trip)

Goal: pair + send a text message + get a (non-streaming) reply.

WS server (ws_server.py): bind, hello auth (constant-time), hello.ack, heartbeat, connection registry.
AndroidAdapter.send() → message frame; inbound message.send → MessageEvent → handle_message.
Pairing store + ANDROID_TOKEN; QR payload in interactive_setup.
App: Connect screen (URL+token, real hello test), GatewayClient (connect + reconnect), ChatScreen sends + renders message.
ws_probe.py harness drives a real turn.
Demo (on-device): pair the phone, send "hello", see the agent's reply.
Accept: text round-trip works on-device; wrong token is rejected; reconnect after gateway restart re-pairs.

M2 — Streaming + reasoning + tools + commentary

Goal: the "agent transparency" features.

Map consumer send/edit_message → message.start/update/stop.
Reasoning: set show_reasoning for android; adapter splits prefix → reasoning field. Verify format with ws_probe.py. (The model returns a separate reasoning_content field. In the streaming case the gateway drops it — the stream consumer only forwards content and the final send is suppressed — so the adapter captures it via the on_stream_delta plugin hook (kind="reasoning", gated by plugins.stream_reasoning_deltas: true) and attaches it to message.stop. Non-streaming already carried it via the prefix split.)
Tool events: classify tool-progress send()s → structured tool.start/progress/end (turn-state machine). Verify with probe.
Commentary → commentary frames. Typing → typing.
App: live bubble (coalesced updates), ReasoningBlock (collapse + copy), ToolCard with Everything/Truncated/Nothing setting, dimmed commentary bubble.
Demo (on-device): a multi-step prompt streams, shows reasoning above the answer, tool cards (toggle verbosity), and an intermediate beat. (Verified on the MIX 2S: streaming, tool cards + verbosity toggle, and the reasoning panel rendering above the answer. The commentary beat is model/agent-dependent and did not fire with the current model, but the rendering + frame handling are in place.)
Accept: all four render correctly; tool verbosity setting changes rendering; reasoning copy button works; frame shapes match 04-wire-protocol.

M3 — Channels/threads + cron + search

Goal: organization + cron delegation + search.

Channel directory (SQLite): default channel ensured; channel.create/ rename/set_default/delete + channel.* frames.
Threads: toggle in default chat; thread_id lanes; create_handoff_thread.
parse_target_ref_fn + cron_deliver_env_var → cron deliver=android:<chat>[:<thread>] works.
search.py FTS5 bridge; search frame (all / this-chat) → results.
App: channel list (drawer/rail), thread toggle + topic switcher, "new channel" + "set as cron target", SearchScreen with scope toggle + jump.
Demo (on-device): create "Cron Reports"; create a cron job delivering to it; it fires into that channel; search finds a message (both scopes).
Accept: cron output lands in the chosen channel (not default); threads isolate context; search scoping correct; channel list reconciles on events.
Status (complete): channel directory + threads + search + outbox sync verified end-to-end via ws_probe.py (create/rename/set_default/delete, thread lanes, FTS5 search, sync); cron deliver=android:<chat>[:<thread>] target resolution verified via resolve_send_target. App on-device: channel drawer, thread toggle + topic switcher, new channel, search overlay with jump. Minor UI gaps deferred to M7 polish: "set as cron target" is set via the cron job's deliver field (no dedicated button); the search overlay defaults to scope "all" (the all/this-chat toggle is supported in the controller but not yet exposed). An actual cron job firing into a channel was not e2e-tested (it shares the verified resolution path).

M4 — Media

Goal: attach + receive + play media.

Inbound: media.upload chunked → cache_*_from_bytes → media_urls; size limit + sha256 + MIME re-sniff.
Outbound: send_* → media.offer; media.pull chunked; delivery-path security.
App: SAF pickers + preview chips + upload; media.pull → cache; ExoPlayer inline (audio mini-player, video fullscreen/PiP); image/doc viewers.
Demo (on-device): attach a photo + video (agent sees them); ask agent to send an image/video → plays live in-app.
Accept: both directions work; over-limit rejected; playback is live; only allowed files are servable.
Status (complete): Both directions verified end-to-end on the MIX 2S. Inbound: SAF-picked photo → chunked binary media.upload (256 KiB) → gateway size+sha256 verify + MIME re-sniff → cache_image_from_bytes → media.upload.ack → message.send with media_refs → agent vision described the image accurately. Outbound: agent send_image → media.offer → app auto-media.pull (chunked) → cache → image rendered inline. Live playback verified on-device: agent offered a 2s MP4 + 2s MP3 → app pulled both → ExoPlayer video player (blue frame, 00:02/00:02, controls) + audio mini-player rendered and playable. Over-limit rejection, sha256 mismatch, and delivery-path security are covered by the 17-test tests/gateway/test_android.py suite (all pass). Note: the app serializes pulls (single binarySession slot) via a Mutex so concurrent offers don't interleave their byte streams. media.upload.ack documented in 04-wire-protocol.md + frames.schema.json.

M5 — Push + offline (FCM + ntfy)

Goal: reach the phone when backgrounded; catch up on reconnect.

Outbox (SQLite) + sync cursor; sync/sync.done; retention prune (row cap 5000 + prune banner, throttled 1/h).
push.py: FcmBackend (HTTP v1 + service account, httpx; JWT via PyJWT+cryptography) + NtfyBackend (X-Data header); selected by ANDROID_PUSH_BACKEND. ntfy server exposed in server_caps.push_ntfy_server.
Fire push on no-live-subscriber; data payload for silent sync. High-priority kinds (approval/clarify/cron) push even when live.
App: FCM service (onNewToken → fcm.register; inert without a Firebase project), per-chat notification channels, deep-link (custom action + iris://), ntfy listener foreground-service fallback. App generates a device-specific ntfy topic; auto-sync on reconnect.
In-app notification banners (channel events, approval, clarify, cron, outbox-pruned); system notification mirror when backgrounded.
Demo (on-device, verified 2026-08-19): device offline → message broadcast → frame parked in outbox → push via ntfy -> <device> fired → app reconnect auto-synced (cursor 0→8) → parked message rendered in-app. ntfy publish confirmed (POST 200). ntfy listener service runs (foreground notification posted); listener→system-notification leg not exercised live because ntfy.sh's public SSE endpoint was serving its web UI (not streams) during the test — verify against a self-hosted ntfy.
Accept: push arrives when backgrounded (ntfy verified; FCM code path complete, needs a Firebase project to exercise); reconnect syncs with no loss/dup (verified); banners show for foreground events (implemented).

M6 — Desktop app

Goal: the same app on a big screen.

desktopMain: tray + OS notifications; MediaPlayer actual (mpv/WebView); MediaPicker actual (file dialog); SecureStore actual; window mgmt.
Two-pane default layout; keyboard shortcuts; optional inspector pane.
Parity pass vs Android feature checklist (11-desktop-app.md).
jpackage builds (Linux first; macOS/Windows as available).
Demo: desktop app pairs to the same gateway; full feature parity; tray notifications; media plays.
Accept: all Android features work on desktop; tray + shortcuts work; native binary launches.
Status (complete, 2026-08-19): All desktopMain actuals implemented and verified on Linux (X11). DesktopSecureStore: non-secrets in ~/.iris/settings.json; the gateway token goes to the OS keyring with read-back verification, falling back to an AES-GCM encrypted file (~/.iris/pairing.enc + owner-only ~/.iris/.key) when the keyring doesn't persist — KWallet on this host accepts writes without storing, so the fallback is exercised (a stale keyring entry is cleared before falling back). DesktopMedia: JFileChooser picker, ImageIO thumbnails, mpv playback (audio via a JSON-IPC unix socket, video in a separate mpv window), docs via xdg-open. DesktopPush: notify-send OS notifications + in-app banners, foreground-aware via window focus (DesktopBridge.foreground). Tray (Compose Tray): connection-state icon + tooltip, Show/Quit menu, close-to-tray. Window size/position persisted to ~/.iris/window.json. Two-pane layout (channel rail + chat) + optional inspector pane + keyboard shortcuts (Ctrl/Cmd N/T/F/K, Esc, / command palette) + search scope toggle. Verified on-device: the app pairs to the same gateway, connects (green "connected" chip), renders the two-pane layout, and replays the outbox. jpackage Linux app-image builds (./gradlew :desktopApp:jpackage) and the native binary launches + connects. Known: the jpackage launcher prints a non-fatal "pure virtual method called" (JDK-8348560, a jpackage/Linux launcher bug on JDK 17; the app runs and connects regardless). Parity is achieved via shared-code reuse; a formal feature-by-feature parity pass and macOS/Windows packaging are deferred to M7.

M7 — Polish + E2E + docs

Goal: ship-quality.

Telegram-style layout pass (per reference image): header, bubbles, date separators, ✓✓, model/token footer, banner, bottom bar.
Theming (dark default, accent), onboarding/pairing UX, empty/loading/ reconnecting/degraded states with honest copy.
Full E2E suite (13-testing.md scenarios 1–12) automated where possible.
Docs: docs/protocol/frames.schema.json finalized; docs/setup.md (user-facing pairing + FCM/ntfy + remote access); root README.
Security hardening checklist (09-pairing-security.md) verified.
Demo: end-to-end on phone + desktop simultaneously; cron into a channel; push; media; search.
Accept: all feature-checklist items pass on-device; E2E green; docs complete; hermes-agent/ still never committed.
Status (2026-08-20): Layout pass verified on-device (MIX 2S): header with avatar + "Bot" subtitle + overflow menu (rename channel, forget pairing), centered date-separator pill, bubbles with in-bubble timestamps, user ✓/✓✓ driven by the new read.receipt frame (gateway emits it when the agent takes the message; late-joining clients also get the current status state on hello.ack), model/token footer, letter-avatar channel rail/drawer with active highlight, restyled bottom bar. Theming centralized in ui/theme/Theme.kt (dark default, single accent; all hard-coded colors replaced). States: dedicated connecting screen, reconnecting + degraded/restarting banners (new status frame), send-failure rollback with tap-to-retry. E2E: tests/e2e.py driver automates scenarios 1–12 against the live gateway — 9 PASS / 2 PARTIAL (push device-notification leg + gateway-kill leg are manual) / 1 SKIP (commentary is model-dependent) / 0 FAIL; ws_probe.py gained --assert-turn/ reasoning/tools/commentary/read-receipt/status plus --search, --channel-*, --watch modes. Docs: frames.schema.json finalized (mirrors code exactly; 17 unimplemented frames moved to x-planned-frames; 6 deltas + 3 drift fixes), docs/setup.md added (pairing + FCM/ntfy + remote access + troubleshooting), root README quickstart + status updated. Security: inbound JSON-frame rate limit (20/s, burst 40, binary upload chunks exempt) with rate_limited error + close; Android token moved to EncryptedSharedPreferences with one-time plain→encrypted migration; .pre-commit-config.yaml commits the hermes-agent guard; 09-pairing-security.md M7 verification table (5 items verified, 3 documented gaps: in-app QR scan, WSS cert pinning, mechanical log redaction). Known: commentary scenario is model-dependent (SKIP); FCM path needs a Firebase project to exercise; M6's formal desktop parity pass + macOS/Windows packaging remain open.

Sequencing notes

M1/M2 depend on the ws_probe.py harness to lock frame shapes early — build it in M1.
M3 (cron) and M5 (push) both touch the outbox — build the outbox in M3, extend for push in M5.
M6 (desktop) reuses M1–M5 shared code — do it after the Android features are stable so the shared module is settled.
Parallelizable: plugin (Python) and app (Kotlin) can be worked on concurrently once the protocol (04-wire-protocol.md) is agreed; the probe harness is the integration seam.

14 KiB Raw Blame History Unescape Escape