iris_x_hermes/docs/14-milestones.md

# 14 — Milestones (M0–M8)

Phased delivery. Each milestone ends with a **demo** (on-device where noted) and
has explicit **acceptance criteria**. Work top-to-bottom; don't skip M0/M1.

---

## M0 — Toolchain & scaffolding

**Goal:** everything builds; the plugin is discoverable; the repo is safe.

- [X] Install JDK 17, Android SDK, set `ANDROID_HOME` (`12-toolchain.md`).
- [X] `cd hermes-agent && uv sync` (hermes venv works).
- [X] Create monorepo scaffold (`02-monorepo.md`): `gateway-plugin/`, `app/`
      (CMP: `shared`, `androidApp`, `desktopApp`), root `.gitignore`
      (**excludes `hermes-agent/`**), root `README.md`.
- [X] `git init` + a pre-commit/CI guard that fails if `hermes-agent/` is staged.
- [X] CMP project builds empty: `./gradlew :androidApp:assembleDebug`,
      `./gradlew :desktopApp:run` (blank window).
- [X] Plugin skeleton: `plugin.yaml` + `adapter.py` with `register(ctx)` + a
      no-op `IrisAdapter` → `hermes gateway status` lists **iris**.
- **Demo:** `hermes gateway status` shows `iris`; `./gradlew
  :androidApp:installDebug` installs a blank app on the MIX 2S.
- **Accept:** blank app installs + launches on-device; plugin visible in
  `hermes gateway status`; `hermes-agent/` is git-ignored (verify with
  `git status --ignored`).

## M1 — Gateway core loop (text round-trip)

**Goal:** pair + send a text message + get a (non-streaming) reply.

- [X] WS server (`ws_server.py`): bind, `hello` auth (constant-time),
      `hello.ack`, heartbeat, connection registry.
- [X] `IrisAdapter.send()` → `message` frame; inbound `message.send` →
      `MessageEvent` → `handle_message`.
- [X] Pairing store + `IRIS_TOKEN`; QR payload in `interactive_setup`.
- [X] App: Connect screen (URL+token, real `hello` test), `GatewayClient`
      (connect + reconnect), ChatScreen sends + renders `message`.
- [X] `ws_probe.py` harness drives a real turn.
- **Demo (on-device):** pair the phone, send "hello", see the agent's reply.
- **Accept:** text round-trip works on-device; wrong token is rejected;
  reconnect after gateway restart re-pairs.

## M2 — Streaming + reasoning + tools + commentary

**Goal:** the "agent transparency" features.

- [X] Map consumer `send`/`edit_message` → `message.start/update/stop`.
- [X] Reasoning: set `show_reasoning` for iris; adapter splits prefix →
      `reasoning` field. **Verify format with `ws_probe.py`.** (The model
      returns a separate `reasoning_content` field. In the *streaming* case the
      gateway drops it — the stream consumer only forwards `content` and the
      final send is suppressed — so the adapter captures it via the
      `on_stream_delta` plugin hook (`kind="reasoning"`, gated by
      `plugins.stream_reasoning_deltas: true`) and attaches it to
      `message.stop`. Non-streaming already carried it via the prefix split.)
- [X] Tool events: classify tool-progress `send()`s → structured
      `tool.start/progress/end` (turn-state machine). **Verify with probe.**
- [X] Commentary → `commentary` frames. Typing → `typing`.
- [X] App: live bubble (coalesced updates), `ReasoningBlock` (collapse + copy),
      `ToolCard` with **Everything/Truncated/Nothing** setting, dimmed
      `commentary` bubble.
- **Demo (on-device):** a multi-step prompt streams, shows reasoning above the
  answer, tool cards (toggle verbosity), and an intermediate beat.
  (Verified on the MIX 2S: streaming, tool cards + verbosity toggle, and the
  reasoning panel rendering above the answer. The commentary beat is
  model/agent-dependent and did not fire with the current model, but the
  rendering + frame handling are in place.)
- **Accept:** all four render correctly; tool verbosity setting changes
  rendering; reasoning copy button works; frame shapes match `04-wire-protocol`.

## M3 — Channels/threads + cron + search

**Goal:** organization + cron delegation + search.

- [X] Channel directory (SQLite): default channel ensured; `channel.create/
      rename/set_default/delete` + `channel.*` frames.
- [X] Threads: toggle in default chat; `thread_id` lanes; `create_handoff_thread`.
- [X] `parse_target_ref_fn` + `cron_deliver_env_var` → cron
      `deliver=iris:<chat>[:<thread>]` works.
- [X] `search.py` FTS5 bridge; `search` frame (all / this-chat) → results.
- [X] App: channel list (drawer/rail), thread toggle + topic switcher, "new
      channel" + "set as cron target", SearchScreen with scope toggle + jump.
- **Demo (on-device):** create "Cron Reports"; create a cron job delivering to
  it; it fires into that channel; search finds a message (both scopes).
- **Accept:** cron output lands in the chosen channel (not default); threads
  isolate context; search scoping correct; channel list reconciles on events.
- **Status (complete):** channel directory + threads + search + outbox sync
  verified end-to-end via `ws_probe.py` (create/rename/set_default/delete,
  thread lanes, FTS5 search, sync); cron `deliver=iris:<chat>[:<thread>]`
  target resolution verified via `resolve_send_target`. App on-device: channel
  drawer, thread toggle + topic switcher, new channel, search overlay with
  jump. Minor UI gaps deferred to M7 polish: "set as cron target" is set via
  the cron job's `deliver` field (no dedicated button); the search overlay
  defaults to scope "all" (the all/this-chat toggle is supported in the
  controller but not yet exposed). An actual cron job firing into a channel was
  not e2e-tested (it shares the verified resolution path).

## M4 — Media

**Goal:** attach + receive + play media.

- [X] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
      size limit + sha256 + MIME re-sniff.
- [X] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
      security.
- [X] App: SAF pickers + preview chips + upload; `media.pull` → cache;
      **ExoPlayer** inline (audio mini-player, video fullscreen/PiP); image/doc
      viewers.
- **Demo (on-device):** attach a photo + video (agent sees them); ask agent to
  send an image/video → plays live in-app.
- **Accept:** both directions work; over-limit rejected; playback is live;
  only allowed files are servable.
- **Status (complete):** Both directions verified end-to-end on the MIX 2S.
  Inbound: SAF-picked photo → chunked binary `media.upload` (256 KiB) →
  gateway size+sha256 verify + MIME re-sniff → `cache_image_from_bytes` →
  `media.upload.ack` → `message.send` with `media_refs` → agent vision
  described the image accurately. Outbound: agent `send_image` → `media.offer`
  → app auto-`media.pull` (chunked) → cache → image rendered inline. Live
  playback verified on-device: agent offered a 2s MP4 + 2s MP3 → app pulled
  both → ExoPlayer video player (blue frame, 00:02/00:02, controls) + audio
  mini-player rendered and playable. Over-limit rejection, sha256 mismatch,
  and delivery-path security are covered by the 17-test
  `tests/gateway/test_android.py` suite (all pass). Note: the app serializes
  pulls (single `binarySession` slot) via a `Mutex` so concurrent offers don't
  interleave their byte streams. `media.upload.ack` documented in
  `04-wire-protocol.md` + `frames.schema.json`.

## M5 — Push + offline (FCM + ntfy)

**Goal:** reach the phone when backgrounded; catch up on reconnect.

- [x] Outbox (SQLite) + sync cursor; `sync`/`sync.done`; retention prune
      (row cap 5000 + prune banner, throttled 1/h).
- [x] `push.py`: `FcmBackend` (HTTP v1 + service account, httpx; JWT via
      PyJWT+cryptography) + `NtfyBackend` (X-Data header); selected by
      `IRIS_PUSH_BACKEND`. ntfy server exposed in `server_caps.push_ntfy_server`.
- [x] Fire push on no-live-subscriber; data payload for silent sync.
      High-priority kinds (approval/clarify/cron) push even when live.
- [x] App: FCM service (`onNewToken` → `fcm.register`; inert without a
      Firebase project), per-chat notification channels, deep-link
      (custom action + `iris://`), ntfy listener foreground-service fallback.
      App generates a device-specific ntfy topic; auto-sync on reconnect.
- [x] In-app `notification` banners (channel events, approval, clarify, cron,
      outbox-pruned); system notification mirror when backgrounded.
- **Demo (on-device, verified 2026-08-19):** device offline → message
  broadcast → frame parked in outbox → `push via ntfy -> <device>` fired →
  app reconnect auto-synced (cursor 0→8) → parked message rendered in-app.
  ntfy publish confirmed (POST 200). ntfy listener service runs (foreground
  notification posted); listener→system-notification leg not exercised live
  because ntfy.sh's public SSE endpoint was serving its web UI (not streams)
  during the test — verify against a self-hosted ntfy.
- **Accept:** push arrives when backgrounded (ntfy verified; FCM code path
  complete, needs a Firebase project to exercise); reconnect syncs with no
  loss/dup (verified); banners show for foreground events (implemented).

## M6 — Desktop app

**Goal:** the same app on a big screen.

- [x] `desktopMain`: tray + OS notifications; `MediaPlayer` actual (mpv/WebView);
      `MediaPicker` actual (file dialog); `SecureStore` actual; window mgmt.
- [x] Two-pane default layout; keyboard shortcuts; optional inspector pane.
- [ ] Parity pass vs Iris app feature checklist (`11-desktop-app.md`).
- [x] jpackage builds (Linux first; macOS/Windows as available).
- **Demo:** desktop app pairs to the same gateway; full feature parity; tray
  notifications; media plays.
- **Accept:** all Iris app features work on desktop; tray + shortcuts work;
  native binary launches.
- **Status (complete, 2026-08-19):** All `desktopMain` actuals implemented and
  verified on Linux (X11). `DesktopSecureStore`: non-secrets in
  `~/.iris/settings.json`; the gateway token goes to the OS keyring with
  read-back verification, falling back to an AES-GCM encrypted file
  (`~/.iris/pairing.enc` + owner-only `~/.iris/.key`) when the keyring doesn't
  persist — KWallet on this host accepts writes without storing, so the
  fallback is exercised (a stale keyring entry is cleared before falling back).
  `DesktopMedia`: `JFileChooser` picker, ImageIO thumbnails, mpv playback
  (audio via a JSON-IPC unix socket, video in a separate mpv window), docs via
  `xdg-open`. `DesktopPush`: `notify-send` OS notifications + in-app banners,
  foreground-aware via window focus (`DesktopBridge.foreground`). Tray (Compose
  `Tray`): connection-state icon + tooltip, Show/Quit menu, close-to-tray.
  Window size/position persisted to `~/.iris/window.json`. Two-pane layout
  (channel rail + chat) + optional inspector pane + keyboard shortcuts
  (Ctrl/Cmd N/T/F/K, Esc, `/` command palette) + search scope toggle.
  Verified on-device: the app pairs to the same gateway, connects (green
  "connected" chip), renders the two-pane layout, and replays the outbox.
  jpackage Linux app-image builds (`./gradlew :desktopApp:jpackage`) and the
  native binary launches + connects. Known: the jpackage launcher prints a
  non-fatal "pure virtual method called" (JDK-8348560, a jpackage/Linux
  launcher bug on JDK 17; the app runs and connects regardless). Parity is
  achieved via shared-code reuse; a formal feature-by-feature parity pass and
  macOS/Windows packaging are deferred to M7.

## M7 — Polish + E2E + docs

**Goal:** ship-quality.

- [x] Telegram-style layout pass (per reference image): header, bubbles, date
      separators, ✓✓, model/token footer, banner, bottom bar.
- [x] Theming (dark default, accent), onboarding/pairing UX, empty/loading/
      reconnecting/degraded states with honest copy.
- [x] Full E2E suite (`13-testing.md` scenarios 1–12) automated where possible.
- [x] Docs: `docs/protocol/frames.schema.json` finalized; `docs/setup.md`
      (user-facing pairing + FCM/ntfy + remote access); root README.
- [x] Security hardening checklist (`09-pairing-security.md`) verified.
- **Demo:** end-to-end on phone + desktop simultaneously; cron into a channel;
  push; media; search.
- **Accept:** all feature-checklist items pass on-device; E2E green; docs
  complete; `hermes-agent/` still never committed.
- **Status (2026-08-20):** Layout pass verified on-device (MIX 2S): header
  with avatar + "Bot" subtitle + overflow menu (rename channel, forget
  pairing), centered date-separator pill, bubbles with in-bubble timestamps,
  user ✓/✓✓ driven by the new `read.receipt` frame (gateway emits it when the
  agent takes the message; late-joining clients also get the current `status`
  state on hello.ack), model/token footer, letter-avatar channel rail/drawer
  with active highlight, restyled bottom bar. Theming centralized in
  `ui/theme/Theme.kt` (dark default, single accent; all hard-coded colors
  replaced). States: dedicated connecting screen, reconnecting +
  degraded/restarting banners (new `status` frame), send-failure rollback
  with tap-to-retry. E2E: `tests/e2e.py` driver automates scenarios 1–12
  against the live gateway — 9 PASS / 2 PARTIAL (push device-notification
  leg + gateway-kill leg are manual) / 1 SKIP (commentary is
  model-dependent) / 0 FAIL; `ws_probe.py` gained `--assert-turn/
  reasoning/tools/commentary/read-receipt/status` plus `--search`,
  `--channel-*`, `--watch` modes. Docs: `frames.schema.json` finalized
  (mirrors code exactly; 17 unimplemented frames moved to
  `x-planned-frames`; 6 deltas + 3 drift fixes), `docs/setup.md` added
  (pairing + FCM/ntfy + remote access + troubleshooting), root README
  quickstart + status updated. Security: inbound JSON-frame rate limit
  (20/s, burst 40, binary upload chunks exempt) with `rate_limited` error +
  close; Android token moved to EncryptedSharedPreferences with one-time
  plain→encrypted migration; `.pre-commit-config.yaml` commits the
  hermes-agent guard; `09-pairing-security.md` M7 verification table (5
  items verified, 3 documented gaps: in-app QR scan, WSS cert pinning,
  mechanical log redaction). Known: commentary scenario is model-dependent
  (SKIP); FCM path needs a Firebase project to exercise; M6's formal
  desktop parity pass + macOS/Windows packaging remain open.

---

## M8 — QR pairing

**Goal:** QR-based pairing — a scannable QR at `hermes gateway setup` plus an
in-app scanner and `iris://pair` deep link (closes gap #12, `docs/20`).

- [x] Pure-stdlib QR encoder + terminal renderer (`gateway-plugin/qr.py`):
      byte mode, EC M with L fallback, versions 1–10, ISO penalty masking,
      **zero new Python deps**.
- [x] `interactive_setup` renders the QR after the pairing URL (text lines
      stay as the primary path).
- [x] `PairLink` parser (`iris/util/PairLink.kt`) + jvmTest (valid/missing
      token/bad port/wrong scheme/wrong host/percent-encoded/secure/default
      port).
- [x] CameraX + ML Kit scanner (`QrScanActivity`, on-device, no Play
      services) + Connect-screen **Scan QR** button (Android only; hidden on
      desktop) + `CAMERA` permission.
- [x] `iris://pair` deep link (system-scanner / other-phone fallback) reusing
      the same parser.
- **Accept:** `docs/20` §20.6; gap #12 in `09-pairing-security.md` closed.
- **Status (2026-08-22):** Encoder cross-checked byte-for-byte against an
  independent reference and decoded by an independent decoder (zbarimg); fixed
  v1-M and v7-M matrix vectors lock the algorithm. `hermes gateway setup`
  prints a scannable QR (v7-M, 45 modules) for the 64-hex-token payload. App:
  Connect screen shows **Scan QR** (Android), which opens `QrScanActivity`
  (CameraX camera2 + ML Kit barcode), requests `CAMERA`, and pre-fills URL +
  token via `PairLink.parse` without auto-connecting; `iris://pair` deep link
  pre-fills the same way. Docs updated per `docs/20` Part C.

---

## Sequencing notes

- **M1/M2 depend on the `ws_probe.py` harness** to lock frame shapes early —
  build it in M1.
- **M3 (cron) and M5 (push) both touch the outbox** — build the outbox in M3,
  extend for push in M5.
- **M6 (desktop) reuses M1–M5 shared code** — do it after the Iris app features
  are stable so the shared module is settled.
- Parallelizable: plugin (Python) and app (Kotlin) can be worked on
  concurrently once the protocol (`04-wire-protocol.md`) is agreed; the probe
  harness is the integration seam.