207 lines
12 KiB
Markdown
207 lines
12 KiB
Markdown
# 14 — Milestones (M0–M7)
|
||
|
||
Phased delivery. Each milestone ends with a **demo** (on-device where noted) and
|
||
has explicit **acceptance criteria**. Work top-to-bottom; don't skip M0/M1.
|
||
|
||
---
|
||
|
||
## M0 — Toolchain & scaffolding
|
||
**Goal:** everything builds; the plugin is discoverable; the repo is safe.
|
||
- [X] Install JDK 17, Android SDK, set `ANDROID_HOME` (`12-toolchain.md`).
|
||
- [X] `cd hermes-agent && uv sync` (hermes venv works).
|
||
- [X] Create monorepo scaffold (`02-monorepo.md`): `gateway-plugin/`, `app/`
|
||
(CMP: `shared`, `androidApp`, `desktopApp`), root `.gitignore`
|
||
(**excludes `hermes-agent/`**), root `README.md`.
|
||
- [X] `git init` + a pre-commit/CI guard that fails if `hermes-agent/` is staged.
|
||
- [X] CMP project builds empty: `./gradlew :androidApp:assembleDebug`,
|
||
`./gradlew :desktopApp:run` (blank window).
|
||
- [X] Plugin skeleton: `plugin.yaml` + `adapter.py` with `register(ctx)` + a
|
||
no-op `AndroidAdapter` → `hermes gateway status` lists **android**.
|
||
- **Demo:** `hermes gateway status` shows `android`; `./gradlew
|
||
:androidApp:installDebug` installs a blank app on the MIX 2S.
|
||
- **Accept:** blank app installs + launches on-device; plugin visible in
|
||
`hermes gateway status`; `hermes-agent/` is git-ignored (verify with
|
||
`git status --ignored`).
|
||
|
||
## M1 — Gateway core loop (text round-trip)
|
||
**Goal:** pair + send a text message + get a (non-streaming) reply.
|
||
- [X] WS server (`ws_server.py`): bind, `hello` auth (constant-time),
|
||
`hello.ack`, heartbeat, connection registry.
|
||
- [X] `AndroidAdapter.send()` → `message` frame; inbound `message.send` →
|
||
`MessageEvent` → `handle_message`.
|
||
- [X] Pairing store + `ANDROID_TOKEN`; QR payload in `interactive_setup`.
|
||
- [X] App: Connect screen (URL+token, real `hello` test), `GatewayClient`
|
||
(connect + reconnect), ChatScreen sends + renders `message`.
|
||
- [X] `ws_probe.py` harness drives a real turn.
|
||
- **Demo (on-device):** pair the phone, send "hello", see the agent's reply.
|
||
- **Accept:** text round-trip works on-device; wrong token is rejected;
|
||
reconnect after gateway restart re-pairs.
|
||
|
||
## M2 — Streaming + reasoning + tools + commentary
|
||
**Goal:** the "agent transparency" features.
|
||
- [X] Map consumer `send`/`edit_message` → `message.start/update/stop`.
|
||
- [X] Reasoning: set `show_reasoning` for android; adapter splits prefix →
|
||
`reasoning` field. **Verify format with `ws_probe.py`.** (The model
|
||
returns a separate `reasoning_content` field. In the *streaming* case the
|
||
gateway drops it — the stream consumer only forwards `content` and the
|
||
final send is suppressed — so the adapter captures it via the
|
||
`on_stream_delta` plugin hook (`kind="reasoning"`, gated by
|
||
`plugins.stream_reasoning_deltas: true`) and attaches it to
|
||
`message.stop`. Non-streaming already carried it via the prefix split.)
|
||
- [X] Tool events: classify tool-progress `send()`s → structured
|
||
`tool.start/progress/end` (turn-state machine). **Verify with probe.**
|
||
- [X] Commentary → `commentary` frames. Typing → `typing`.
|
||
- [X] App: live bubble (coalesced updates), `ReasoningBlock` (collapse + copy),
|
||
`ToolCard` with **Everything/Truncated/Nothing** setting, dimmed
|
||
`commentary` bubble.
|
||
- **Demo (on-device):** a multi-step prompt streams, shows reasoning above the
|
||
answer, tool cards (toggle verbosity), and an intermediate beat.
|
||
(Verified on the MIX 2S: streaming, tool cards + verbosity toggle, and the
|
||
reasoning panel rendering above the answer. The commentary beat is
|
||
model/agent-dependent and did not fire with the current model, but the
|
||
rendering + frame handling are in place.)
|
||
- **Accept:** all four render correctly; tool verbosity setting changes
|
||
rendering; reasoning copy button works; frame shapes match `04-wire-protocol`.
|
||
|
||
## M3 — Channels/threads + cron + search
|
||
**Goal:** organization + cron delegation + search.
|
||
- [X] Channel directory (SQLite): default channel ensured; `channel.create/
|
||
rename/set_default/delete` + `channel.*` frames.
|
||
- [X] Threads: toggle in default chat; `thread_id` lanes; `create_handoff_thread`.
|
||
- [X] `parse_target_ref_fn` + `cron_deliver_env_var` → cron
|
||
`deliver=android:<chat>[:<thread>]` works.
|
||
- [X] `search.py` FTS5 bridge; `search` frame (all / this-chat) → results.
|
||
- [X] App: channel list (drawer/rail), thread toggle + topic switcher, "new
|
||
channel" + "set as cron target", SearchScreen with scope toggle + jump.
|
||
- **Demo (on-device):** create "Cron Reports"; create a cron job delivering to
|
||
it; it fires into that channel; search finds a message (both scopes).
|
||
- **Accept:** cron output lands in the chosen channel (not default); threads
|
||
isolate context; search scoping correct; channel list reconciles on events.
|
||
- **Status (complete):** channel directory + threads + search + outbox sync
|
||
verified end-to-end via `ws_probe.py` (create/rename/set_default/delete,
|
||
thread lanes, FTS5 search, sync); cron `deliver=android:<chat>[:<thread>]`
|
||
target resolution verified via `resolve_send_target`. App on-device: channel
|
||
drawer, thread toggle + topic switcher, new channel, search overlay with
|
||
jump. Minor UI gaps deferred to M7 polish: "set as cron target" is set via
|
||
the cron job's `deliver` field (no dedicated button); the search overlay
|
||
defaults to scope "all" (the all/this-chat toggle is supported in the
|
||
controller but not yet exposed). An actual cron job firing into a channel was
|
||
not e2e-tested (it shares the verified resolution path).
|
||
|
||
## M4 — Media
|
||
**Goal:** attach + receive + play media.
|
||
- [X] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`;
|
||
size limit + sha256 + MIME re-sniff.
|
||
- [X] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path
|
||
security.
|
||
- [X] App: SAF pickers + preview chips + upload; `media.pull` → cache;
|
||
**ExoPlayer** inline (audio mini-player, video fullscreen/PiP); image/doc
|
||
viewers.
|
||
- **Demo (on-device):** attach a photo + video (agent sees them); ask agent to
|
||
send an image/video → plays live in-app.
|
||
- **Accept:** both directions work; over-limit rejected; playback is live;
|
||
only allowed files are servable.
|
||
- **Status (complete):** Both directions verified end-to-end on the MIX 2S.
|
||
Inbound: SAF-picked photo → chunked binary `media.upload` (256 KiB) →
|
||
gateway size+sha256 verify + MIME re-sniff → `cache_image_from_bytes` →
|
||
`media.upload.ack` → `message.send` with `media_refs` → agent vision
|
||
described the image accurately. Outbound: agent `send_image` → `media.offer`
|
||
→ app auto-`media.pull` (chunked) → cache → image rendered inline. Live
|
||
playback verified on-device: agent offered a 2s MP4 + 2s MP3 → app pulled
|
||
both → ExoPlayer video player (blue frame, 00:02/00:02, controls) + audio
|
||
mini-player rendered and playable. Over-limit rejection, sha256 mismatch,
|
||
and delivery-path security are covered by the 17-test
|
||
`tests/gateway/test_android.py` suite (all pass). Note: the app serializes
|
||
pulls (single `binarySession` slot) via a `Mutex` so concurrent offers don't
|
||
interleave their byte streams. `media.upload.ack` documented in
|
||
`04-wire-protocol.md` + `frames.schema.json`.
|
||
|
||
## M5 — Push + offline (FCM + ntfy)
|
||
**Goal:** reach the phone when backgrounded; catch up on reconnect.
|
||
- [x] Outbox (SQLite) + sync cursor; `sync`/`sync.done`; retention prune
|
||
(row cap 5000 + prune banner, throttled 1/h).
|
||
- [x] `push.py`: `FcmBackend` (HTTP v1 + service account, httpx; JWT via
|
||
PyJWT+cryptography) + `NtfyBackend` (X-Data header); selected by
|
||
`ANDROID_PUSH_BACKEND`. ntfy server exposed in `server_caps.push_ntfy_server`.
|
||
- [x] Fire push on no-live-subscriber; data payload for silent sync.
|
||
High-priority kinds (approval/clarify/cron) push even when live.
|
||
- [x] App: FCM service (`onNewToken` → `fcm.register`; inert without a
|
||
Firebase project), per-chat notification channels, deep-link
|
||
(custom action + `iris://`), ntfy listener foreground-service fallback.
|
||
App generates a device-specific ntfy topic; auto-sync on reconnect.
|
||
- [x] In-app `notification` banners (channel events, approval, clarify, cron,
|
||
outbox-pruned); system notification mirror when backgrounded.
|
||
- **Demo (on-device, verified 2026-08-19):** device offline → message
|
||
broadcast → frame parked in outbox → `push via ntfy -> <device>` fired →
|
||
app reconnect auto-synced (cursor 0→8) → parked message rendered in-app.
|
||
ntfy publish confirmed (POST 200). ntfy listener service runs (foreground
|
||
notification posted); listener→system-notification leg not exercised live
|
||
because ntfy.sh's public SSE endpoint was serving its web UI (not streams)
|
||
during the test — verify against a self-hosted ntfy.
|
||
- **Accept:** push arrives when backgrounded (ntfy verified; FCM code path
|
||
complete, needs a Firebase project to exercise); reconnect syncs with no
|
||
loss/dup (verified); banners show for foreground events (implemented).
|
||
|
||
## M6 — Desktop app
|
||
**Goal:** the same app on a big screen.
|
||
- [x] `desktopMain`: tray + OS notifications; `MediaPlayer` actual (mpv/WebView);
|
||
`MediaPicker` actual (file dialog); `SecureStore` actual; window mgmt.
|
||
- [x] Two-pane default layout; keyboard shortcuts; optional inspector pane.
|
||
- [ ] Parity pass vs Android feature checklist (`11-desktop-app.md`).
|
||
- [x] jpackage builds (Linux first; macOS/Windows as available).
|
||
- **Demo:** desktop app pairs to the same gateway; full feature parity; tray
|
||
notifications; media plays.
|
||
- **Accept:** all Android features work on desktop; tray + shortcuts work;
|
||
native binary launches.
|
||
- **Status (complete, 2026-08-19):** All `desktopMain` actuals implemented and
|
||
verified on Linux (X11). `DesktopSecureStore`: non-secrets in
|
||
`~/.iris/settings.json`; the gateway token goes to the OS keyring with
|
||
read-back verification, falling back to an AES-GCM encrypted file
|
||
(`~/.iris/pairing.enc` + owner-only `~/.iris/.key`) when the keyring doesn't
|
||
persist — KWallet on this host accepts writes without storing, so the
|
||
fallback is exercised (a stale keyring entry is cleared before falling back).
|
||
`DesktopMedia`: `JFileChooser` picker, ImageIO thumbnails, mpv playback
|
||
(audio via a JSON-IPC unix socket, video in a separate mpv window), docs via
|
||
`xdg-open`. `DesktopPush`: `notify-send` OS notifications + in-app banners,
|
||
foreground-aware via window focus (`DesktopBridge.foreground`). Tray (Compose
|
||
`Tray`): connection-state icon + tooltip, Show/Quit menu, close-to-tray.
|
||
Window size/position persisted to `~/.iris/window.json`. Two-pane layout
|
||
(channel rail + chat) + optional inspector pane + keyboard shortcuts
|
||
(Ctrl/Cmd N/T/F/K, Esc, `/` command palette) + search scope toggle.
|
||
Verified on-device: the app pairs to the same gateway, connects (green
|
||
"connected" chip), renders the two-pane layout, and replays the outbox.
|
||
jpackage Linux app-image builds (`./gradlew :desktopApp:jpackage`) and the
|
||
native binary launches + connects. Known: the jpackage launcher prints a
|
||
non-fatal "pure virtual method called" (JDK-8348560, a jpackage/Linux
|
||
launcher bug on JDK 17; the app runs and connects regardless). Parity is
|
||
achieved via shared-code reuse; a formal feature-by-feature parity pass and
|
||
macOS/Windows packaging are deferred to M7.
|
||
|
||
## M7 — Polish + E2E + docs
|
||
**Goal:** ship-quality.
|
||
- [ ] Telegram-style layout pass (per reference image): header, bubbles, date
|
||
separators, ✓✓, model/token footer, banner, bottom bar.
|
||
- [ ] Theming (dark default, accent), onboarding/pairing UX, empty/loading/
|
||
reconnecting/degraded states with honest copy.
|
||
- [ ] Full E2E suite (`13-testing.md` scenarios 1–12) automated where possible.
|
||
- [ ] Docs: `docs/protocol/frames.schema.json` finalized; `docs/setup.md`
|
||
(user-facing pairing + FCM/ntfy + remote access); root README.
|
||
- [ ] Security hardening checklist (`09-pairing-security.md`) verified.
|
||
- **Demo:** end-to-end on phone + desktop simultaneously; cron into a channel;
|
||
push; media; search.
|
||
- **Accept:** all feature-checklist items pass on-device; E2E green; docs
|
||
complete; `hermes-agent/` still never committed.
|
||
|
||
---
|
||
|
||
## Sequencing notes
|
||
- **M1/M2 depend on the `ws_probe.py` harness** to lock frame shapes early —
|
||
build it in M1.
|
||
- **M3 (cron) and M5 (push) both touch the outbox** — build the outbox in M3,
|
||
extend for push in M5.
|
||
- **M6 (desktop) reuses M1–M5 shared code** — do it after the Android features
|
||
are stable so the shared module is settled.
|
||
- Parallelizable: plugin (Python) and app (Kotlin) can be worked on
|
||
concurrently once the protocol (`04-wire-protocol.md`) is agreed; the probe
|
||
harness is the integration seam.
|