iris_x_hermes/docs/13-testing.md

# 13 — Testing & Debugging (incl. ADB on-device)

Three layers: **Python plugin tests**, **Kotlin unit/UI tests**, and **on-device
E2E via ADB**. Plus a **WS test-client harness** to drive the real gateway
without the app (critical for verifying frame shapes early).

## 13.1 Python plugin tests

- Location: `tests/` (and, for hermes-integration tests, mirror
  into the hermes `tests/gateway/test_android.py` pattern when running under
  hermes's suite).
- **Run with hermes's hermetic runner** (never bare `pytest`):

  ```bash
  cd hermes-agent
  scripts/run_tests.sh tests/gateway/test_android.py
  scripts/run_tests.sh                          # full suite (CI parity)
  ```

- Coverage to write (behavioral, not change-detector — per hermes test policy):
  - `register(ctx)` produces a valid `PlatformEntry` (name, cron env var,
    parse_target_ref).
  - `check_requirements` / `validate_config` / `is_connected` truth table.
  - `_parse_target_ref`: `iris:<chat>`, `iris:<chat>:<thread>`, non-iris
    → None.
  - **Reasoning split:** given a `show_reasoning`-style final text, `send()`
    emits `message {reasoning, text}` correctly; no-prefix → no reasoning field.
  - **Outbox:** frame with no subscriber → written with a cursor; `sync` replays
    the right range; retention prunes.
  - **Pairing:** valid token → `hello.ack`; wrong token → `error auth` + close.
    Constant-time compare used.
  - **Media:** upload start→binary→end reassembles + sha256 verified; over-limit
    → `media_too_large`; pull serves only allowed paths.
  - **Push backend selection:** fcm vs ntfy chosen by config; `configured()`
    reflects missing creds.
  - **Channel directory:** create/rename/set_default; cron target resolution.
  - **Auto-threading:** `message.send {auto_thread:true}` in a flat lane mints a
    thread (derived name, `channel.created {auto:true}`) and the echo/event
    carry the new `thread_id`; no-op with an existing `thread_id`, for slash
    commands, or for media-only sends; the LLM upgrade renames the thread
    (`channel.renamed`).
- **Slash catalog:** `commands.catalog` request → response with the
    gateway-available `COMMAND_REGISTRY` subset + plugin commands (each entry
    `name`/`description`/`args_hint`/`category`/`aliases`); `cli_only` commands
    excluded; response `id` matches the request `id`.
- **No `~/.hermes` writes in tests** — use the `_isolate_hermes_home` fixture
  pattern (temp `HERMES_HOME`). Profile tests also mock `Path.home()`.

## 13.2 WS test-client harness (do this FIRST, in M1/M2)

A small Python script (`tests/ws_probe.py`) that connects to the
**real running gateway** and drives a turn, printing every frame. This is how we
**empirically confirm** the exact frame shapes (especially tool-progress vs
commentary classification and the reasoning prefix) before/while building the
Kotlin client.

```bash
hermes gateway &                      # with the iris plugin
python tests/ws_probe.py --token <IRIS_TOKEN> \
    --send "list the files and summarize"
# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end,
#         commentary, message.stop {reasoning,…}, …
```

Use it to lock `04-wire-protocol.md` against reality and to debug the adapter
without waiting for the app.

## 13.3 Kotlin unit / UI tests

- **Protocol codec:** round-trip every frame type (serialize → deserialize →
  equal); unknown `type`/fields ignored (forward-compat).
- **Repositories:** merge-don't-clobber on `channel.*`; optimistic send + rollback
  on `error`; sync dedupe by `message_id`; cursor monotonic.
- **`GatewayClient`:** reconnect/backoff; `message.update` coalescing (latest
  wins); request/response correlation by `id`.
- **ViewModels:** state transitions (connecting→connected→reconnecting); tool
  verbosity filtering (everything/truncated/nothing); reasoning present/absent.
- **Compose UI tests:** ReasoningBlock collapse/expand + copy; ToolCard
  spinner→done; composer auto-grow cap; channel list active highlight.
- Run: `./gradlew :shared:testDebugUnitTest` (and `:shared:testDesktopTest`).

## 13.4 On-device E2E via ADB (the MIX 2S, API 29)

Device: `a5ca2a4b` (Xiaomi MIX 2S). Workflow:

```bash
# install + launch
cd app
./gradlew :androidApp:installDebug
adb shell am start -n dev.iris.app/.MainActivity

# logs (filter our tags + WS + FCM + ExoPlayer)
adb logcat -c
adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec"

# screenshots for visual checks
adb exec-out screencap -p > /tmp/shot.png

# clear app data between pairing attempts
adb shell pm clear dev.iris.app

# push a file to the app's cache (for media tests) / pull logs
adb shell run-as dev.iris.app ls files
adb logcat -d > /tmp/logcat.txt
```

**E2E scenarios (script where possible):**

1. **Pair:** connect screen → enter URL+token → `hello.ack` → main. (Verify auth
   leg, not just TCP.)
2. **Text round-trip:** send "hello" → streamed reply appears (message.start →
   updates → stop).
3. **Reasoning:** ask a reasoning-model question → ReasoningBlock shows above the
   answer; copy button works.
4. **Tools:** trigger a tool (e.g. "list files") → ToolCard shows; toggle
   verbosity in Settings → rendering changes.
5. **Intermediate:** a multi-step prompt → commentary bubble appears dimmed.
6. **Channels:** create "Cron Reports" → appears in list; set as cron target.
7. **Cron delivery:** create a cron job `deliver=iris:chan_<n>` → it
   fires → lands in that channel (not default).
8. **Search:** "search everywhere" vs "this chat" → correct scoping; tap → jump.
9. **Media (in):** attach a photo + a video → agent receives (vision) → reply.
10. **Media (out):** ask the agent to send an image/video → `media.offer` →
    inline player plays it live.
11. **Push:** background the app (`adb shell am start` another app / lock) →
    trigger a message → FCM notification appears → tap → syncs + deep-links.
12. **Reconnect/sync:** kill the WS (stop gateway briefly) → restart → app
    reconnects → `sync` catches up (no lost/dup messages).
13. **Auto-threading:** Threads on → send a message in the Default channel's
    flat lane → a new topic appears (derived name), the app jumps into it, the
    reply streams there, and the topic is renamed to the AI's title a moment
    later. Slash commands / media-only sends stay in the flat lane.
14. **Slash drawer:** type `/` in the composer → the drawer rolls up over the
    input with the gateway catalog; keep typing → fuzzy filtering (unrelated
    entries drop out); tap a row → the command is sent and the drawer closes;
    type an unknown command → the drawer closes (the raw text can still be
    sent; hermes answers with its unknown-command reply).
15. **HTTP fallback (docs/19):** with the WS port unreachable (e.g. the
    gateway bound WS to a dead port, or a firewall dropping 8790 but not
    8791), the app stays sendable: the status pill shows "connected · http"
    (green), a sent message echoes back within ~1 s and the agent reply
    streams in over SSE; the attach button is disabled (media needs the
    live WS). When the WS comes back the pill returns to "connected" and
    media works again. Automated: `e2e.py` scenario 13 (health +
    `POST /v1/frame` + SSE turn, user-echo < 1.5 s) and
    `ws_probe.py --http` (same assertion flags as the WS leg).

## 13.5 Debugging tips

- **Gateway side:** `~/.hermes/logs/gateway.log` (and `hermes logs --follow`).
  Our plugin logs under the `iris` adapter name; secrets redacted.
- **WS framing bugs:** use the `ws_probe.py` harness — it isolates the protocol
  from the app.
- **Streaming jitter:** the consumer edits at intervals; if updates look chunky,
  check `display.platforms.iris.streaming` and the consumer's edit interval.
- **Media pull stalls:** check chunk size + backpressure; confirm the file is
  within hermes delivery roots (`validate_media_delivery_path`).
- **FCM not arriving:** confirm the token registered (`devices.db`), the service
  account can mint a token, and the app's `onNewToken` re-registered after
  `pm clear`.
- **Profile leaks:** if tokens look wrong under multiple profiles, verify the
  scope-aware secret read (`_get_scoped_secret`) is used.