iris_x_hermes/docs/13-testing.md

13 — Testing & Debugging (incl. ADB on-device)

Three layers: Python plugin tests, Kotlin unit/UI tests, and on-device E2E via ADB. Plus a WS test-client harness to drive the real gateway without the app (critical for verifying frame shapes early).

13.1 Python plugin tests

• Location: tests/ (and, for hermes-integration tests, mirror into the hermes tests/gateway/test_android.py pattern when running under hermes's suite).

• Run with hermes's hermetic runner (never bare pytest):

  cd hermes-agent
  scripts/run_tests.sh tests/gateway/test_android.py
  scripts/run_tests.sh                          # full suite (CI parity)
  

• Coverage to write (behavioral, not change-detector — per hermes test policy):

  • register(ctx) produces a valid PlatformEntry (name, cron env var, parse_target_ref).
  • check_requirements / validate_config / is_connected truth table.
  • _parse_target_ref: iris:<chat>, iris:<chat>:<thread>, non-iris → None.
  • Reasoning split: given a show_reasoning-style final text, send() emits message {reasoning, text} correctly; no-prefix → no reasoning field.
  • Outbox: frame with no subscriber → written with a cursor; sync replays the right range; retention prunes.
  • Pairing: valid token → hello.ack; wrong token → error auth + close. Constant-time compare used.
  • Media: upload start→binary→end reassembles + sha256 verified; over-limit → media_too_large; pull serves only allowed paths.
  • Push backend selection: fcm vs ntfy chosen by config; configured() reflects missing creds.
  • Channel directory: create/rename/set_default; cron target resolution.
  • Auto-threading: message.send {auto_thread:true} in a flat lane mints a thread (derived name, channel.created {auto:true}) and the echo/event carry the new thread_id; no-op with an existing thread_id, for slash commands, or for media-only sends; the LLM upgrade renames the thread (channel.renamed).

• Slash catalog: commands.catalog request → response with the gateway-available COMMAND_REGISTRY subset + plugin commands (each entry name/description/args_hint/category/aliases); cli_only commands excluded; response id matches the request id.

• No ~/.hermes writes in tests — use the _isolate_hermes_home fixture pattern (temp HERMES_HOME). Profile tests also mock Path.home().

13.2 WS test-client harness (do this FIRST, in M1/M2)

A small Python script (tests/ws_probe.py) that connects to the real running gateway and drives a turn, printing every frame. This is how we empirically confirm the exact frame shapes (especially tool-progress vs commentary classification and the reasoning prefix) before/while building the Kotlin client.

hermes gateway &                      # with the iris plugin
python tests/ws_probe.py --token <IRIS_TOKEN> \
    --send "list the files and summarize"
# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end,
#         commentary, message.stop {reasoning,…}, …

Use it to lock 04-wire-protocol.md against reality and to debug the adapter without waiting for the app.

13.3 Kotlin unit / UI tests

• Protocol codec: round-trip every frame type (serialize → deserialize → equal); unknown type/fields ignored (forward-compat).
• Repositories: merge-don't-clobber on channel.*; optimistic send + rollback on error; sync dedupe by message_id; cursor monotonic.
• GatewayClient: reconnect/backoff; message.update coalescing (latest wins); request/response correlation by id.
• ViewModels: state transitions (connecting→connected→reconnecting); tool verbosity filtering (everything/truncated/nothing); reasoning present/absent.
• Compose UI tests: ReasoningBlock collapse/expand + copy; ToolCard spinner→done; composer auto-grow cap; channel list active highlight.
• Run: ./gradlew :shared:testDebugUnitTest (and :shared:testDesktopTest).

13.4 On-device E2E via ADB (the MIX 2S, API 29)

Device: a5ca2a4b (Xiaomi MIX 2S). Workflow:

# install + launch
cd app
./gradlew :androidApp:installDebug
adb shell am start -n dev.iris.app/.MainActivity

# logs (filter our tags + WS + FCM + ExoPlayer)
adb logcat -c
adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec"

# screenshots for visual checks
adb exec-out screencap -p > /tmp/shot.png

# clear app data between pairing attempts
adb shell pm clear dev.iris.app

# push a file to the app's cache (for media tests) / pull logs
adb shell run-as dev.iris.app ls files
adb logcat -d > /tmp/logcat.txt

E2E scenarios (script where possible):

1. Pair: connect screen → enter URL+token → hello.ack → main. (Verify auth leg, not just TCP.)
2. Text round-trip: send "hello" → streamed reply appears (message.start → updates → stop).
3. Reasoning: ask a reasoning-model question → ReasoningBlock shows above the answer; copy button works.
4. Tools: trigger a tool (e.g. "list files") → ToolCard shows; toggle verbosity in Settings → rendering changes.
5. Intermediate: a multi-step prompt → commentary bubble appears dimmed.
6. Channels: create "Cron Reports" → appears in list; set as cron target.
7. Cron delivery: create a cron job deliver=iris:chan_<n> → it fires → lands in that channel (not default).
8. Search: "search everywhere" vs "this chat" → correct scoping; tap → jump.
9. Media (in): attach a photo + a video → agent receives (vision) → reply.
10. Media (out): ask the agent to send an image/video → media.offer → inline player plays it live.
11. Push: background the app (adb shell am start another app / lock) → trigger a message → FCM notification appears → tap → syncs + deep-links.
12. Reconnect/sync: kill the WS (stop gateway briefly) → restart → app reconnects → sync catches up (no lost/dup messages).
13. Auto-threading: Threads on → send a message in the Default channel's flat lane → a new topic appears (derived name), the app jumps into it, the reply streams there, and the topic is renamed to the AI's title a moment later. Slash commands / media-only sends stay in the flat lane.
14. Slash drawer: type / in the composer → the drawer rolls up over the input with the gateway catalog; keep typing → fuzzy filtering (unrelated entries drop out); tap a row → the command is sent and the drawer closes; type an unknown command → the drawer closes (the raw text can still be sent; hermes answers with its unknown-command reply).
15. HTTP fallback (docs/19): with the WS port unreachable (e.g. the gateway bound WS to a dead port, or a firewall dropping 8790 but not 8791), the app stays sendable: the status pill shows "connected · http" (green), a sent message echoes back within ~1 s and the agent reply streams in over SSE; the attach button is disabled (media needs the live WS). When the WS comes back the pill returns to "connected" and media works again. Automated: e2e.py scenario 13 (health + POST /v1/frame + SSE turn, user-echo < 1.5 s) and ws_probe.py --http (same assertion flags as the WS leg).

13.5 Debugging tips

• Gateway side: ~/.hermes/logs/gateway.log (and hermes logs --follow). Our plugin logs under the iris adapter name; secrets redacted.
• WS framing bugs: use the ws_probe.py harness — it isolates the protocol from the app.
• Streaming jitter: the consumer edits at intervals; if updates look chunky, check display.platforms.iris.streaming and the consumer's edit interval.
• Media pull stalls: check chunk size + backpressure; confirm the file is within hermes delivery roots (validate_media_delivery_path).
• FCM not arriving: confirm the token registered (devices.db), the service account can mint a token, and the app's onNewToken re-registered after pm clear.
• Profile leaks: if tokens look wrong under multiple profiles, verify the scope-aware secret read (_get_scoped_secret) is used.