13 — Testing & Debugging (incl. ADB on-device)

Three layers: Python plugin tests, Kotlin unit/UI tests, and on-device E2E via ADB. Plus a WS test-client harness to drive the real gateway without the app (critical for verifying frame shapes early).

13.1 Python plugin tests

Location: gateway-plugin/tests/ (and, for hermes-integration tests, mirror into the hermes tests/gateway/test_android.py pattern when running under hermes's suite).

Run with hermes's hermetic runner (never bare pytest):

cd hermes-agent
scripts/run_tests.sh tests/gateway/test_android.py
scripts/run_tests.sh                          # full suite (CI parity)

Coverage to write (behavioral, not change-detector — per hermes test policy):
- register(ctx) produces a valid PlatformEntry (name, cron env var, parse_target_ref).
- check_requirements / validate_config / is_connected truth table.
- _parse_target_ref: iris:<chat>, iris:<chat>:<thread>, non-android → None.
- Reasoning split: given a show_reasoning-style final text, send() emits message {reasoning, text} correctly; no-prefix → no reasoning field.
- Outbox: frame with no subscriber → written with a cursor; sync replays the right range; retention prunes.
- Pairing: valid token → hello.ack; wrong token → error auth + close. Constant-time compare used.
- Media: upload start→binary→end reassembles + sha256 verified; over-limit → media_too_large; pull serves only allowed paths.
- Push backend selection: fcm vs ntfy chosen by config; configured() reflects missing creds.
- Channel directory: create/rename/set_default; cron target resolution.
- Auto-threading: message.send {auto_thread:true} in a flat lane mints a thread (derived name, channel.created {auto:true}) and the echo/event carry the new thread_id; no-op with an existing thread_id, for slash commands, or for media-only sends; the LLM upgrade renames the thread (channel.renamed).
Slash catalog: commands.catalog request → response with the gateway-available COMMAND_REGISTRY subset + plugin commands (each entry name/description/args_hint/category/aliases); cli_only commands excluded; response id matches the request id.
No ~/.hermes writes in tests — use the _isolate_hermes_home fixture pattern (temp HERMES_HOME). Profile tests also mock Path.home().

13.2 WS test-client harness (do this FIRST, in M1/M2)

A small Python script (gateway-plugin/tests/ws_probe.py) that connects to the real running gateway and drives a turn, printing every frame. This is how we empirically confirm the exact frame shapes (especially tool-progress vs commentary classification and the reasoning prefix) before/while building the Kotlin client.

hermes gateway &                      # with the android plugin
python gateway-plugin/tests/ws_probe.py --token <IRIS_TOKEN> \
    --send "list the files and summarize"
# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end,
#         commentary, message.stop {reasoning,…}, …

Use it to lock 04-wire-protocol.md against reality and to debug the adapter without waiting for the app.

13.3 Kotlin unit / UI tests

Protocol codec: round-trip every frame type (serialize → deserialize → equal); unknown type/fields ignored (forward-compat).
Repositories: merge-don't-clobber on channel.*; optimistic send + rollback on error; sync dedupe by message_id; cursor monotonic.
GatewayClient: reconnect/backoff; message.update coalescing (latest wins); request/response correlation by id.
ViewModels: state transitions (connecting→connected→reconnecting); tool verbosity filtering (everything/truncated/nothing); reasoning present/absent.
Compose UI tests: ReasoningBlock collapse/expand + copy; ToolCard spinner→done; composer auto-grow cap; channel list active highlight.
Run: ./gradlew :shared:testDebugUnitTest (and :shared:testDesktopTest).

13.4 On-device E2E via ADB (the MIX 2S, API 29)

Device: a5ca2a4b (Xiaomi MIX 2S). Workflow:

# install + launch
cd app
./gradlew :androidApp:installDebug
adb shell am start -n dev.iris.app/.MainActivity

# logs (filter our tags + WS + FCM + ExoPlayer)
adb logcat -c
adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec"

# screenshots for visual checks
adb exec-out screencap -p > /tmp/shot.png

# clear app data between pairing attempts
adb shell pm clear dev.iris.app

# push a file to the app's cache (for media tests) / pull logs
adb shell run-as dev.iris.app ls files
adb logcat -d > /tmp/logcat.txt

E2E scenarios (script where possible):

Pair: connect screen → enter URL+token → hello.ack → main. (Verify auth leg, not just TCP.)
Text round-trip: send "hello" → streamed reply appears (message.start → updates → stop).
Reasoning: ask a reasoning-model question → ReasoningBlock shows above the answer; copy button works.
Tools: trigger a tool (e.g. "list files") → ToolCard shows; toggle verbosity in Settings → rendering changes.
Intermediate: a multi-step prompt → commentary bubble appears dimmed.
Channels: create "Cron Reports" → appears in list; set as cron target.
Cron delivery: create a cron job deliver=iris:chan_<n> → it fires → lands in that channel (not default).
Search: "search everywhere" vs "this chat" → correct scoping; tap → jump.
Media (in): attach a photo + a video → agent receives (vision) → reply.
Media (out): ask the agent to send an image/video → media.offer → inline player plays it live.
Push: background the app (adb shell am start another app / lock) → trigger a message → FCM notification appears → tap → syncs + deep-links.
Reconnect/sync: kill the WS (stop gateway briefly) → restart → app reconnects → sync catches up (no lost/dup messages).
Auto-threading: Threads on → send a message in the Default channel's flat lane → a new topic appears (derived name), the app jumps into it, the reply streams there, and the topic is renamed to the AI's title a moment later. Slash commands / media-only sends stay in the flat lane.
Slash drawer: type / in the composer → the drawer rolls up over the input with the gateway catalog; keep typing → fuzzy filtering (unrelated entries drop out); tap a row → the command is sent and the drawer closes; type an unknown command → the drawer closes (the raw text can still be sent; hermes answers with its unknown-command reply).
HTTP fallback (docs/19): with the WS port unreachable (e.g. the gateway bound WS to a dead port, or a firewall dropping 8790 but not 8791), the app stays sendable: the status pill shows "connected · http" (green), a sent message echoes back within ~1 s and the agent reply streams in over SSE; the attach button is disabled (media needs the live WS). When the WS comes back the pill returns to "connected" and media works again. Automated: e2e.py scenario 13 (health + POST /v1/frame + SSE turn, user-echo < 1.5 s) and ws_probe.py --http (same assertion flags as the WS leg).

13.5 Debugging tips

Gateway side: ~/.hermes/logs/gateway.log (and hermes logs --follow). Our plugin logs under the iris adapter name; secrets redacted.
WS framing bugs: use the ws_probe.py harness — it isolates the protocol from the app.
Streaming jitter: the consumer edits at intervals; if updates look chunky, check display.platforms.iris.streaming and the consumer's edit interval.
Media pull stalls: check chunk size + backpressure; confirm the file is within hermes delivery roots (validate_media_delivery_path).
FCM not arriving: confirm the token registered (devices.db), the service account can mint a token, and the app's onNewToken re-registered after pm clear.
Profile leaks: if tokens look wrong under multiple profiles, verify the scope-aware secret read (_get_scoped_secret) is used.

8.2 KiB Raw Blame History