ARIA
a61b47a947
The plugin is named 'iris' (IrisAdapter, IRIS_HOME_CHANNEL, label Iris), but several docs still referred to it as the android platform/plugin and to the product as 'the Android app'. Rename name-mentions to iris/IRIS and product-mentions to 'Iris app'; keep legitimate OS references (androidApp, Android SDK, Android 10, androidx, test_android.py, ...). Also includes pi-lens markdown-lint autofixes (table spacing, trailing newlines) in the touched files.
163 lines
8.2 KiB
Markdown
163 lines
8.2 KiB
Markdown
# 13 — Testing & Debugging (incl. ADB on-device)
|
|
|
|
Three layers: **Python plugin tests**, **Kotlin unit/UI tests**, and **on-device
|
|
E2E via ADB**. Plus a **WS test-client harness** to drive the real gateway
|
|
without the app (critical for verifying frame shapes early).
|
|
|
|
## 13.1 Python plugin tests
|
|
|
|
- Location: `gateway-plugin/tests/` (and, for hermes-integration tests, mirror
|
|
into the hermes `tests/gateway/test_android.py` pattern when running under
|
|
hermes's suite).
|
|
- **Run with hermes's hermetic runner** (never bare `pytest`):
|
|
|
|
```bash
|
|
cd hermes-agent
|
|
scripts/run_tests.sh tests/gateway/test_android.py
|
|
scripts/run_tests.sh # full suite (CI parity)
|
|
```
|
|
|
|
- Coverage to write (behavioral, not change-detector — per hermes test policy):
|
|
- `register(ctx)` produces a valid `PlatformEntry` (name, cron env var,
|
|
parse_target_ref).
|
|
- `check_requirements` / `validate_config` / `is_connected` truth table.
|
|
- `_parse_target_ref`: `iris:<chat>`, `iris:<chat>:<thread>`, non-iris
|
|
→ None.
|
|
- **Reasoning split:** given a `show_reasoning`-style final text, `send()`
|
|
emits `message {reasoning, text}` correctly; no-prefix → no reasoning field.
|
|
- **Outbox:** frame with no subscriber → written with a cursor; `sync` replays
|
|
the right range; retention prunes.
|
|
- **Pairing:** valid token → `hello.ack`; wrong token → `error auth` + close.
|
|
Constant-time compare used.
|
|
- **Media:** upload start→binary→end reassembles + sha256 verified; over-limit
|
|
→ `media_too_large`; pull serves only allowed paths.
|
|
- **Push backend selection:** fcm vs ntfy chosen by config; `configured()`
|
|
reflects missing creds.
|
|
- **Channel directory:** create/rename/set_default; cron target resolution.
|
|
- **Auto-threading:** `message.send {auto_thread:true}` in a flat lane mints a
|
|
thread (derived name, `channel.created {auto:true}`) and the echo/event
|
|
carry the new `thread_id`; no-op with an existing `thread_id`, for slash
|
|
commands, or for media-only sends; the LLM upgrade renames the thread
|
|
(`channel.renamed`).
|
|
- **Slash catalog:** `commands.catalog` request → response with the
|
|
gateway-available `COMMAND_REGISTRY` subset + plugin commands (each entry
|
|
`name`/`description`/`args_hint`/`category`/`aliases`); `cli_only` commands
|
|
excluded; response `id` matches the request `id`.
|
|
- **No `~/.hermes` writes in tests** — use the `_isolate_hermes_home` fixture
|
|
pattern (temp `HERMES_HOME`). Profile tests also mock `Path.home()`.
|
|
|
|
## 13.2 WS test-client harness (do this FIRST, in M1/M2)
|
|
|
|
A small Python script (`gateway-plugin/tests/ws_probe.py`) that connects to the
|
|
**real running gateway** and drives a turn, printing every frame. This is how we
|
|
**empirically confirm** the exact frame shapes (especially tool-progress vs
|
|
commentary classification and the reasoning prefix) before/while building the
|
|
Kotlin client.
|
|
|
|
```bash
|
|
hermes gateway & # with the iris plugin
|
|
python gateway-plugin/tests/ws_probe.py --token <IRIS_TOKEN> \
|
|
--send "list the files and summarize"
|
|
# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end,
|
|
# commentary, message.stop {reasoning,…}, …
|
|
```
|
|
|
|
Use it to lock `04-wire-protocol.md` against reality and to debug the adapter
|
|
without waiting for the app.
|
|
|
|
## 13.3 Kotlin unit / UI tests
|
|
|
|
- **Protocol codec:** round-trip every frame type (serialize → deserialize →
|
|
equal); unknown `type`/fields ignored (forward-compat).
|
|
- **Repositories:** merge-don't-clobber on `channel.*`; optimistic send + rollback
|
|
on `error`; sync dedupe by `message_id`; cursor monotonic.
|
|
- **`GatewayClient`:** reconnect/backoff; `message.update` coalescing (latest
|
|
wins); request/response correlation by `id`.
|
|
- **ViewModels:** state transitions (connecting→connected→reconnecting); tool
|
|
verbosity filtering (everything/truncated/nothing); reasoning present/absent.
|
|
- **Compose UI tests:** ReasoningBlock collapse/expand + copy; ToolCard
|
|
spinner→done; composer auto-grow cap; channel list active highlight.
|
|
- Run: `./gradlew :shared:testDebugUnitTest` (and `:shared:testDesktopTest`).
|
|
|
|
## 13.4 On-device E2E via ADB (the MIX 2S, API 29)
|
|
|
|
Device: `a5ca2a4b` (Xiaomi MIX 2S). Workflow:
|
|
|
|
```bash
|
|
# install + launch
|
|
cd app
|
|
./gradlew :androidApp:installDebug
|
|
adb shell am start -n dev.iris.app/.MainActivity
|
|
|
|
# logs (filter our tags + WS + FCM + ExoPlayer)
|
|
adb logcat -c
|
|
adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec"
|
|
|
|
# screenshots for visual checks
|
|
adb exec-out screencap -p > /tmp/shot.png
|
|
|
|
# clear app data between pairing attempts
|
|
adb shell pm clear dev.iris.app
|
|
|
|
# push a file to the app's cache (for media tests) / pull logs
|
|
adb shell run-as dev.iris.app ls files
|
|
adb logcat -d > /tmp/logcat.txt
|
|
```
|
|
|
|
**E2E scenarios (script where possible):**
|
|
|
|
1. **Pair:** connect screen → enter URL+token → `hello.ack` → main. (Verify auth
|
|
leg, not just TCP.)
|
|
2. **Text round-trip:** send "hello" → streamed reply appears (message.start →
|
|
updates → stop).
|
|
3. **Reasoning:** ask a reasoning-model question → ReasoningBlock shows above the
|
|
answer; copy button works.
|
|
4. **Tools:** trigger a tool (e.g. "list files") → ToolCard shows; toggle
|
|
verbosity in Settings → rendering changes.
|
|
5. **Intermediate:** a multi-step prompt → commentary bubble appears dimmed.
|
|
6. **Channels:** create "Cron Reports" → appears in list; set as cron target.
|
|
7. **Cron delivery:** create a cron job `deliver=iris:chan_<n>` → it
|
|
fires → lands in that channel (not default).
|
|
8. **Search:** "search everywhere" vs "this chat" → correct scoping; tap → jump.
|
|
9. **Media (in):** attach a photo + a video → agent receives (vision) → reply.
|
|
10. **Media (out):** ask the agent to send an image/video → `media.offer` →
|
|
inline player plays it live.
|
|
11. **Push:** background the app (`adb shell am start` another app / lock) →
|
|
trigger a message → FCM notification appears → tap → syncs + deep-links.
|
|
12. **Reconnect/sync:** kill the WS (stop gateway briefly) → restart → app
|
|
reconnects → `sync` catches up (no lost/dup messages).
|
|
13. **Auto-threading:** Threads on → send a message in the Default channel's
|
|
flat lane → a new topic appears (derived name), the app jumps into it, the
|
|
reply streams there, and the topic is renamed to the AI's title a moment
|
|
later. Slash commands / media-only sends stay in the flat lane.
|
|
14. **Slash drawer:** type `/` in the composer → the drawer rolls up over the
|
|
input with the gateway catalog; keep typing → fuzzy filtering (unrelated
|
|
entries drop out); tap a row → the command is sent and the drawer closes;
|
|
type an unknown command → the drawer closes (the raw text can still be
|
|
sent; hermes answers with its unknown-command reply).
|
|
15. **HTTP fallback (docs/19):** with the WS port unreachable (e.g. the
|
|
gateway bound WS to a dead port, or a firewall dropping 8790 but not
|
|
8791), the app stays sendable: the status pill shows "connected · http"
|
|
(green), a sent message echoes back within ~1 s and the agent reply
|
|
streams in over SSE; the attach button is disabled (media needs the
|
|
live WS). When the WS comes back the pill returns to "connected" and
|
|
media works again. Automated: `e2e.py` scenario 13 (health +
|
|
`POST /v1/frame` + SSE turn, user-echo < 1.5 s) and
|
|
`ws_probe.py --http` (same assertion flags as the WS leg).
|
|
|
|
## 13.5 Debugging tips
|
|
|
|
- **Gateway side:** `~/.hermes/logs/gateway.log` (and `hermes logs --follow`).
|
|
Our plugin logs under the `iris` adapter name; secrets redacted.
|
|
- **WS framing bugs:** use the `ws_probe.py` harness — it isolates the protocol
|
|
from the app.
|
|
- **Streaming jitter:** the consumer edits at intervals; if updates look chunky,
|
|
check `display.platforms.iris.streaming` and the consumer's edit interval.
|
|
- **Media pull stalls:** check chunk size + backpressure; confirm the file is
|
|
within hermes delivery roots (`validate_media_delivery_path`).
|
|
- **FCM not arriving:** confirm the token registered (`devices.db`), the service
|
|
account can mint a token, and the app's `onNewToken` re-registered after
|
|
`pm clear`.
|
|
- **Profile leaks:** if tokens look wrong under multiple profiles, verify the
|
|
scope-aware secret read (`_get_scoped_secret`) is used.
|