ARIA
59acf66c89
- gateway-plugin/: android platform plugin (plugin.yaml + adapter.py register(ctx) + no-op AndroidAdapter) + stub modules for M1-M5 - app/: Compose Multiplatform project (shared KMP + androidApp + desktopApp) with Gradle wrapper; builds :androidApp:assembleDebug and :desktopApp:compileKotlin - scripts/guard_hermes_agent.sh + pre-commit hook: fail if hermes-agent/ is staged (read-only reference, never committed) - .gitignore excludes hermes-agent/; docs/ reference library
131 lines
6.3 KiB
Markdown
131 lines
6.3 KiB
Markdown
# 13 — Testing & Debugging (incl. ADB on-device)
|
|
|
|
Three layers: **Python plugin tests**, **Kotlin unit/UI tests**, and **on-device
|
|
E2E via ADB**. Plus a **WS test-client harness** to drive the real gateway
|
|
without the app (critical for verifying frame shapes early).
|
|
|
|
## 13.1 Python plugin tests
|
|
|
|
- Location: `gateway-plugin/tests/` (and, for hermes-integration tests, mirror
|
|
into the hermes `tests/gateway/test_android.py` pattern when running under
|
|
hermes's suite).
|
|
- **Run with hermes's hermetic runner** (never bare `pytest`):
|
|
```bash
|
|
cd hermes-agent
|
|
scripts/run_tests.sh tests/gateway/test_android.py
|
|
scripts/run_tests.sh # full suite (CI parity)
|
|
```
|
|
- Coverage to write (behavioral, not change-detector — per hermes test policy):
|
|
- `register(ctx)` produces a valid `PlatformEntry` (name, cron env var,
|
|
parse_target_ref).
|
|
- `check_requirements` / `validate_config` / `is_connected` truth table.
|
|
- `_parse_target_ref`: `android:<chat>`, `android:<chat>:<thread>`, non-android
|
|
→ None.
|
|
- **Reasoning split:** given a `show_reasoning`-style final text, `send()`
|
|
emits `message {reasoning, text}` correctly; no-prefix → no reasoning field.
|
|
- **Outbox:** frame with no subscriber → written with a cursor; `sync` replays
|
|
the right range; retention prunes.
|
|
- **Pairing:** valid token → `hello.ack`; wrong token → `error auth` + close.
|
|
Constant-time compare used.
|
|
- **Media:** upload start→binary→end reassembles + sha256 verified; over-limit
|
|
→ `media_too_large`; pull serves only allowed paths.
|
|
- **Push backend selection:** fcm vs ntfy chosen by config; `configured()`
|
|
reflects missing creds.
|
|
- **Channel directory:** create/rename/set_default; cron target resolution.
|
|
- **No `~/.hermes` writes in tests** — use the `_isolate_hermes_home` fixture
|
|
pattern (temp `HERMES_HOME`). Profile tests also mock `Path.home()`.
|
|
|
|
## 13.2 WS test-client harness (do this FIRST, in M1/M2)
|
|
|
|
A small Python script (`gateway-plugin/tests/ws_probe.py`) that connects to the
|
|
**real running gateway** and drives a turn, printing every frame. This is how we
|
|
**empirically confirm** the exact frame shapes (especially tool-progress vs
|
|
commentary classification and the reasoning prefix) before/while building the
|
|
Kotlin client.
|
|
|
|
```bash
|
|
hermes gateway & # with the android plugin
|
|
python gateway-plugin/tests/ws_probe.py --token <ANDROID_TOKEN> \
|
|
--send "list the files and summarize"
|
|
# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end,
|
|
# commentary, message.stop {reasoning,…}, …
|
|
```
|
|
Use it to lock `04-wire-protocol.md` against reality and to debug the adapter
|
|
without waiting for the app.
|
|
|
|
## 13.3 Kotlin unit / UI tests
|
|
|
|
- **Protocol codec:** round-trip every frame type (serialize → deserialize →
|
|
equal); unknown `type`/fields ignored (forward-compat).
|
|
- **Repositories:** merge-don't-clobber on `channel.*`; optimistic send + rollback
|
|
on `error`; sync dedupe by `message_id`; cursor monotonic.
|
|
- **`GatewayClient`:** reconnect/backoff; `message.update` coalescing (latest
|
|
wins); request/response correlation by `id`.
|
|
- **ViewModels:** state transitions (connecting→connected→reconnecting); tool
|
|
verbosity filtering (everything/truncated/nothing); reasoning present/absent.
|
|
- **Compose UI tests:** ReasoningBlock collapse/expand + copy; ToolCard
|
|
spinner→done; composer auto-grow cap; channel list active highlight.
|
|
- Run: `./gradlew :shared:testDebugUnitTest` (and `:shared:testDesktopTest`).
|
|
|
|
## 13.4 On-device E2E via ADB (the MIX 2S, API 29)
|
|
|
|
Device: `a5ca2a4b` (Xiaomi MIX 2S). Workflow:
|
|
|
|
```bash
|
|
# install + launch
|
|
cd app
|
|
./gradlew :androidApp:installDebug
|
|
adb shell am start -n dev.iris.app/.MainActivity
|
|
|
|
# logs (filter our tags + WS + FCM + ExoPlayer)
|
|
adb logcat -c
|
|
adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec"
|
|
|
|
# screenshots for visual checks
|
|
adb exec-out screencap -p > /tmp/shot.png
|
|
|
|
# clear app data between pairing attempts
|
|
adb shell pm clear dev.iris.app
|
|
|
|
# push a file to the app's cache (for media tests) / pull logs
|
|
adb shell run-as dev.iris.app ls files
|
|
adb logcat -d > /tmp/logcat.txt
|
|
```
|
|
|
|
**E2E scenarios (script where possible):**
|
|
1. **Pair:** connect screen → enter URL+token → `hello.ack` → main. (Verify auth
|
|
leg, not just TCP.)
|
|
2. **Text round-trip:** send "hello" → streamed reply appears (message.start →
|
|
updates → stop).
|
|
3. **Reasoning:** ask a reasoning-model question → ReasoningBlock shows above the
|
|
answer; copy button works.
|
|
4. **Tools:** trigger a tool (e.g. "list files") → ToolCard shows; toggle
|
|
verbosity in Settings → rendering changes.
|
|
5. **Intermediate:** a multi-step prompt → commentary bubble appears dimmed.
|
|
6. **Channels:** create "Cron Reports" → appears in list; set as cron target.
|
|
7. **Cron delivery:** create a cron job `deliver=android:android:chan_<n>` → it
|
|
fires → lands in that channel (not default).
|
|
8. **Search:** "search everywhere" vs "this chat" → correct scoping; tap → jump.
|
|
9. **Media (in):** attach a photo + a video → agent receives (vision) → reply.
|
|
10. **Media (out):** ask the agent to send an image/video → `media.offer` →
|
|
inline player plays it live.
|
|
11. **Push:** background the app (`adb shell am start` another app / lock) →
|
|
trigger a message → FCM notification appears → tap → syncs + deep-links.
|
|
12. **Reconnect/sync:** kill the WS (stop gateway briefly) → restart → app
|
|
reconnects → `sync` catches up (no lost/dup messages).
|
|
|
|
## 13.5 Debugging tips
|
|
|
|
- **Gateway side:** `~/.hermes/logs/gateway.log` (and `hermes logs --follow`).
|
|
Our plugin logs under the `android` adapter name; secrets redacted.
|
|
- **WS framing bugs:** use the `ws_probe.py` harness — it isolates the protocol
|
|
from the app.
|
|
- **Streaming jitter:** the consumer edits at intervals; if updates look chunky,
|
|
check `display.platforms.android.streaming` and the consumer's edit interval.
|
|
- **Media pull stalls:** check chunk size + backpressure; confirm the file is
|
|
within hermes delivery roots (`validate_media_delivery_path`).
|
|
- **FCM not arriving:** confirm the token registered (`devices.db`), the service
|
|
account can mint a token, and the app's `onNewToken` re-registered after
|
|
`pm clear`.
|
|
- **Profile leaks:** if tokens look wrong under multiple profiles, verify the
|
|
scope-aware secret read (`_get_scoped_secret`) is used. |