commit 59acf66c898c09cdeef5b0426f473507fe5c91b5 Author: ARIA Date: Wed Aug 19 11:27:02 2026 +0200 M0: toolchain, monorepo scaffold, gateway plugin skeleton, CMP app - gateway-plugin/: android platform plugin (plugin.yaml + adapter.py register(ctx) + no-op AndroidAdapter) + stub modules for M1-M5 - app/: Compose Multiplatform project (shared KMP + androidApp + desktopApp) with Gradle wrapper; builds :androidApp:assembleDebug and :desktopApp:compileKotlin - scripts/guard_hermes_agent.sh + pre-commit hook: fail if hermes-agent/ is staged (read-only reference, never committed) - .gitignore excludes hermes-agent/; docs/ reference library diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d179f3c --- /dev/null +++ b/.gitignore @@ -0,0 +1,34 @@ +# ── HARD RULE: hermes-agent is a read-only research reference ───────────── +# It must NEVER be committed, pushed, or shipped. +/hermes-agent/ + +# ── Python (gateway-plugin) ─────────────────────────────────────────────── +__pycache__/ +*.py[cod] +*.egg-info/ +.venv/ +venv/ +.pytest_cache/ +.ruff_cache/ + +# ── Kotlin / Gradle (app) ───────────────────────────────────────────────── +.gradle/ +build/ +local.properties +*.iml +.idea/ +captures/ +.cxx/ + +# ── Android / secrets ───────────────────────────────────────────────────── +app/androidApp/src/main/res/values/secrets.xml +google-services.json +*.jks +keystore.jks +*.keystore + +# ── OS / misc ───────────────────────────────────────────────────────────── +.DS_Store +Thumbs.db +*.log +*.tmp \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..ee1c553 --- /dev/null +++ b/README.md @@ -0,0 +1,42 @@ +# Iris × Hermes + +A **native Android + Desktop** experience for [hermes-agent](https://github.com/NousResearch/hermes-agent), connected through a **gateway platform plugin**. + +> ## ⚠️ HARD RULES +> 1. **`hermes-agent/` is a read-only research reference. It is git-ignored and +> must NEVER be committed, pushed, or shipped.** We only *install* our plugin +> into a live hermes home (`~/.hermes/plugins/android`); we never modify hermes +> core. +> 2. **ADB is installed and a device is connected** (`a5ca2a4b`, Xiaomi MIX 2S, +> Android 10 / API 29). Use it to install/launch/debug the app on-device. + +## Start here + +📚 **The full implementation reference library is in [`docs/`](docs/README.md).** +Read `docs/00-overview.md` first, then follow the numbered docs. + +## The three deliverables (this monorepo) + +| Path | What | +|---|---| +| `gateway-plugin/` | Python hermes **platform plugin** (`android`). Runs in the `hermes gateway` process; opens a WebSocket server the apps connect to. Zero new Python deps, zero hermes-core changes. | +| `app/androidApp` | Native **Kotlin + Jetpack Compose** client. | +| `app/desktopApp` | **Kotlin + Compose Multiplatform** client — the Android app, tweaked for a big screen (shares `app/shared`). | + +The Android and Desktop clients live in **one Compose Multiplatform Gradle +project** (`app/`) with a shared KMP module (`app/shared`). + +## Status + +- **Phase:** Planning complete → ready to implement (Milestone M0). +- **Milestones:** see [`docs/14-milestones.md`](docs/14-milestones.md). +- **Locked decisions:** see [`docs/16-open-questions.md`](docs/16-open-questions.md). + +## Quick orientation + +- Architecture + rationale → [`docs/01-architecture.md`](docs/01-architecture.md) +- Wire protocol → [`docs/04-wire-protocol.md`](docs/04-wire-protocol.md) (+ [`docs/protocol/frames.schema.json`](docs/protocol/frames.schema.json)) +- Gateway plugin design → [`docs/03-gateway-plugin.md`](docs/03-gateway-plugin.md) +- hermes source cheat-sheet → [`docs/15-hermes-reference.md`](docs/15-hermes-reference.md) +- Toolchain setup → [`docs/12-toolchain.md`](docs/12-toolchain.md) + \ No newline at end of file diff --git a/Screenshot_20260819_083446~2.jpg b/Screenshot_20260819_083446~2.jpg new file mode 100644 index 0000000..ad3bf73 Binary files /dev/null and b/Screenshot_20260819_083446~2.jpg differ diff --git a/app/androidApp/build.gradle.kts b/app/androidApp/build.gradle.kts new file mode 100644 index 0000000..89cc72b --- /dev/null +++ b/app/androidApp/build.gradle.kts @@ -0,0 +1,42 @@ +plugins { + id("com.android.application") + kotlin("android") + id("org.jetbrains.kotlin.plugin.compose") +} + +android { + namespace = "dev.iris.app" + compileSdk = 34 + + defaultConfig { + applicationId = "dev.iris.app" + minSdk = 29 + targetSdk = 34 + versionCode = 1 + versionName = "0.1.0" + } + + buildTypes { + release { + isMinifyEnabled = false + } + } + + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } + + kotlinOptions { + jvmTarget = "17" + } +} + +dependencies { + implementation(project(":shared")) + implementation(platform("androidx.compose:compose-bom:2024.09.02")) + implementation("androidx.compose.material3:material3") + implementation("androidx.compose.ui:ui") + implementation("androidx.activity:activity-compose:1.9.2") + implementation("androidx.core:core-splashscreen:1.0.1") +} \ No newline at end of file diff --git a/app/androidApp/src/main/AndroidManifest.xml b/app/androidApp/src/main/AndroidManifest.xml new file mode 100644 index 0000000..fda8506 --- /dev/null +++ b/app/androidApp/src/main/AndroidManifest.xml @@ -0,0 +1,22 @@ + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/app/androidApp/src/main/kotlin/dev/iris/app/MainActivity.kt b/app/androidApp/src/main/kotlin/dev/iris/app/MainActivity.kt new file mode 100644 index 0000000..071c5ea --- /dev/null +++ b/app/androidApp/src/main/kotlin/dev/iris/app/MainActivity.kt @@ -0,0 +1,15 @@ +package dev.iris.app + +import android.os.Bundle +import androidx.activity.ComponentActivity +import androidx.activity.compose.setContent +import iris.IrisApp + +class MainActivity : ComponentActivity() { + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + setContent { + IrisApp() + } + } +} \ No newline at end of file diff --git a/app/build.gradle.kts b/app/build.gradle.kts new file mode 100644 index 0000000..3cccc37 --- /dev/null +++ b/app/build.gradle.kts @@ -0,0 +1,13 @@ +plugins { + kotlin("multiplatform") version "2.1.0" apply false + kotlin("android") version "2.1.0" apply false + kotlin("jvm") version "2.1.0" apply false + id("com.android.application") version "8.7.3" apply false + id("com.android.library") version "8.7.3" apply false + id("org.jetbrains.compose") version "1.7.3" apply false + id("org.jetbrains.kotlin.plugin.compose") version "2.1.0" apply false +} + +tasks.register("clean") { + delete(rootProject.layout.buildDirectory) +} \ No newline at end of file diff --git a/app/desktopApp/build.gradle.kts b/app/desktopApp/build.gradle.kts new file mode 100644 index 0000000..c2af2c2 --- /dev/null +++ b/app/desktopApp/build.gradle.kts @@ -0,0 +1,27 @@ +import org.gradle.internal.os.OperatingSystem + +plugins { + kotlin("jvm") + id("org.jetbrains.compose") + id("org.jetbrains.kotlin.plugin.compose") +} + +val composeVersion = "1.7.3" +val os = OperatingSystem.current() +val arch = System.getProperty("os.arch") ?: "amd64" +val desktopTarget = when { + os.isMacOsX -> if (arch == "aarch64") "macos-arm64" else "macos-x64" + os.isWindows -> "windows-x64" + else -> if (arch == "aarch64") "linux-arm64" else "linux-x64" +} + +dependencies { + implementation(project(":shared")) + implementation("org.jetbrains.compose.desktop:desktop-jvm-$desktopTarget:$composeVersion") +} + +compose.desktop { + application { + mainClass = "iris.desktop.MainKt" + } +} \ No newline at end of file diff --git a/app/desktopApp/src/main/kotlin/iris/desktop/Main.kt b/app/desktopApp/src/main/kotlin/iris/desktop/Main.kt new file mode 100644 index 0000000..c67e057 --- /dev/null +++ b/app/desktopApp/src/main/kotlin/iris/desktop/Main.kt @@ -0,0 +1,11 @@ +package iris.desktop + +import androidx.compose.ui.window.Window +import androidx.compose.ui.window.application +import iris.IrisApp + +fun main() = application { + Window(onCloseRequest = ::exitApplication, title = "Iris") { + IrisApp() + } +} \ No newline at end of file diff --git a/app/gradle.properties b/app/gradle.properties new file mode 100644 index 0000000..52115af --- /dev/null +++ b/app/gradle.properties @@ -0,0 +1,11 @@ +org.gradle.jvmargs=-Xmx4g -Dfile.encoding=UTF-8 +org.gradle.caching=true +org.gradle.configuration-cache=true + +android.useAndroidX=true +android.nonTransitiveRClass=true + +kotlin.code.style=official + +# Compose Multiplatform +org.jetbrains.compose.experimental.jscanvas.enabled=false \ No newline at end of file diff --git a/app/gradle/wrapper/gradle-wrapper.jar b/app/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000..a4b76b9 Binary files /dev/null and b/app/gradle/wrapper/gradle-wrapper.jar differ diff --git a/app/gradle/wrapper/gradle-wrapper.properties b/app/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..df97d72 --- /dev/null +++ b/app/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,7 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip +networkTimeout=10000 +validateDistributionUrl=true +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists diff --git a/app/gradlew b/app/gradlew new file mode 100755 index 0000000..f5feea6 --- /dev/null +++ b/app/gradlew @@ -0,0 +1,252 @@ +#!/bin/sh + +# +# Copyright © 2015-2021 the original authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# + +############################################################################## +# +# Gradle start up script for POSIX generated by Gradle. +# +# Important for running: +# +# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is +# noncompliant, but you have some other compliant shell such as ksh or +# bash, then to run this script, type that shell name before the whole +# command line, like: +# +# ksh Gradle +# +# Busybox and similar reduced shells will NOT work, because this script +# requires all of these POSIX shell features: +# * functions; +# * expansions «$var», «${var}», «${var:-default}», «${var+SET}», +# «${var#prefix}», «${var%suffix}», and «$( cmd )»; +# * compound commands having a testable exit status, especially «case»; +# * various built-in commands including «command», «set», and «ulimit». +# +# Important for patching: +# +# (2) This script targets any POSIX shell, so it avoids extensions provided +# by Bash, Ksh, etc; in particular arrays are avoided. +# +# The "traditional" practice of packing multiple parameters into a +# space-separated string is a well documented source of bugs and security +# problems, so this is (mostly) avoided, by progressively accumulating +# options in "$@", and eventually passing that to Java. +# +# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS, +# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly; +# see the in-line comments for details. +# +# There are tweaks for specific operating systems such as AIX, CygWin, +# Darwin, MinGW, and NonStop. +# +# (3) This script is generated from the Groovy template +# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt +# within the Gradle project. +# +# You can find Gradle at https://github.com/gradle/gradle/. +# +############################################################################## + +# Attempt to set APP_HOME + +# Resolve links: $0 may be a link +app_path=$0 + +# Need this for daisy-chained symlinks. +while + APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path + [ -h "$app_path" ] +do + ls=$( ls -ld "$app_path" ) + link=${ls#*' -> '} + case $link in #( + /*) app_path=$link ;; #( + *) app_path=$APP_HOME$link ;; + esac +done + +# This is normally unused +# shellcheck disable=SC2034 +APP_BASE_NAME=${0##*/} +# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036) +APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s +' "$PWD" ) || exit + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD=maximum + +warn () { + echo "$*" +} >&2 + +die () { + echo + echo "$*" + echo + exit 1 +} >&2 + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "$( uname )" in #( + CYGWIN* ) cygwin=true ;; #( + Darwin* ) darwin=true ;; #( + MSYS* | MINGW* ) msys=true ;; #( + NONSTOP* ) nonstop=true ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD=$JAVA_HOME/jre/sh/java + else + JAVACMD=$JAVA_HOME/bin/java + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD=java + if ! command -v java >/dev/null 2>&1 + then + die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +fi + +# Increase the maximum file descriptors if we can. +if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then + case $MAX_FD in #( + max*) + # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC2039,SC3045 + MAX_FD=$( ulimit -H -n ) || + warn "Could not query maximum file descriptor limit" + esac + case $MAX_FD in #( + '' | soft) :;; #( + *) + # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC2039,SC3045 + ulimit -n "$MAX_FD" || + warn "Could not set maximum file descriptor limit to $MAX_FD" + esac +fi + +# Collect all arguments for the java command, stacking in reverse order: +# * args from the command line +# * the main class name +# * -classpath +# * -D...appname settings +# * --module-path (only if needed) +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables. + +# For Cygwin or MSYS, switch paths to Windows format before running java +if "$cygwin" || "$msys" ; then + APP_HOME=$( cygpath --path --mixed "$APP_HOME" ) + CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" ) + + JAVACMD=$( cygpath --unix "$JAVACMD" ) + + # Now convert the arguments - kludge to limit ourselves to /bin/sh + for arg do + if + case $arg in #( + -*) false ;; # don't mess with options #( + /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath + [ -e "$t" ] ;; #( + *) false ;; + esac + then + arg=$( cygpath --path --ignore --mixed "$arg" ) + fi + # Roll the args list around exactly as many times as the number of + # args, so each arg winds up back in the position where it started, but + # possibly modified. + # + # NB: a `for` loop captures its iteration list before it begins, so + # changing the positional parameters here affects neither the number of + # iterations, nor the values presented in `arg`. + shift # remove old arg + set -- "$@" "$arg" # push replacement arg + done +fi + + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Collect all arguments for the java command: +# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments, +# and any embedded shellness will be escaped. +# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be +# treated as '${Hostname}' itself on the command line. + +set -- \ + "-Dorg.gradle.appname=$APP_BASE_NAME" \ + -classpath "$CLASSPATH" \ + org.gradle.wrapper.GradleWrapperMain \ + "$@" + +# Stop when "xargs" is not available. +if ! command -v xargs >/dev/null 2>&1 +then + die "xargs is not available" +fi + +# Use "xargs" to parse quoted args. +# +# With -n1 it outputs one arg per line, with the quotes and backslashes removed. +# +# In Bash we could simply go: +# +# readarray ARGS < <( xargs -n1 <<<"$var" ) && +# set -- "${ARGS[@]}" "$@" +# +# but POSIX shell has neither arrays nor command substitution, so instead we +# post-process each arg (as a line of input to sed) to backslash-escape any +# character that might be a shell metacharacter, then use eval to reverse +# that process (while maintaining the separation between arguments), and wrap +# the whole thing up as a single "set" statement. +# +# This will of course break if any of these variables contains a newline or +# an unmatched quote. +# + +eval "set -- $( + printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" | + xargs -n1 | + sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' | + tr '\n' ' ' + )" '"$@"' + +exec "$JAVACMD" "$@" diff --git a/app/gradlew.bat b/app/gradlew.bat new file mode 100644 index 0000000..9b42019 --- /dev/null +++ b/app/gradlew.bat @@ -0,0 +1,94 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem +@rem SPDX-License-Identifier: Apache-2.0 +@rem + +@if "%DEBUG%"=="" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%"=="" set DIRNAME=. +@rem This is normally unused +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if %ERRORLEVEL% equ 0 goto execute + +echo. 1>&2 +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto execute + +echo. 1>&2 +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 + +goto fail + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %* + +:end +@rem End local scope for the variables with windows NT shell +if %ERRORLEVEL% equ 0 goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +set EXIT_CODE=%ERRORLEVEL% +if %EXIT_CODE% equ 0 set EXIT_CODE=1 +if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE% +exit /b %EXIT_CODE% + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/app/settings.gradle.kts b/app/settings.gradle.kts new file mode 100644 index 0000000..ed24893 --- /dev/null +++ b/app/settings.gradle.kts @@ -0,0 +1,20 @@ +pluginManagement { + repositories { + mavenCentral() + gradlePluginPortal() + google() + } +} + +dependencyResolutionManagement { + repositories { + mavenCentral() + google() + } +} + +rootProject.name = "iris" + +include(":shared") +include(":androidApp") +include(":desktopApp") \ No newline at end of file diff --git a/app/shared/build.gradle.kts b/app/shared/build.gradle.kts new file mode 100644 index 0000000..37cc195 --- /dev/null +++ b/app/shared/build.gradle.kts @@ -0,0 +1,34 @@ +plugins { + kotlin("multiplatform") + id("com.android.library") + id("org.jetbrains.compose") + id("org.jetbrains.kotlin.plugin.compose") +} + +val composeVersion = "1.7.3" + +kotlin { + androidTarget() + jvm("desktop") + + sourceSets { + commonMain.dependencies { + implementation("org.jetbrains.compose.runtime:runtime:$composeVersion") + implementation("org.jetbrains.compose.foundation:foundation:$composeVersion") + implementation("org.jetbrains.compose.material3:material3:$composeVersion") + implementation("org.jetbrains.compose.ui:ui:$composeVersion") + } + } +} + +android { + namespace = "iris.shared" + compileSdk = 34 + defaultConfig { + minSdk = 29 + } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} \ No newline at end of file diff --git a/app/shared/src/commonMain/kotlin/iris/IrisApp.kt b/app/shared/src/commonMain/kotlin/iris/IrisApp.kt new file mode 100644 index 0000000..8a6ef87 --- /dev/null +++ b/app/shared/src/commonMain/kotlin/iris/IrisApp.kt @@ -0,0 +1,53 @@ +package iris + +import androidx.compose.foundation.background +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.shape.CircleShape +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp + +/** + * Root composable shared by the Android and Desktop shells. + * + * M0: a visible placeholder to confirm the Compose stack renders. M1+ + * replaces this with the Connect screen, chat UI, and the rest of the app + * (see docs/10-android-app.md). + */ +@Composable +fun IrisApp() { + MaterialTheme { + Surface(modifier = Modifier.fillMaxSize()) { + Box( + modifier = Modifier + .fillMaxSize() + .background(Color(0xFF1B1E28)), + contentAlignment = Alignment.Center, + ) { + Column(horizontalAlignment = Alignment.CenterHorizontally) { + Box( + modifier = Modifier + .size(72.dp) + .background(Color(0xFF4F7CFF), CircleShape), + contentAlignment = Alignment.Center, + ) { + Text(text = "I", color = Color.White, fontSize = 40.sp) + } + Spacer(modifier = Modifier.height(16.dp)) + Text(text = "Iris × Hermes", color = Color.White, fontSize = 22.sp) + } + } + } + } +} \ No newline at end of file diff --git a/docs/00-overview.md b/docs/00-overview.md new file mode 100644 index 0000000..1fe862e --- /dev/null +++ b/docs/00-overview.md @@ -0,0 +1,93 @@ +# 00 — Overview + +## Vision + +A **native, Telegram-quality chat experience** for a personal hermes-agent: +install an app on your phone (and a desktop app on your PC), pair it to your +running `hermes gateway`, and talk to your agent with streaming replies, +visible reasoning, structured tool activity, channels/threads, media, search, +and push notifications — with cron jobs able to post into any channel you +create. + +## Goals + +- **Native feel.** Real Android app (Kotlin/Compose), not a WebView. Desktop + app that is the same app, resized for a big screen. +- **First-class gateway citizen.** The app is a hermes *messaging platform*, so + everything the gateway already does "just works": slash commands, cron + delivery, `send_message` routing, coexistence with Telegram/Discord/etc. +- **Full agent transparency.** Streaming text, reasoning shown *before* the + answer, structured tool events (the app chooses how much to show), and + intermediate assistant beats. +- **Organized by default.** A default chat with optional threads, plus + user-created channels that cron jobs can target. +- **Reachable anywhere.** Live over a WebSocket; background push via FCM + (primary) or ntfy (fallback). + +## In scope (v1) + +Everything in the feature checklist below. + +## Out of scope / stretch (v1) + +- **Standalone-cron delivery while the gateway process is fully down** — + best-effort FCM/ntfy only (the outbox is served by the running gateway). +- **Multi-user / group chat** — this is a *personal* 1-user agent. +- **End-to-end encryption** — transport security (WSS) only. +- **iOS** — Android + Desktop only (the protocol is transport-agnostic, so an + iOS client is a future port, not a v1 goal). + +## Feature checklist → where it's handled + +| Requirement | Gateway plugin | App | +|---|---|---| +| Input box, auto-grow (max height) | — | Compose `TextField` + bounded `heightIn` | +| Menu button → all slash commands | Dispatches `/…`; serves command catalog | Bottom-sheet menu + `/` autocomplete | +| Tool output (app decides how much) | Emits **structured** tool events | App setting: everything / truncated / nothing | +| Reasoning shown before message | Captures + splits reasoning | Collapsible "Reasoning" block above message | +| Intermediate messages | Forwards `Commentary` events | Distinct dimmed bubble | +| Threading + channels; default chat; user channels for cron | `chat_id`/`thread_id` model; cron `deliver=android:[:]` | Channel list, thread toggle, "new channel" | +| Search ("everywhere" / "this chat/channel") | FTS5 session search bridge | Search UI + scope toggle | +| Attach media (music/video/images/docs) | Inbound cache; outbound `send_*` | Pickers + chunked upload + preview | +| Push notifications | FCM (primary) / ntfy (fallback) | FCM token / ntfy topic + notification service | +| Live playback of AI-sent music/video | Serves media bytes over WS | ExoPlayer inline player | + +## Locked decisions (from planning) + +| Decision | Choice | +|---|---| +| Desktop app tech | **Compose Multiplatform** (shares Android code; "tweaked" for big screen) | +| Push backend | **Both** — FCM primary, ntfy fallback (`ANDROID_PUSH_BACKEND`) | +| Media transport | **Over the WebSocket** (chunked binary frames; no extra Python deps) | +| Phone default layout | **User-toggleable, single-pane default** (auto two-pane on large screens) | + +## Disclaimers (hard rules) + +1. **`hermes-agent/` is a read-only research reference.** It lives next to this + folder for study only. It is **git-ignored** and must **never** be committed, + pushed, or included in any artifact. Our plugin is *installed* into a live + hermes home (`~/.hermes/plugins/android`); we never edit hermes core files. +2. **ADB is available and a device is connected** (`a5ca2a4b`, Xiaomi MIX 2S, + Android 10 / API 29). Use `adb install` / `adb logcat` / `adb shell am start` + to install, launch, and debug the app on-device throughout the build. + +## Verified environment state (2026-08-19) + +| Item | State | +|---|---| +| OS | CachyOS (Arch-based), `pacman` present | +| JDK | **Not installed** → Milestone M0 (`pacman -S jdk17-openjdk`) | +| Android SDK | **Not installed** → M0 (cmdline-tools + sdkmanager) | +| Gradle | Via project wrapper (`gradlew`), no system install | +| ADB | Installed; device `a5ca2a4b` (MIX 2S, API 29) connected | +| Python | 3.14.7; `uv` 0.12.3 present | +| hermes venv | **Not created** → M0 (`cd hermes-agent && uv sync`) | +| hermes core deps | `websockets==15.0.1` and `httpx` are **core** deps → plugin needs **zero new Python deps** | +| Disk / RAM | 522 GB free / 62 GB RAM — ample | + +## Naming + +- Product/effort name: **Iris × Hermes** (folder `iris_x_hermes`). +- hermes platform name: **`android`** (the plugin registers `Platform("android")`). +- WS default port: **8790** (configurable). +- Default chat id: **`android:default`** (the home channel). \ No newline at end of file diff --git a/docs/01-architecture.md b/docs/01-architecture.md new file mode 100644 index 0000000..49d32a3 --- /dev/null +++ b/docs/01-architecture.md @@ -0,0 +1,106 @@ +# 01 — Architecture + +## System diagram + +``` +┌────────────────────────────── User's machine (home server / PC) ─────────────────────────────┐ +│ │ +│ hermes gateway (ONE process) │ +│ ┌──────────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Agent core (run_agent.py) ── sessions (SQLite + FTS5) ── cron scheduler │ │ +│ │ │ │ │ +│ │ ▼ legacy stream callbacks (delta / tool-progress / commentary) │ │ +│ │ ┌──────────────────────────────┐ ┌────────────────────────────────┐ │ │ +│ │ │ android PLATFORM PLUGIN │◄───────►│ WebSocket SERVER (websockets) │ │ │ +│ │ │ AndroidAdapter │ JSON │ ws://host:8790/ws │ │ │ +│ │ │ (BasePlatformAdapter) │ frames │ + media + FCM-token + pairing │ │ │ +│ │ │ • send / edit / stream │ └───────────────┬────────────────┘ │ │ +│ │ │ • media cache │ │ WSS │ │ +│ │ │ • outbox (SQLite) │ │ │ │ +│ │ │ • push (FCM / ntfy) │─────────────────────────┼──────────┐ │ │ +│ │ └──────────────────────────────┘ │ │ │ │ +│ └───────────────────────────────────────────────────────────┼──────────┼───────────┘ │ +└───────────────────────────────────────────────────────────────┼──────────┼───────────────────────┘ + │ │ push (FCM HTTP v1 / ntfy) + ┌──────────────────┼──────────▼─────────┐ + │ │ Google FCM cloud │ + ▼ │ │ │ + ┌────────────────────────┐ │ ▼ │ + │ ANDROID APP │◄──┴── (wake) ┌──────────┐ + │ (Kotlin / Compose) │ WSS │ PHONE │ + │ • WS client (OkHttp) │◄───────────►│ MIX 2S │ + │ • ExoPlayer │ │ (API 29) │ + │ • FCM service │ └──────────┘ + └────────────────────────┘ + DESKTOP APP (Compose Multiplatform) + • same shared code, WSS to same server + • tray + OS notifications (no FCM), big-screen two-pane +``` + +## Process model + +- **One `hermes gateway` process** hosts the agent core, the session store, the + cron scheduler, *and* our `android` platform plugin. The plugin's WebSocket + server runs on the gateway's asyncio loop (started in `AndroidAdapter.connect()`). +- **The app is a client.** It *initiates* the WS connection to the gateway + (outbound), so no inbound port is needed on the phone. For LAN/remote access + the user points the app at the gateway's LAN IP / Tailscale name / a WSS + tunnel (see `09-pairing-security.md`). +- **Push is the only inbound path to a sleeping phone**, and it goes through a + cloud relay (FCM or ntfy), not a direct connection. + +## Why the *messaging gateway* is the connection point (not `tui_gateway`) + +hermes has two "gateways": the **messaging gateway** (`hermes gateway`, which +serves Telegram/Discord/… and cron) and the **`tui_gateway`** JSON-RPC backend +(used by the TUI and the existing Electron desktop app). We deliberately use the +**messaging gateway** because: + +1. **Cron delivery is native.** Cron jobs resolve `deliver=android:[:]` + through the platform registry and call our adapter's `send()`. No bridging. +2. **`send_message` tool routing** works out of the box (plugin + `parse_target_ref_fn`). +3. **Slash commands** are dispatched by the gateway's command pipeline — the app + just sends `/cmd args` as a message. +4. **Coexistence.** The same agent is reachable via Telegram *and* the app at + once; sessions/channels are shared. + +The `tui_gateway` WS protocol is *not* reused; we define a clean, purpose-built +protocol (`04-wire-protocol.md`) that borrows familiar names (`message.*`, +`tool.*`, `reasoning`) but is owned by our plugin. + +## Key architectural decisions + rationale + +| Decision | Rationale | +|---|---| +| **Community-style platform plugin** (`register(ctx)` → `ctx.register_platform`) | Zero hermes-core changes; survives hermes updates; follows `ADDING_A_PLATFORM.md` "Plugin Path". | +| **Single WS transport** for chat, streaming, tools, media, pairing, FCM-token | One connection, one auth, one dependency (`websockets`, already core). Media as chunked binary frames avoids adding an HTTP server. | +| **`websockets` + `httpx` only** | Both are hermes *core* deps → the plugin adds **zero** new Python dependencies (respects hermes supply-chain pinning policy). | +| **Structured tool events, app-side verbosity** | Per the requirement: the gateway sends full tool data; the *app* decides everything/truncated/nothing. | +| **Reasoning split in the adapter** | The gateway prepends reasoning to the final text when `show_reasoning` is on; the adapter splits the stable prefix into a `reasoning` field so the app renders a clean collapsible block. | +| **Channels/threads = `chat_id`/`thread_id`** | The gateway's `SessionSource` already models this; cron delivery already targets `platform:chat_id:thread_id`. We map app concepts onto existing primitives. | +| **SQLite outbox + sync cursor** | Offline delivery + reconnect catch-up without re-reading full history. | +| **Compose Multiplatform** | Desktop is "the Android app, tweaked" → share protocol/state/UI; only platform services + layout differ. | + +## Data flow (one turn) + +1. App sends `message.send {text}` (or `/cmd`). +2. Plugin builds a `MessageEvent` (+ `media_urls` if attachments) → + `AndroidAdapter.handle_message(event)`. +3. Gateway resolves the session (`chat_id`/`thread_id`), runs the agent. +4. Agent streams: `stream_delta_callback` → `GatewayStreamConsumer` → + `adapter.send()` (first) / `adapter.edit_message()` (updates) → + `message.start` / `message.update` frames. +5. Tool calls: `tool_progress_callback` → progress queue → `adapter.send()` → + `tool.*` frames (structured). +6. Intermediate beats: `interim_assistant_callback` → consumer → `commentary` frames. +7. Final answer: consumer finalizes → `adapter.send()` → `message` frame + (reasoning split into its own field). +8. If the app is disconnected at any point: frame is dropped to the **outbox** + and a **push** is fired; on reconnect the app `sync`s the delta. + +> Implementation note: the main gateway uses the **legacy callback path** +> (not the ACP-only event-native `render_message_event` path). We therefore map +> the legacy `send`/`edit_message`/progress calls to our frames. The exact +> tool-progress vs commentary classification is verified empirically in M2 by +> running the real gateway with a test WS client (see `13-testing.md`). \ No newline at end of file diff --git a/docs/02-monorepo.md b/docs/02-monorepo.md new file mode 100644 index 0000000..252e525 --- /dev/null +++ b/docs/02-monorepo.md @@ -0,0 +1,130 @@ +# 02 — Monorepo Structure + +One repository, three artifacts. `hermes-agent/` is **not** part of the repo +(git-ignored reference). + +## Top-level tree + +``` +iris_x_hermes/ +├── .gitignore # MUST exclude hermes-agent/ (see below) +├── README.md # Repo root readme (short; points to docs/) +├── docs/ # ← THIS reference library +│ +├── hermes-agent/ # ⚠️ READ-ONLY REFERENCE — NEVER PUSHED (git-ignored) +│ +├── gateway-plugin/ # ① Python plugin → installed to ~/.hermes/plugins/android +│ ├── plugin.yaml # manifest (kind: platform, env vars, home channel) +│ ├── __init__.py +│ ├── adapter.py # AndroidAdapter(BasePlatformAdapter) + register(ctx) +│ ├── ws_server.py # websockets server, connection registry, framing +│ ├── protocol.py # frame schemas (source of truth, mirrored in Kotlin) +│ ├── media.py # inbound cache + outbound chunked streaming +│ ├── outbox.py # SQLite offline outbox + sync cursor +│ ├── push.py # PushBackend: FcmBackend + NtfyBackend +│ ├── pairing.py # token gen/verify, device registry +│ ├── search.py # FTS5 session search bridge +│ └── tests/ # pytest (run via hermes scripts/run_tests.sh) +│ +├── app/ # ② + ③ Compose Multiplatform project (Kotlin) +│ ├── settings.gradle.kts +│ ├── build.gradle.kts +│ ├── gradle.properties +│ ├── gradle/ gradlew gradlew.bat +│ ├── shared/ # KMP module — the bulk of the code +│ │ ├── build.gradle.kts +│ │ └── src/ +│ │ ├── commonMain/kotlin/iris/… # protocol, WS client, repo, state, Compose UI +│ │ ├── androidMain/kotlin/… # FCM, ExoPlayer, SAF picker, notifications +│ │ └── desktopMain/kotlin/… # tray, file dialog, player, window +│ ├── androidApp/ # thin Android shell (Application, MainActivity) +│ │ ├── build.gradle.kts +│ │ └── src/main/… # AndroidManifest, res, Firebase options +│ └── desktopApp/ # thin Desktop shell (main(), window) +│ ├── build.gradle.kts +│ └── src/main/kotlin/… +│ +└── (no other top-level code) +``` + +## Module responsibilities + +### `gateway-plugin/` (Python) +- **`plugin.yaml`** — manifest: `name: android-platform`, `kind: platform`, + `requires_env` / `optional_env` (surfaced in `hermes config`/setup). +- **`adapter.py`** — `AndroidAdapter(BasePlatformAdapter)` + `register(ctx)`. + The heart of the plugin. See `03-gateway-plugin.md`. +- **`ws_server.py`** — `websockets` server, per-device connection registry, + frame encode/decode, heartbeat, broadcast routing to all connected devices. +- **`protocol.py`** — dataclasses/constants for every frame (single source of + truth; `docs/protocol/frames.schema.json` is generated/mirrored from it). +- **`media.py`** — inbound chunked upload → `cache_*_from_bytes`; outbound + `media.offer`/`media.pull` chunked streaming. +- **`outbox.py`** — SQLite outbox per `chat_id` + monotonic sync cursor. +- **`push.py`** — `PushBackend` interface; `FcmBackend` (httpx, FCM HTTP v1) and + `NtfyBackend` (reuses hermes ntfy publish). Selected by `ANDROID_PUSH_BACKEND`. +- **`pairing.py`** — token generation/verification (constant-time), device + registry (SQLite), QR payload. +- **`search.py`** — FTS5 query bridge over the hermes session store. + +### `app/shared` (Kotlin KMP) +- **`commonMain`** — protocol models (kotlinx-serialization), `GatewayClient` + (OkHttp WS), repositories (Room), ViewModels (StateFlow), and the Compose UI + (design system, screens). ~80% of app code. +- **`androidMain`** — FCM service, ExoPlayer, SAF media picker, system + notifications, `MediaPlayer` actual. +- **`desktopMain`** — tray + OS notifications, desktop player, file dialog, + window management, `MediaPlayer` actual. + +### `app/androidApp` / `app/desktopApp` +Thin shells: `Application`/`MainActivity` (Android) and `main()`/window +(Desktop). They compose the `shared` UI and inject platform services. + +## Build systems + +- **Python plugin:** no build step (pure Python, stdlib + hermes core deps). + Installed by copying/symlinking into `~/.hermes/plugins/android`. Tested with + hermes's `scripts/run_tests.sh`. +- **Kotlin/CMP:** Gradle (Kotlin DSL) with the Compose Multiplatform plugin. + `./gradlew :androidApp:installDebug`, `./gradlew :desktopApp:run`, + `./gradlew :shared:testDebugUnitTest`. + +## `.gitignore` (root) — critical + +```gitignore +# hermes-agent is a read-only research reference — NEVER commit/push it +/hermes-agent/ + +# Python +__pycache__/ +*.pyc +.venv/ +venv/ + +# Kotlin / Gradle +.gradle/ +build/ +local.properties +*.iml +.idea/ + +# Android / Firebase +app/androidApp/src/main/res/values/secrets.xml +google-services.json +*.jks +keystore.jks + +# OS / misc +.DS_Store +*.log +``` + +> The `/hermes-agent/` line is non-negotiable. Add a pre-commit guard (or CI +> check) that fails if any path under `hermes-agent/` is staged. + +## Install layout (runtime) + +- **Plugin:** `~/.hermes/plugins/android/` ← copy of `gateway-plugin/` + (or a symlink for dev). Discovered by hermes's `PluginManager`. +- **App (dev):** installed on-device via `./gradlew :androidApp:installDebug`. +- **App (desktop, dev):** `./gradlew :desktopApp:run`. \ No newline at end of file diff --git a/docs/03-gateway-plugin.md b/docs/03-gateway-plugin.md new file mode 100644 index 0000000..8920397 --- /dev/null +++ b/docs/03-gateway-plugin.md @@ -0,0 +1,263 @@ +# 03 — Gateway Plugin (Python) + +The plugin is a **community-style hermes platform plugin** named `android`. +It follows the "Plugin Path" in `hermes-agent/gateway/platforms/ADDING_A_PLATFORM.md` +and the canonical example `hermes-agent/plugins/platforms/irc/adapter.py`. +**Zero hermes-core changes. Zero new Python dependencies** (`websockets` and +`httpx` are already core deps). + +> Source map for every hermes integration point: see `15-hermes-reference.md`. + +## 3.1 `plugin.yaml` (manifest) + +```yaml +name: android-platform +label: Android +kind: platform +version: 0.1.0 +description: > + Native Android / Desktop client gateway adapter for Hermes Agent. + Runs a WebSocket server inside the gateway; the app connects with a + pairing token. Supports streaming, reasoning, structured tool events, + channels/threads, media, FTS5 search, and FCM/ntfy push. +author: +requires_env: + - name: ANDROID_TOKEN + description: "Shared pairing token the app presents on connect" + prompt: "Android pairing token" + password: true +optional_env: + - name: ANDROID_WS_HOST + description: "WS bind host (default 127.0.0.1; use 0.0.0.0 for LAN)" + prompt: "WS host" + password: false + - name: ANDROID_WS_PORT + description: "WS port (default 8790)" + prompt: "WS port" + password: false + - name: ANDROID_HOME_CHANNEL + description: "Default chat id for cron/notification delivery (default android:default)" + prompt: "Home channel" + password: false + - name: ANDROID_ALLOWED_USERS + description: "Comma-separated allowed device_ids (empty = token-only auth)" + prompt: "Allowed device ids" + password: false + - name: ANDROID_ALLOW_ALL_USERS + description: "Allow any paired device (dev only)" + prompt: "Allow all devices? (true/false)" + password: false + - name: ANDROID_PUSH_BACKEND + description: "Push backend: fcm (default) or ntfy" + prompt: "Push backend" + password: false + - name: ANDROID_FCM_SERVICE_ACCOUNT + description: "Path to Firebase service-account JSON (FCM HTTP v1)" + prompt: "FCM service account path" + password: true + - name: ANDROID_FCM_SERVER_KEY + description: "Legacy FCM server key (fallback if no service account)" + prompt: "FCM server key" + password: true + - name: NTFY_TOPIC + description: "ntfy topic for push (when ANDROID_PUSH_BACKEND=ntfy)" + prompt: "ntfy topic" + password: false + - name: NTFY_SERVER_URL + description: "ntfy server URL (default https://ntfy.sh)" + prompt: "ntfy server URL" + password: false + - name: ANDROID_WS_CERT + description: "TLS cert path for WSS (optional)" + prompt: "WSS cert" + password: false + - name: ANDROID_WS_KEY + description: "TLS key path for WSS (optional)" + prompt: "WSS key" + password: false +``` + +Behavioral (non-secret) settings live in `config.yaml` under +`gateway.platforms.android.extra` (host, port, home_channel, outbox retention, +max upload bytes, tls). Secrets live in `.env`. (hermes policy: `.env` = secrets +only.) + +## 3.2 `register(ctx)` entry point + +```python +def register(ctx): + ctx.register_platform( + name="android", + label="Android", + adapter_factory=lambda cfg: AndroidAdapter(cfg), + check_fn=check_requirements, # passive: websockets importable + token set + validate_config=validate_config, # host/port/token present + is_connected=is_connected, + required_env=["ANDROID_TOKEN"], + install_hint="No extra packages needed (websockets + httpx are core deps)", + setup_fn=interactive_setup, # hermes gateway setup flow + env_enablement_fn=_env_enablement, # seed extra + home_channel from env + cron_deliver_env_var="ANDROID_HOME_CHANNEL", + standalone_sender_fn=_standalone_send, # best-effort out-of-proc cron (stretch) + parse_target_ref_fn=_parse_target_ref, # "android:[:]" + allowed_users_env="ANDROID_ALLOWED_USERS", + allow_all_env="ANDROID_ALLOW_ALL_USERS", + max_message_length=0, # 0 = no limit (WS has none) + emoji="📱", + pii_safe=False, + platform_hint=( + "You are chatting with the user through their native Iris app " + "(Android/Desktop). It renders Markdown, inline code, images, " + "audio and video, and shows your reasoning and tool activity. " + "Conversations are organized into channels and optional threads. " + "Keep formatting rich but readable." + ), + ) +``` + +Field reference (all from `PlatformEntry`, `gateway/platform_registry.py:63`): +`adapter_factory`, `check_fn`, `validate_config`, `is_connected`, `required_env`, +`install_hint`, `setup_fn`, `env_enablement_fn`, `apply_yaml_config_fn`, +`cron_deliver_env_var`, `parse_target_ref_fn`, `allowed_users_env`, +`allow_all_env`, `max_message_length`, `pii_safe`, `platform_hint`, `emoji`, +`ensure_deps_fn`. + +- **`check_requirements()`** — passive probe: `import websockets` succeeds and + `ANDROID_TOKEN` is set. Never installs. +- **`_env_enablement()`** — returns a dict seeding `PlatformConfig.extra` + (host/port/home_channel/push_backend) + a `home_channel` key + `{"chat_id": "android:default", "name": "Default"}` so `hermes gateway status` + and cron home-channel resolution work without instantiating the adapter. +- **`_parse_target_ref(ref)`** — if `ref` starts with `android:`, return + `(chat_id, thread_id)` parsed from `android:[:]`; else `None`. +- **`interactive_setup()`** — prompts for token (or generates one), host/port, + push backend + credentials, prints a QR code (pairing) and the app URL. + +## 3.3 `AndroidAdapter(BasePlatformAdapter)` + +Constructor: `super().__init__(config=config, platform=Platform("android"))`. +Reads `config.extra` (env overrides win). Initializes: WS server (not started +until `connect()`), connection registry, outbox (SQLite under +`get_hermes_home()/"android"`), push backend, pairing store, channel directory. + +### Lifecycle +- **`connect(*, is_reconnect=False) -> bool`** + - Acquire scoped lock (`gateway.status.acquire_scoped_lock("android", key)`) + so two profiles can't bind the same port/identity. + - Start the `websockets` server on `host:port` (TLS if cert/key set). + - `_mark_connected()`; return True. +- **`disconnect()`** + - Stop server, close all device sockets, release lock, `_mark_disconnected()`. + +### Inbound (app → agent) +- WS `message.send {text, reply_to?, media_refs?}` → build `SessionSource` via + `self.build_source(chat_id, chat_name, chat_type, user_id, user_name, + thread_id)` → build `MessageEvent(text=…, message_type=TEXT, source=…, + media_urls=[cached paths], media_types=[…], reply_to_message_id=…)` → + `await self.handle_message(event)`. +- Slash commands arrive as plain text starting with `/`; the gateway's command + pipeline resolves + dispatches them (no special handling needed). +- `picker.select {picker_id, value}` → route to the gateway-side resolvers + (model picker / choice picker / clarify / approval / slash-confirm) using the + shared callback-id conventions (`cl::`, `appr::`, + `sc::`). +- `channel.create` / `channel.rename` / `channel.set_default` → mutate the + channel directory (SQLite) + emit `channel.*` frames to all devices. +- `search {query, scope, chat_id?, thread_id?}` → `search.py` → `search.results`. +- `media.upload` (chunked) → `media.py` → `cache_*_from_bytes` → `media_ref`. +- `media.pull {media_id}` → stream cached bytes as binary frames. +- `fcm.register {token}` / `hello` → update device registry. +- `read.receipt {message_id}` → mark delivered/read (drives ✓✓), ack. +- `sync {cursor}` → `outbox.py` → replay frames since cursor. + +### Outbound (agent → app) +- **`send(chat_id, content, reply_to=None, metadata=None) -> SendResult`** + - Split reasoning prefix (see `05-streaming.md`) → `reasoning` field. + - If **any** device is connected: broadcast `message` frame to all. + - Else (no live devices): append to **outbox** + fire **push** (FCM/ntfy). + - Return `SendResult(success=True, message_id=)`. +- **`edit_message(chat_id, message_id, content)`** → `message.update` frame + (drives streaming). If no device, no-op (outbox holds the final `send`). +- **`send_typing(chat_id, metadata=None)`** → `typing` frame. +- **`get_chat_info(chat_id) -> dict`** → `{"name": , "type": "dm"|"channel"}` + from the channel directory. +- **Media send** — `send_image / send_video / send_document / send_voice / + send_image_file / send_multiple_images`: stage the file in the media cache, + mint a `media_id`, emit `media.offer {media_id, mime, size, filename, kind}`, + serve bytes on `media.pull`. (Base-class `extract_media`/`extract_images` + already pull `MEDIA:`/image tags out of agent text and call these.) +- **Interactive pickers** — `send_model_picker(...)`, `send_choice_picker(...)`, + `send_clarify(...)`, `send_exec_approval(...)`, `send_slash_confirm(...)`: + emit `picker.model` / `picker.choice` / `picker.clarify` / `picker.approval` / + `picker.confirm` frames with options; store pending state keyed by + `picker_id`; resolve on `picker.select`. +- **`create_handoff_thread(chat_id, name)`** → create a thread id, register in + channel directory, return it (used by cron "continuable" threads). + +### Streaming hooks +The main gateway drives delivery through the **legacy callback path**: +- `stream_delta_callback` → `GatewayStreamConsumer` → `send()` (first) + + `edit_message()` (updates) → `message.start` / `message.update`. +- `tool_progress_callback` → progress queue → `send_progress_messages` → + `send()` → `tool.*` frames (structured; classified in the adapter). +- `interim_assistant_callback` → consumer `on_commentary` → `send()` → + `commentary` frames. + +The adapter tracks per-chat **turn state** (in-turn, current streaming +`message_id`, last tool index) to classify outbound `send()` calls into +`message` vs `tool.*` vs `commentary`. The exact classification markers are +verified empirically in M2 (see `13-testing.md`). + +## 3.4 WebSocket server (`ws_server.py`) + +- Library: **`websockets`** (core dep, v15). `websockets.serve(handler, host, + port, ssl=ctx)`. +- **Handler** per connection: + 1. Await first frame; must be `hello {token, device_id, device_name, caps, + fcm_token?}`. Verify token (constant-time) + allowlist. On failure: send + `error {code:"auth"}` and close. + 2. On success: register in connection registry + (`device_id → {ws, caps, fcm_token}`), send + `hello.ack {server_caps, sync_cursor, channels[]}`. + 3. Loop: decode frames, dispatch to adapter inbound handlers. + 4. On close: deregister; if no devices remain, ensure pending outbox + frames have push fired. +- **Routing:** `emit(chat_id, frame)` → broadcast to **all** connected + devices (no per-chat subscribe; single-user model). Global frames + (`channel.*`, `status`) also broadcast to all. +- **Heartbeat:** WS ping/pong + app-level `ping`/`pong`; dead peers reaped. +- **Backpressure:** per-connection send queue with a bounded buffer; drop + `message.update` (coalesce to latest) under pressure, never drop + `message`/`tool.end`/`notification`. + +## 3.5 State & storage (all under `get_hermes_home()/"android"`) + +> Use `get_hermes_home()` from `hermes_constants` for **all** paths (profile-safe). +> Never hardcode `~/.hermes`. + +- `devices.db` — device registry (device_id, name, caps, fcm_token, ntfy_topic, + last_seen, created). +- `channels.db` — channel directory (chat_id, name, kind: default|channel|thread, + parent_chat_id, created, is_default). +- `outbox.db` — undelivered frames per chat_id + monotonic cursor. +- `media/` — inbound + outbound media cache (reuse hermes `cache_*_from_bytes` + dirs where possible). + +## 3.6 Config resolution + +- **Secrets (`.env`):** `ANDROID_TOKEN`, `ANDROID_FCM_SERVICE_ACCOUNT`, + `ANDROID_FCM_SERVER_KEY`, `ANDROID_WS_CERT/KEY`, `NTFY_TOPIC` (if secret). +- **Behavioral (`config.yaml` → `gateway.platforms.android.extra`):** `host`, + `port`, `home_channel`, `allowed_users`, `push_backend`, `outbox_retention_hours`, + `max_upload_bytes`, `tls`. +- Env vars override `config.yaml` (hermes convention). Read secrets with the + scope-aware `_get_scoped_secret` pattern (see `plugins/platforms/irc/adapter.py:42`) + so multiplexed profiles don't leak each other's tokens. + +## 3.7 Failure & lifecycle safety + +- WS server bind failure → `_set_fatal_error("bind_failed", …, retryable=True)`. +- All outbound sends are best-effort; a dead socket latches and the frame falls + to the outbox. +- `disconnect()` cancels the server task and closes sockets cleanly. +- Token/PII redaction in all logs (hermes PII policy). \ No newline at end of file diff --git a/docs/04-wire-protocol.md b/docs/04-wire-protocol.md new file mode 100644 index 0000000..7d37614 --- /dev/null +++ b/docs/04-wire-protocol.md @@ -0,0 +1,327 @@ +# 04 — Wire Protocol + +JSON frames over a single WebSocket. One connection per device. Text frames are +JSON; media travels as **binary frames** (chunked) referenced by a header frame. + +## Envelope + +Every frame: + +```json +{ + "v": 1, + "id": 42, // optional; present on requests + their responses + "type": "message", // frame type (below) + "chat_id": "android:default", // optional; scope for chat-scoped frames + "thread_id": "t_123", // optional + "payload": { } // type-specific object +} +``` + +- `v` — protocol version (currently `1`). Server rejects unknown major versions. +- `id` — request id (client-chosen). Responses/acks echo it. Events have no `id`. +- `chat_id` / `thread_id` — top-level for convenience; may also be in `payload`. +- Unknown `type`s are ignored (forward-compat); unknown `payload` fields ignored. + +**Binary media frames** are not JSON. A media transfer is: one JSON header frame +(`media.upload.start` / `media.pull` ack) followed by raw binary frames, then a +JSON `media.upload.end` / final ack. See `07-media.md`. + +## Server → App (events / responses) + +### `hello.ack` +Pairing succeeded. +```json +{"type":"hello.ack","payload":{ + "server_caps":{"streaming":true,"reasoning":true,"tools":true,"media":true, + "search":true,"push":"fcm","pickers":true}, + "sync_cursor":1042, + "channels":[{"chat_id":"android:default","name":"Default","kind":"default","is_default":true}] +}} +``` + +### `message` +A final / standalone message. +```json +{"type":"message","chat_id":"android:default","thread_id":null, + "payload":{ + "message_id":"m_9001","role":"assistant", + "text":"Here is the answer…", + "reasoning":"The user asked… so I will…", // optional; render ABOVE text + "media":[{"media_id":"md_5","kind":"video","mime":"video/mp4","size":123456, + "filename":"clip.mp4"}], // optional + "reply_to":"m_8999", // optional + "model":"qwen3-27b","tokens":11,"ts":1724000000000 + }} +``` +`role` ∈ `user | assistant | system | cron`. `reasoning` present only when the +agent produced reasoning and `show_reasoning` is on. + +### `message.start` / `message.update` / `message.stop` +Streaming a bubble. `update` carries the **full** current text (app replaces). +```json +{"type":"message.start","chat_id":"…","payload":{"message_id":"m_9002","role":"assistant"}} +{"type":"message.update","chat_id":"…","payload":{"message_id":"m_9002","text":"partial…"}} +{"type":"message.stop","chat_id":"…","payload":{"message_id":"m_9002","final_text":"full…", + "reasoning":"…","model":"…","tokens":11}} +``` + +### `commentary` +Intermediate assistant beat (between tool iterations). +```json +{"type":"commentary","chat_id":"…","payload":{"message_id":"m_9003","text":"Let me inspect the repo first."}} +``` + +### `tool.start` / `tool.progress` / `tool.end` +**Structured** tool events. The app decides how much to show (everything / +truncated / nothing). +```json +{"type":"tool.start","chat_id":"…","payload":{ + "index":3,"name":"terminal","preview":"pytest -q","args":{"command":"pytest -q"}}} +{"type":"tool.progress","chat_id":"…","payload":{"index":3,"name":"terminal","note":"running…"}} +{"type":"tool.end","chat_id":"…","payload":{"index":3,"name":"terminal","ok":true,"duration":12.4, + "output_preview":"12 passed"}} +``` +`args` may be large; the app truncates per its setting. `output_preview` is a +short tail (full output is not streamed — it lives in agent history). + +### `typing` / `typing.stop` +```json +{"type":"typing","chat_id":"…","payload":{"on":true}} +``` + +### `notification` +In-app banner (foreground) and/or push mirror (background). +```json +{"type":"notification","chat_id":"…","payload":{ + "kind":"channel_renamed","title":"ARIA","body":"Renamed topic to …","ts":1724000000000}} +``` +`kind` ∈ `channel_renamed | channel_created | cron | approval | clarify | generic`. + +### `picker.model` / `picker.choice` / `picker.clarify` / `picker.approval` / `picker.confirm` +Interactive prompts. App renders a native picker; answers via `picker.select`. +```json +{"type":"picker.model","chat_id":"…","payload":{ + "picker_id":"pm_1","current_model":"qwen3-27b","current_provider":"local", + "providers":[{"id":"local","label":"Local","models":[{"id":"qwen3-27b","label":"Qwen3 27B"}]}]}} +{"type":"picker.choice","chat_id":"…","payload":{ + "picker_id":"pc_1","title":"Reasoning effort","choices":[ + {"value":"low","label":"Low"},{"value":"high","label":"High","is_current":true}]}} +``` + +### `channel.list` / `channel.created` / `channel.renamed` / `channel.deleted` +Channel directory updates. **Broadcast to all connected devices** (no explicit +subscribe; the server pushes to every open WS). +```json +{"type":"channel.created","payload":{"chat_id":"android:chan_7","name":"Cron Reports", + "kind":"channel","parent_chat_id":null}} +``` + +### `history` +Response to a `history` request. Returns a page of messages for a chat/thread. +```json +{"type":"history","id":20,"chat_id":"android:default","thread_id":null, + "payload":{ + "messages":[ + {"message_id":"m_8990","role":"user","text":"Hi","ts":1723990000000}, + {"message_id":"m_8991","role":"assistant","text":"Hello!","reasoning":"…", + "model":"qwen3-27b","tokens":8,"ts":1723990001000} + ], + "has_more":true, + "oldest_message_id":"m_8990" + }} +``` +`messages` are ordered oldest → newest. Paginate with `before_message_id` in the +request. The app uses this to **populate the initial view** when a channel is +opened (complements `sync`, which only replays undelivered outbox frames). + +### `commands.catalog` +Response to a `commands.catalog` request. Full slash-command list. +```json +{"type":"commands.catalog","id":21,"payload":{ + "commands":[ + {"name":"/new","description":"Start a new session","args_hint":"","category":"session"}, + {"name":"/model","description":"Switch model","args_hint":"","category":"config"}, + {"name":"/reasoning","description":"Toggle reasoning effort","args_hint":"[low|medium|high]","category":"config"}, + {"name":"/status","description":"Show session status","args_hint":"","category":"info"}, + {"name":"/cron","description":"Manage cron jobs","args_hint":"","category":"automation"}, + {"name":"/tools","description":"List available tools","args_hint":"","category":"info"} + ]}} +``` + +### `commands.complete` +Response to a `commands.complete` request. Autocomplete matches for a typed prefix. +```json +{"type":"commands.complete","id":22,"payload":{ + "prefix":"/mod", + "matches":[ + {"name":"/model","description":"Switch model","args_hint":""} + ]}} +``` + +### `agent.busy` / `agent.idle` +Agent lifecycle for a chat/thread. App shows a "thinking…" indicator on `busy`. +```json +{"type":"agent.busy","chat_id":"android:default","thread_id":null, + "payload":{"reason":"processing"}} +{"type":"agent.idle","chat_id":"android:default","thread_id":null,"payload":{}} +``` +`reason` ∈ `processing | tool | waiting_input | cron`. + +### `search.results` +```json +{"type":"search.results","id":7,"payload":{ + "query":"deploy","scope":"all","hits":[ + {"message_id":"m_123","chat_id":"android:chan_7","thread_id":null, + "role":"assistant","snippet":"…deploy the service…","ts":1723900000000}]}} +``` + +### `media.offer` +Agent-sent media is available; app pulls bytes. +```json +{"type":"media.offer","chat_id":"…","payload":{ + "media_id":"md_5","kind":"video","mime":"video/mp4","size":123456,"filename":"clip.mp4"}} +``` + +### `status` +Gateway lifecycle / session info. +```json +{"type":"status","payload":{"state":"online","session":{"chat_id":"…","model":"…","tokens":11}}} +``` +`state` ∈ `online | restarting | degraded`. + +### `error` +```json +{"type":"error","id":7,"payload":{"code":"not_found","message":"chat_id unknown"}} +``` +`code` ∈ `auth | not_found | rate_limited | media_too_large | unsupported | internal`. + +### `pong` +Keepalive reply to `ping`. + +## App → Server (requests / actions) + +### `hello` +First frame; auth + caps. +```json +{"type":"hello","payload":{ + "token":"","device_id":"dev_a1b2","device_name":"MIX 2S", + "caps":{"min_protocol":1,"media":true,"push":"fcm"}, + "fcm_token":"","ntfy_topic":""}} +``` + +### `message.send` +Send text (or a `/slash-command`). +```json +{"type":"message.send","id":10,"chat_id":"android:default","thread_id":null, + "payload":{"text":"/model qwen3-27b","reply_to":"m_9001","media_refs":["mu_1"]}} +``` +`media_refs` reference completed `media.upload`s to attach. + +### `media.upload.start` / (binary) / `media.upload.end` +See `07-media.md`. +```json +{"type":"media.upload.start","id":11,"payload":{ + "media_ref":"mu_1","kind":"image","mime":"image/jpeg","size":204800,"filename":"a.jpg"}} +// … binary frames … +{"type":"media.upload.end","id":11,"payload":{"media_ref":"mu_1","sha256":"…"}} +``` + +### `media.pull` +Request agent-sent media bytes. +```json +{"type":"media.pull","id":12,"payload":{"media_id":"md_5"}} +// server replies: binary frames, then {"type":"media.pull.end","id":12,"payload":{"ok":true}} +``` + +### `picker.select` +Answer an interactive picker. +```json +{"type":"picker.select","id":13,"payload":{"picker_id":"pm_1","value":"local/qwen3-27b"}} +``` + +### `channel.create` / `channel.rename` / `channel.set_default` / `channel.delete` +```json +{"type":"channel.create","id":14,"payload":{"name":"Cron Reports","kind":"channel"}} +{"type":"channel.rename","id":15,"chat_id":"android:chan_7","payload":{"name":"Reports"}} +{"type":"channel.set_default","id":16,"chat_id":"android:chan_7","payload":{}} +``` + +### `search` +```json +{"type":"search","id":17,"payload":{"query":"deploy","scope":"all"}} +{"type":"search","id":18,"payload":{"query":"deploy","scope":"chat","chat_id":"android:chan_7","thread_id":null}} +``` +`scope` ∈ `all | chat`. + +### `read.receipt` +App → server: "user has viewed this message." Server stores the read state and +broadcasts to other devices (for multi-device ✓✓ sync). The app uses it to +mark messages as read locally (✓✓ on user bubbles). +```json +{"type":"read.receipt","payload":{"chat_id":"android:default","message_id":"m_9001"}} +``` + +### `history` +Load a page of messages for a chat/thread (initial open, scroll-up pagination). +```json +{"type":"history","id":20,"chat_id":"android:default","thread_id":null, + "payload":{"before_message_id":"m_8990","limit":50}} +``` +`before_message_id` — return messages older than this (omit for newest page). +`limit` — max messages (default 50, max 200). + +### `commands.catalog` +Fetch the full slash-command catalog (for the `Menü` bottom sheet). +```json +{"type":"commands.catalog","id":21,"payload":{}} +``` + +### `commands.complete` +Autocomplete for a typed `/prefix`. +```json +{"type":"commands.complete","id":22,"payload":{"prefix":"/mod"}} +``` + +### `agent.stop` +Stop the current agent turn (abort generation / tool execution). +```json +{"type":"agent.stop","id":23,"chat_id":"android:default","thread_id":null,"payload":{}} +``` + +### `agent.steer` +Inject a steering message mid-turn (redirects the agent without a new turn). +```json +{"type":"agent.steer","id":24,"chat_id":"android:default","thread_id":null, + "payload":{"text":"Actually, focus on the error case."}} +``` + +### `sync` +Reconnect catch-up. Replays **undelivered outbox frames** (frames sent while +this device was offline). Does NOT load full history — use `history` for that. +```json +{"type":"sync","id":19,"payload":{"cursor":1042}} +// server replays outbox frames with cursor > 1042, then {"type":"sync.done","id":19,"payload":{"cursor":1099}} +``` + +### `fcm.register` +Update push token. +```json +{"type":"fcm.register","payload":{"fcm_token":"","ntfy_topic":""}} +``` + +### `ping` +Keepalive. `{"type":"ping","payload":{"ts":1724000000000}}` → `pong`. + +## Ordering & reliability + +- Frames are ordered per connection (TCP/WS). Streaming `message.update` frames + for a `message_id` are monotonic; the app may coalesce to the latest. +- Terminal frames (`message`, `message.stop`, `tool.end`, `notification`, + `picker.*`, `channel.*`, `agent.busy`, `agent.idle`, `history`, + `commands.catalog`, `commands.complete`) are **never dropped** under + backpressure; only intermediate `message.update`/`tool.progress` are coalesced. +- Anything not delivered live goes to the **outbox** and is replayed by `sync`. +- **Broadcast:** channel directory events (`channel.*`) and read-receipts are + pushed to **all** connected devices for that gateway (no subscribe step). +- Requests get exactly one response or `error` (matched by `id`). \ No newline at end of file diff --git a/docs/05-streaming.md b/docs/05-streaming.md new file mode 100644 index 0000000..5d85a64 --- /dev/null +++ b/docs/05-streaming.md @@ -0,0 +1,143 @@ +# 05 — Streaming, Reasoning, Tools, Intermediate Messages + +This doc covers the four "agent transparency" features and exactly how each is +produced by the gateway and rendered by the app. + +> The main hermes gateway delivers via the **legacy callback path** (not the +> ACP-only event-native `render_message_event` path). We map the legacy +> `send`/`edit_message`/progress calls to our WS frames. + +## 5.1 Text streaming + +**Gateway side.** The agent's `stream_delta_callback` feeds a +`GatewayStreamConsumer` (`gateway/stream_consumer.py:156`). The consumer +accumulates text and, at intervals / thresholds, calls: +- `adapter.send(chat_id, text)` — first time a bubble is created. +- `adapter.edit_message(chat_id, message_id, text)` — subsequent updates + (each carries the **full** accumulated text). + +**Adapter → frames.** +- First `send()` of a turn segment → `message.start {message_id, role}`. +- Each `edit_message()` → `message.update {message_id, text}` (full text). +- Segment/turn finalization → `message.stop {message_id, final_text, reasoning?, + model?, tokens?}` (or a final `message` frame when not streaming). + +**App side.** Maintain a live bubble per `message_id`. On `message.update`, +replace the bubble text (cheap: it's a full snapshot). On `message.stop`, +finalize (attach reasoning/model/tokens footer, stop the cursor). Auto-scroll +while the user is at the bottom. + +**Streaming on/off.** Controlled by hermes `display.platforms.android.streaming` +(default follows global). When off, the app just gets one final `message` frame. + +## 5.2 Reasoning (shown *before* the message) + +**Requirement:** reasoning appears as a block **above** the answer (like the +reference screenshot's "Reasoning:" panel with a copy button). + +**Gateway side.** hermes prepends reasoning to the final response when +`show_reasoning` is enabled (`gateway/run.py:20089`). The format is stable and +chosen by `reasoning_style` (`gateway/display_config.py:37`): +- `code` (default): `💭 **Reasoning:**\n```\n\n```\n\n` +- `blockquote`: `> 💭 **Reasoning:**\n> …\n\n` +- `subtext`: `-# 💭 Reasoning\n-# …\n\n` (Discord-style) + +**Plugin config.** Set for the `android` platform: +```yaml +display: + platforms: + android: + show_reasoning: true + reasoning_style: code # we split on the code-fence form +``` + +**Adapter split.** In `send()`, detect the `code`-style prefix and split: +``` +prefix = "💭 **Reasoning:**\n```\n" +# find the closing "\n```\n\n" after the prefix +reasoning = text[len(prefix):close_idx] +body = text[close_idx + len("\n```\n\n"):] +``` +Emit `message {reasoning: , text: , …}`. If no prefix is found +(reasoning off / no reasoning), emit `message {text: …}` with no `reasoning`. + +> Robustness: the split is a best-effort parse of a *stable, gateway-owned* +> format. If the format ever changes, the fallback is "no reasoning field, full +> text" — the app still shows the answer. Verified in M2 against the live +> gateway. + +**App side.** `ReasoningBlock` composable: collapsible, header "💭 Reasoning", +monospace body, a **copy** button (matches reference). Rendered **above** the +message body. Collapsed by default if long; expanded tap. + +## 5.3 Tool output (app controls verbosity) + +**Requirement:** hermes supports several tool-display modes; the **gateway sends +full structured data**, and the **app** chooses how much to show +(everything / truncated / nothing). + +**Gateway side.** Tool activity flows through `tool_progress_callback` → the +gateway progress queue → `send_progress_messages` (`gateway/run.py:4603`) → +`adapter.send()`. The adapter receives tool lines during a turn. + +**Adapter → frames.** The adapter classifies tool activity (via turn-state + +line format) and emits **structured** frames — not pre-formatted strings: +- `tool.start {index, name, preview, args}` — a tool call began. +- `tool.progress {index, name, note}` — in-progress update (optional). +- `tool.end {index, name, ok, duration, output_preview}` — completed. + +`index` is a monotonic per-turn counter so the app correlates start→end. +`args` is the full argument dict (the app truncates). `output_preview` is a +short tail; full tool output is **not** streamed (it lives in agent history and +is reachable via search). + +**App side — the verbosity setting** (Settings → "Tool detail"): +- **Everything** — show tool name, full args (collapsible), and output preview. +- **Truncated** (default) — show `emoji name: "short preview"` one-liner, + collapsible to expand. +- **Nothing** — suppress `tool.*` frames entirely (clean chat). + +Rendering: a `ToolCard` per tool, grouped under the message it belongs to, with +a spinner while `tool.end` hasn't arrived, a ✓/✗ on completion, and duration. + +> Classification detail: the exact way to distinguish a tool-progress `send()` +> from a regular `send()`/commentary is confirmed empirically in M2 by running +> the real gateway with a test WS client and observing the calls. The adapter +> keeps a per-chat turn state machine (turn active, current streaming id, last +> tool index) to make the classification deterministic. + +## 5.4 Intermediate messages (commentary) + +**Requirement:** show the agent's interim beats (e.g. "I'll inspect the repo +first.") as distinct messages. + +**Gateway side.** `interim_assistant_callback` → consumer `on_commentary` +(`gateway/stream_consumer.py:518`) → delivered as its own message. + +**Adapter → frames.** `commentary {message_id, text}`. + +**App side.** Render as a **dimmed / smaller** bubble, visually distinct from +final answers (e.g. reduced opacity, no model footer). It reads as a "beat" in +the conversation, not a full reply. + +## 5.5 Typing indicator + +`send_typing` → `typing {on:true}`; the app shows an animated indicator in the +chat header / above the composer until `typing.stop` or the first +`message.start`. + +## 5.6 Frame sequence for a typical turn + +``` +app → message.send {text:"summarize the repo"} +srv → typing {on:true} +srv → message.start {message_id:m1} +srv → message.update {m1, "Let me look…"} (streaming) +srv → tool.start {index:1, name:terminal, args:{command:"ls -la"}} +srv → tool.end {index:1, name:terminal, ok:true, duration:0.4} +srv → commentary {m2, "Found 12 files."} +srv → message.start {message_id:m3} +srv → message.update {m3, "The repo has…"} +srv → message.stop {m3, final_text:"…", reasoning:"…", model:"…", tokens:42} +srv → typing {on:false} +``` \ No newline at end of file diff --git a/docs/06-channels-cron-search.md b/docs/06-channels-cron-search.md new file mode 100644 index 0000000..05e74de --- /dev/null +++ b/docs/06-channels-cron-search.md @@ -0,0 +1,107 @@ +# 06 — Channels, Threads, Cron Delivery, Search + +## 6.1 Concept model + +The hermes gateway already models conversations as `SessionSource` with +`chat_id` + `thread_id` + `chat_topic` (`gateway/platforms/base.py:7047` +`build_source`). We map app concepts onto these existing primitives — **no new +gateway identity concepts**. + +| App concept | hermes primitive | Example | +|---|---|---| +| Default chat | home channel `chat_id` | `android:default` | +| A thread (inside default chat) | `thread_id` under the default `chat_id` | `chat_id=android:default, thread_id=t_12` | +| A user-created channel | a new `chat_id` | `android:chan_7` | +| A thread inside a channel | `thread_id` under that `chat_id` | `chat_id=android:chan_7, thread_id=t_31` | + +- **`chat_id`** = the conversation lane (a channel or the default chat). +- **`thread_id`** = an optional sub-lane within a `chat_id` (topic-like). +- The **channel directory** (plugin SQLite, `channels.db`) stores + `{chat_id, name, kind: default|channel|thread, parent_chat_id, is_default, + created}` and is the source of truth for the app's channel list. + +## 6.2 Default chat + +- On first connect, the plugin ensures a **default channel** exists: + `chat_id = ANDROID_HOME_CHANNEL` (default `android:default`), `kind=default`, + `is_default=true`, name "Default". +- It is also the **cron home channel** (`cron_deliver_env_var= + ANDROID_HOME_CHANNEL`), so `deliver=android` (bare) routes here. +- The app opens the default chat on launch. + +## 6.3 Threads (toggle for overview) + +- **Requirement:** a default chat where the user can **activate threads** for a + better overview, or not. +- **Thread toggle** (per default chat, in the chat header menu): + - **Threads OFF** — flat conversation; all messages use `thread_id=null`. + - **Threads ON** — the app groups the conversation into topic-like lanes. + Each new "topic" mints a `thread_id` (via `channel.create {kind:thread, + parent_chat_id:android:default}` or an implicit thread). The UI shows a + topic switcher (like Telegram topics) above the message list. +- Threads are **app-organized** but **gateway-real**: each `thread_id` is a + distinct hermes session lane, so context is isolated per thread and cron can + target a specific thread. +- The gateway's `create_handoff_thread` is used where hermes wants to open a + named thread (e.g. continuable cron). + +## 6.4 User-created channels (for cron delegation) + +- **Requirement:** the user creates new channels so **cron job outputs can be + delegated to them** instead of the default chat. +- **`channel.create {name, kind:"channel"}`** → plugin mints + `chat_id = android:chan_`, stores in directory, broadcasts + `channel.created` to all devices. The new channel appears in the channel list. +- **`channel.rename` / `channel.set_default` / `channel.delete`** manage the + directory (rename broadcasts `channel.renamed`; delete is soft — marks + archived, keeps history for search). +- **Cron targeting** (the key payoff): because the plugin registers + `parse_target_ref_fn` and `cron_deliver_env_var`, cron jobs and the + `send_message` tool can target any channel/thread: + - `deliver="android"` → home (default) channel. + - `deliver="android:android:chan_7"` → that channel. + - `deliver="android:android:chan_7:t_31"` → that channel's thread. + - In-chat: the agent's `cronjob` tool can be told "deliver to the *Cron + Reports* channel"; the gateway resolves the name via the channel directory. +- **In-app affordance:** each channel's menu has "Set as cron target" / shows a + badge when a cron job points at it, and the channel name is offered in the + `/cron` creation flow. + +## 6.5 Cron delivery mechanics (how it works under the hood) + +- Cron resolves delivery targets in `cron/scheduler.py:2148` + (`_resolve_single_delivery_target`). For `platform:chat_id[:thread_id]` it + calls `tools.send_message_tool.resolve_send_target`, which uses our + `parse_target_ref_fn` to parse `android:[:]`. +- Delivery then calls the **live adapter's `send(chat_id, text, …)`** (gateway + running) → our WS `message` frame (or outbox+push if the app is offline). +- Cron deliveries are framed with a `[Cron delivery: ]` header by hermes; + the app can style cron messages distinctly (e.g. a small "⏰ " + chip) using the `role:"cron"` / header. +- **Mirror option:** hermes `cron.mirror_delivery` (default off) can also mirror + a cron delivery into the origin session; we leave it off to keep channels + clean. + +## 6.6 Search + +- **Requirement:** search with settings "search everywhere" / "search in this + chat/channel". +- **Backend:** hermes's session store is **SQLite + FTS5** + (`hermes_state.py`, `hermes_state_search.py`). The plugin's `search.py` opens + the session DB read-only and runs FTS5 queries. +- **`search` frame** → `search.results`: + - `scope:"all"` — search **everywhere** (all channels/threads/sessions). + - `scope:"chat"` — restrict to the given `chat_id` (and optional `thread_id`). +- **Result hit:** `{message_id, chat_id, thread_id, role, snippet, ts}`. The app + renders a results list; tapping a hit navigates to that channel/thread and + scrolls to + highlights the message. +- **Query syntax:** plain text (FTS5). Optional `role:` / `channel:` qualifiers + are a nice-to-have; v1 is plain-text + scope. +- **Privacy:** search is local to the user's own hermes home; no data leaves the + machine. + +## 6.7 Channel list frame + +`hello.ack` and `channel.*` frames carry the directory. App keeps a local copy +(Room) and reconciles on `channel.*` events (merge, don't clobber — see +`10-android-app.md` state rules). \ No newline at end of file diff --git a/docs/07-media.md b/docs/07-media.md new file mode 100644 index 0000000..6a220de --- /dev/null +++ b/docs/07-media.md @@ -0,0 +1,96 @@ +# 07 — Media (upload, download, playback) + +Media travels **over the WebSocket** as chunked binary frames (decision: no +separate HTTP server; keeps the plugin to `websockets` only). Both directions +use the same chunking. + +## 7.1 Kinds & MIME + +`kind` ∈ `image | audio | video | document | voice`. +- `image` — `image/*` (jpg/png/webp/gif/heic). +- `audio` — `audio/*` (mp3/m4a/ogg/…) — music. +- `video` — `video/*` (mp4/webm/mov). +- `document` — anything else (pdf, docx, zip, txt, …). +- `voice` — short voice note (`audio/ogg; codecs=opus` typical). + +The app sniffs `kind` from the picked file's MIME; the plugin re-sniffs on +receipt (don't trust the client) using hermes helpers +(`gateway/platforms/base.py` `_sniff_audio_ext`, `_looks_like_image`). + +## 7.2 Inbound (app → agent) — `media.upload` + +**Flow:** +1. App picks a file (SAF) → reads size + MIME. +2. App sends `media.upload.start {media_ref, kind, mime, size, filename}`. +3. App streams the file as **binary WS frames** (e.g. 256 KiB chunks). +4. App sends `media.upload.end {media_ref, sha256}`. +5. Plugin verifies size ≤ `max_upload_bytes` and sha256, writes to the media + cache via hermes `cache_*_from_bytes`: + - image → `cache_image_from_bytes` + - audio/voice → `cache_audio_from_bytes` + - video → `cache_video_from_bytes` + - document → `cache_document_from_bytes` + → returns a local path. +6. The path is attached to the next `message.send` via `media_refs`, becoming + `MessageEvent.media_urls` + `media_types` + (`gateway/platforms/base.py:2337`). The agent's vision/audio tools can then + read the file. + +**Limits:** `get_inbound_media_max_bytes()` / `validate_inbound_media_size` +(`base.py:758/779`) enforce the cap; over-limit → `error {code:"media_too_large"}`. + +**Backpressure:** large uploads use the WS flow control; the plugin reads +binary frames into a temp file (not memory) to bound RAM. + +## 7.3 Outbound (agent → app) — `media.offer` / `media.pull` + +**Flow:** +1. Agent produces/references media (e.g. generates an image, or replies with a + `MEDIA:` tag / image URL). hermes base `extract_media` / `extract_images` + (`base.py:4439/4884`) pull these out and call the adapter's + `send_image / send_video / send_document / send_voice / send_image_file / + send_multiple_images`. +2. Adapter stages the file in the media cache, mints a `media_id`, and emits + `media.offer {media_id, kind, mime, size, filename}` (inside/with the + `message` frame's `media[]`). +3. App sends `media.pull {media_id}`. +4. Plugin streams the file as **binary WS frames**; ends with + `media.pull.end {ok:true}`. +5. App writes to its cache dir and hands the path to the player/viewer. + +**Security:** `validate_media_delivery_path` (`base.py:1684`) + the media +delivery root/recency/denied-path checks (`base.py:1312-1480`) ensure the plugin +only serves files hermes is allowed to deliver (no arbitrary file read). + +## 7.4 Live playback (AI-sent music/video) + +**Requirement:** play AI-sent music and video **in the app**. + +- **Audio (music/voice):** `ExoPlayer` (Media3). Inline player in the bubble + (play/pause, seek, duration); a persistent **mini-player** for music that + survives navigation. Voice notes play inline with a waveform. +- **Video:** `ExoPlayer` inline player (play/pause, seek, fullscreen, PiP on + Android). Streams from the local cache file after `media.pull`. +- **Documents/images:** image viewer (zoom) / open-with for documents (Android + `Intent.ACTION_VIEW` with a `FileProvider` URI; Desktop opens with the system + handler). +- **Desktop:** ExoPlayer is Android-only → the `MediaPlayer` expect/actual uses + a desktop backend (see `11-desktop-app.md`): a `libmpv`/`mpv`-backed surface + or a WebView fallback for video, and a desktop audio player for music. + +## 7.5 Chunking parameters + +- Chunk size: **256 KiB** (tunable). +- Binary frames carry raw bytes only; framing/metadata is in the JSON header + + end frames. +- Reassembly is ordered (WS preserves order); a gap/corruption → abort + + `error {code:"internal"}` + retry the whole transfer. +- `sha256` in `media.upload.end` / a size check on pull verify integrity. + +## 7.6 App-side storage + +- Cache dir: app-specific external cache (`getExternalCacheDir()/media`). +- LRU eviction by size (configurable, default 500 MB) so old media doesn't fill + the device. +- A `MediaRepository` tracks `{media_id, local_path, kind, size, ts}` in Room so + bubbles can re-render players after process death. \ No newline at end of file diff --git a/docs/08-push.md b/docs/08-push.md new file mode 100644 index 0000000..4f356a1 --- /dev/null +++ b/docs/08-push.md @@ -0,0 +1,105 @@ +# 08 — Push Notifications, Outbox & Sync + +The gateway can't reach a sleeping phone directly. Push goes through a cloud +relay. **Decision: FCM primary, ntfy fallback** (`ANDROID_PUSH_BACKEND`). + +## 8.1 When push fires + +- A frame targets a `chat_id` whose device is **disconnected** (WS closed) → + drop to **outbox** + fire **push**. +- Also fire push for high-priority foreground events the user should see even if + the app is backgrounded (approvals, clarifies, cron completions) — the app + decides whether to also show an in-app banner. +- If the device is **connected**, no push (the live frame is enough). + +## 8.2 `PushBackend` interface (`push.py`) + +```python +class PushBackend(Protocol): + name: str + async def send(self, *, device_id: str, chat_id: str, + title: str, body: str, + data: dict) -> bool: ... + def configured(self) -> bool: ... +``` + +Selected at adapter init by `ANDROID_PUSH_BACKEND` (`fcm` default, `ntfy`). + +### 8.2.1 `FcmBackend` (primary) +- **FCM HTTP v1 API** via `httpx` (core dep). Auth = Firebase **service + account** (`ANDROID_FCM_SERVICE_ACCOUNT` JSON path) → mint a short-lived + OAuth2 access token (cached, refreshed before expiry). +- Fallback: legacy **server key** (`ANDROID_FCM_SERVER_KEY`) if no service + account (simpler, but legacy). +- Target = the device's **FCM token** (registered via `hello` / + `fcm.register`, stored in `devices.db`). +- Payload: + - `notification {title, body}` → system notification (tap → open app). + - `data {chat_id, thread_id, kind, message_id, cursor}` → app uses to `sync`. +- **Data-only option:** for background catch-up, send a data message (silent) so + the app's `FirebaseMessagingService` wakes a foreground service and syncs + without a visible notification (used for non-urgent updates). +- Batch: FCM allows up to 500 tokens/message; we send per-device (1 user, few + devices). + +### 8.2.2 `NtfyBackend` (fallback, self-host friendly) +- Reuses hermes's existing ntfy publish path (hermes ships an ntfy adapter). +- Publish to `NTFY_TOPIC` on `NTFY_SERVER_URL` (default `https://ntfy.sh`) via + `httpx` POST, with an `X-Title` / `X-Message` / `X-Tag` / `X-Priority` and a + JSON `data` attachment (`{chat_id, thread_id, kind, cursor}`). +- The app subscribes to the topic (via an ntfy client lib or a lightweight + listener) to receive pushes without Firebase. +- Best for users who self-host ntfy and want zero Firebase. + +## 8.3 Outbox (`outbox.py`) + +- SQLite `outbox.db`: rows `{cursor (monotonic), chat_id, thread_id, frame_json, + ts, delivered}`. +- **Write** on every outbound frame that has no live subscriber (or always, then + mark delivered on live send — simpler + crash-safe). +- **Retention:** `outbox_retention_hours` (default 72h); prune on write. +- **Cursor:** global monotonic int; `hello.ack` returns the current cursor; + `sync {cursor}` replays rows with `cursor > given`. +- Bounded: if the outbox grows past a cap (e.g. 5k rows), oldest are pruned and + a `notification {kind:"generic", body:"Older messages pruned"}` is sent. + +## 8.4 Sync (reconnect catch-up) + +1. App reconnects → `hello` → `hello.ack {sync_cursor}`. +2. App sends `sync {cursor: }`. +3. Server replays outbox frames (in cursor order) → app applies them (messages, + tools, channels, media offers). +4. Server sends `sync.done {cursor: }`. +5. App updates its local cursor (Room) — idempotent (dedupe by `message_id`). + +This makes the app **eventually consistent** across disconnects, restarts, and +gateway restarts. + +## 8.5 App-side push handling (Android) + +- **`FirebaseMessagingService.onMessageReceived`** (foreground): show in-app + banner + optionally sync. +- **`onNewToken`** → `fcm.register` the new token. +- **Background data message** → start a **foreground service** (low-priority, + notification channel "Sync") → open WS → `sync` → stop. +- **Notification channels** (Android 8+): one per chat so cron channels can have + their own style/priority (e.g. "Cron Reports" = high, "Default" = default). + Tapping a notification deep-links to the chat/thread. +- **ntfy mode:** a foreground service maintains the ntfy subscription; incoming + events trigger the same sync path. + +## 8.6 In-app banners (foreground) + +`notification` frames (e.g. `channel_renamed`, `approval`, `clarify`, `cron`) +render as a transient banner above the composer (like the reference screenshot's +"ARIA hat Thema … umbenannt"). Dismissible; high-priority ones (approval/clarify) +persist until acted on. + +## 8.7 Security + +- FCM tokens are per-device, revocable; stored only in `devices.db`. +- Push payloads carry **no secrets** and no full message bodies larger than a + short preview (privacy on lock screen). Full content is fetched via `sync` + over the authenticated WS. +- ntfy: use a **private topic + auth token** for any real trust boundary (hermes + ntfy adapter guidance). \ No newline at end of file diff --git a/docs/09-pairing-security.md b/docs/09-pairing-security.md new file mode 100644 index 0000000..3a2322c --- /dev/null +++ b/docs/09-pairing-security.md @@ -0,0 +1,95 @@ +# 09 — Pairing, Auth & Security + +## 9.1 Threat model + +- **Trusted domain:** a personal agent on the user's own machine; one owner, a + few of their own devices (phone + desktop). +- **Primary risks:** (a) an unauthorized device connecting to the WS and + reading/driving the agent; (b) eavesdropping on the WS in transit; (c) token + leakage in logs; (d) arbitrary file read via media pull. +- **Not addressing (v1):** multi-tenant isolation, adversarial multi-user abuse, + E2E encryption. + +## 9.2 Pairing flow + +1. **Generate a token.** `hermes gateway setup` (our `interactive_setup`) either + uses an existing `ANDROID_TOKEN` or generates a fresh high-entropy token + (e.g. 32 bytes → 64 hex chars) and stores it in `.env`. +2. **Present to the app.** Two options: + - **QR code:** the setup prints a QR encoding + `iris://pair?host=&port=8790&token=` (or a WSS URL). The + phone scans it with the app's camera (or a system scanner) → pre-fills + settings. + - **Manual:** user types the server URL + token in the app's Connect screen. +3. **App connects.** First WS frame is `hello {token, device_id, device_name, + caps, fcm_token?}`. +4. **Server verifies.** Constant-time compare of `token` vs `ANDROID_TOKEN` + (`hmac.compare_digest`). Optionally check `device_id` against + `ANDROID_ALLOWED_USERS` (if set) or `ANDROID_ALLOW_ALL_USERS`. +5. **On success:** register the device in `devices.db`, send `hello.ack`. + **On failure:** send `error {code:"auth"}` and close. + +`device_id` is a stable, app-generated UUID (persisted in the app's +secure storage). It identifies the device for routing + push, **not** as a +security principal (the token is). + +## 9.3 Auth model + +- **Token = the security principal.** Any connection presenting the valid + `ANDROID_TOKEN` is authorized (it's the user's own token). +- **Allowlist (optional):** `ANDROID_ALLOWED_USERS` (comma-separated + `device_id`s) restricts which *devices* may connect even with the token — + useful if the token is shared. `ANDROID_ALLOW_ALL_USERS=true` disables the + allowlist (dev only). +- **Per-device tokens (stretch):** mint a unique token per device at pairing + (revocable) instead of one shared token. v1 uses the shared token + optional + device allowlist. +- **Re-pairing:** rotating `ANDROID_TOKEN` invalidates all devices; they must + re-pair. `hermes android pair` (stretch CLI) re-issues + prints a new QR. + +## 9.4 Transport security + +- **Default (LAN/dev):** plain `ws://` on the trusted LAN. Fine for a home + network. +- **WSS (recommended for remote):** set `ANDROID_WS_CERT` / `ANDROID_WS_KEY` + (self-signed or CA-signed). The app pins/accepts the cert (self-signed → user + confirms fingerprint on first pair, like a SSH host key). +- **Remote reachability options** (documented, user's choice): + - **Tailscale / WireGuard** (recommended): gateway gets a stable tailnet IP; + app connects over the private mesh. No public exposure. + - **Reverse proxy / tunnel** (Caddy, Cloudflare Tunnel, ngrok): terminate TLS + at the edge, forward WS to `127.0.0.1:8790`. + - **Public bind** (`0.0.0.0`) + WSS + strong token — last resort. +- The app stores the server URL + (for self-signed) the pinned cert fingerprint + in secure storage. + +## 9.5 Secret & PII handling + +- **Tokens/keys never logged.** Redact `ANDROID_TOKEN`, FCM tokens/keys, ntfy + tokens in all log output (hermes PII policy; `agent/redact.py` patterns). +- **`device_id`** is a random UUID (not PII). `device_name` is user-chosen. +- **Media pull** is gated by hermes `validate_media_delivery_path` + delivery + root/recency/denied-path checks (`gateway/platforms/base.py:1684`) — the + plugin can only serve files hermes is allowed to deliver (no arbitrary file + read). +- **Search** is local-only (user's own hermes home); no data leaves the machine. + +## 9.6 Profile safety + +- All plugin state lives under `get_hermes_home()/"android"` (profile-aware). +- Secrets are read with the scope-aware `_get_scoped_secret` pattern (see + `plugins/platforms/irc/adapter.py:42`) so multiplexed profiles don't leak + each other's tokens (fail-closed under `gateway.multiplex_profiles`). +- The WS bind uses a scoped lock (`gateway.status.acquire_scoped_lock`) so two + profiles can't bind the same port/identity. + +## 9.7 Hardening checklist + +- [ ] Constant-time token compare. +- [ ] Bounded per-connection send buffer + rate limit on inbound frames. +- [ ] Reject oversized frames / uploads (`max_upload_bytes`). +- [ ] Verify media sha256 + re-sniff MIME (don't trust client). +- [ ] Redact all secrets in logs. +- [ ] WSS + cert pinning for remote. +- [ ] Outbox retention cap + prune. +- [ ] Fail-closed secret reads under multiplexing. \ No newline at end of file diff --git a/docs/10-android-app.md b/docs/10-android-app.md new file mode 100644 index 0000000..b0c4c97 --- /dev/null +++ b/docs/10-android-app.md @@ -0,0 +1,197 @@ +# 10 — Android App (Kotlin + Jetpack Compose) + +Native client. Lives in the Compose Multiplatform project at `app/`; the bulk of +the code is in `app/shared` (commonMain) so the Desktop app reuses it. + +## 10.1 Tech stack + +| Concern | Choice | +|---|---| +| Language | Kotlin | +| UI | Jetpack Compose (Material 3), Compose Navigation | +| Async | Kotlinx Coroutines + Flow | +| WS client | OkHttp (`WebSocketListener`) | +| JSON | kotlinx-serialization | +| Local DB | **SQLDelight** (KMP; channels, messages cache, media index, sync cursor, settings) | +| Media playback | Media3 **ExoPlayer** (audio + video) | +| Push | Firebase Messaging (FCM) [primary] / ntfy listener [fallback] | +| DI | Hilt | +| Images | Coil | +| minSdk / target | **26** / 34 (test device: MIX 2S, API 29) | + +## 10.2 Module layout + +``` +app/shared/src/ +├── commonMain/kotlin/iris/ +│ ├── protocol/ # frame data classes (mirror of gateway protocol.py) +│ ├── net/ # GatewayClient (OkHttp WS), reconnect, heartbeat, dispatch +│ ├── data/ # ChannelRepository, MessageRepository, MediaRepository, +│ │ # SearchRepository, SettingsRepository (SQLDelight + WS) +│ ├── state/ # ViewModels: ChatListVM, ChatVM, ComposerVM, SettingsVM +│ ├── ui/ +│ │ ├── theme/ # Material 3 theme (dark default), type, color +│ │ ├── components/ # MessageBubble, ReasoningBlock, ToolCard, MediaPlayer, +│ │ │ # ChannelRow, Composer, PickerSheet, SearchBar, Banner +│ │ └── screens/ # ConnectScreen, ChatListScreen, ChatScreen, +│ │ # SearchScreen, SettingsScreen, ChannelMenu +│ └── util/ # time formatting, markdown, id gen +├── androidMain/kotlin/iris/ +│ ├── platform/ # MediaPlayer actual (ExoPlayer), MediaPicker (SAF), +│ │ # Notifications, SecureStore (EncryptedSharedPreferences) +│ ├── fcm/ # FirebaseMessagingService, foreground sync service +│ └── IrisApp.kt # Application (Hilt), notification channels +└── androidApp/ # MainActivity, AndroidManifest, res, google-services +``` + +## 10.3 `GatewayClient` (net) + +- OkHttp `WebSocket` to `ws(s)://host:port/ws`. +- **Reconnect:** exponential backoff + jitter; on reconnect send `hello` then + `sync {cursor}` (replays undelivered outbox only). +- **Initial channel open:** on first open of a chat/thread, send `history` + (newest page) to populate the view. `sync` does NOT load history. +- **Heartbeat:** send `ping` every 20s; reap on missed `pong` (3×) → reconnect. +- **Request/response:** map of `id → CompletableDeferred`; events → a + `SharedFlow` consumed by repositories. +- **Backpressure:** collect frames on a bounded channel; coalesce + `message.update` per `message_id` (keep latest) to avoid flooding the UI. +- **Thread:** all frame emission on a single dispatcher; UI observes via Flow. + +## 10.4 State rules (from hermes desktop AGENTS.md — apply here) + +- **Server is authoritative** for channels/messages/cursor; the app's Room copy + is a **cache**. Reconcile (merge, don't clobber) on `channel.*`/`sync`. +- **Optimistic then honest:** send a message → show it immediately (pending), + roll back visibly on `error`, authoritative `sync` gets the last word. +- **Guard against the past:** generation counters / request tokens so a stale + response never overwrites newer intent. +- **Isolate the foreground:** only the visible chat publishes into the shared + view; background chats update their own cache quietly. +- **Coalesce noise, flush signal:** batch cosmetic updates; let terminal + transitions (turn done, needs input, failed) reach the user immediately. + +## 10.5 Feature implementation (your checklist) + +### Input box, auto-grow (max height) +- Compose `BasicTextField` inside a `Box` with + `Modifier.heightIn(min = 1.line, max = 160.dp)`. Grows with lines, caps at + 160dp, then scrolls internally. +- Bottom bar (matches reference): `Menü` button (left), emoji button, the + auto-grow field, attach (paperclip), mic/send (right). Send on Enter + (configurable: Enter=send vs Enter=newline). + +### Menu button → all slash commands +- `Menü` opens a **bottom sheet** listing the command catalog. The catalog is + served by the gateway via `commands.catalog` request → response with + `{name, description, args_hint, category}` per command, so it always matches + hermes (`/new`, `/model`, `/reasoning`, `/status`, `/cron`, `/tools`, …). +- Typing `/` in the field shows **autocomplete** via `commands.complete` + request (gateway matches the typed prefix). Selecting inserts `/cmd `. +- Selecting a command sends `message.send {text:"/cmd args"}`. +- **Interactive commands** (`/model`, `/reasoning`, `/fast`, approvals, + clarifies) render as **native pickers** from `picker.*` frames (a dialog / + sheet with the options; answer via `picker.select`). + +### Tool output (app-controlled verbosity) +- `ToolCard` renders `tool.start/progress/end` frames. +- **Settings → "Tool detail": Everything / Truncated / Nothing.** + - Everything: name + full args (collapsible) + output preview. + - Truncated (default): `emoji name: "short preview"` one-liner, expandable. + - Nothing: suppress tool frames. +- Spinner while running; ✓/✗ + duration on `tool.end`. + +### Reasoning before message +- `ReasoningBlock` (collapsible, "💭 Reasoning" header, monospace body, **copy** + button) rendered **above** the message body from the `reasoning` field. + Matches the reference screenshot. + +### Intermediate messages +- `commentary` frames → dimmed/smaller bubble, distinct from final answers. + +### Agent busy / stop / steer +- `agent.busy` → show "thinking…" indicator in chat header (animated dots). +- `agent.idle` → clear indicator. +- **Stop button** (appears in header while busy): sends `agent.stop`. +- **Steer:** while busy, the composer accepts input; sending it calls + `agent.steer` (injects mid-turn) rather than queuing a new `message.send`. + +### Threading + channels +- **Channel list** (drawer on single-pane; left rail on two-pane) = default chat + + user channels (avatar, name, last-message preview, unread badge, active + highlight bar) — matches the reference left sidebar. +- **Thread toggle** in the default chat header: "Threads on/off". On → topic + switcher above the message list (each topic = a `thread_id`). +- **New channel** (FAB / channel-list menu) → `channel.create` → appears in list; + menu offers "Set as cron target". + +### Search +- Search bar (chat header or top) with a **scope toggle**: "Search everywhere" / + "Search in this chat/channel". → `search` frame → results list → tap jumps to + the message (navigate + highlight). + +### Attach media +- Paperclip → system pickers (Photos / Files / Audio / Video / Docs) via SAF. +- Selected files show as **preview chips** in the composer (thumbnail + name + + remove). On send: `media.upload` (chunked) for each, then `message.send` with + `media_refs`. + +### Voice input (mic button) +- Mic button (right of composer, toggles to send when text is present). +- Tap → request `RECORD_AUDIO` permission → start recording (MediaRecorder, + OGG/Opus, 44.1 kHz mono). +- While recording: timer + waveform; tap again to stop. +- On stop: file becomes a **preview chip** (audio, kind=`voice`) in the + composer, same as attached media. Send → `media.upload` (chunked) → + `message.send` with `media_refs`. +- Hermes receives it as an audio attachment; the agent's STT (if configured) + transcribes it. No client-side STT. + +### Push notifications +- FCM service (see `08-push.md`): foreground banner + background foreground + service → `sync`. Notification channel per chat. Tap → deep-link to chat. +- ntfy fallback: foreground service maintains the subscription. + +### Live playback +- AI-sent audio/video → `media.pull` → cache file → **ExoPlayer** inline player + (audio: mini-player; video: inline + fullscreen + PiP). Documents/images → + viewer / open-with. + +## 10.6 Layout (Telegram-style, per reference image) + +**Two layout modes** (decision: user-toggleable, **single-pane default**): +- **Single-pane (default on phones):** chat full-screen; channel list in a + swipeable drawer (hamburger / edge swipe). +- **Two-pane (Telegram-style, like the reference):** persistent left channel + rail + chat. Auto-enabled on tablets / large screens; toggleable in Settings. + +**Chat screen anatomy (matches reference):** +- **Header:** back (single-pane), avatar, name + "Bot" subtitle, edit + overflow + (⋮) menu (thread toggle, channel menu, set cron target, clear). +- **Message list:** date separators ("7. August"); user bubbles **right** + (accent color, ✓✓ read receipts); agent bubbles **left** (surface color) with + optional ReasoningBlock + ToolCards + media + model/token footer + ("Qwen3-… · 11% · ~") + timestamp. +- **In-app banner** above the composer (e.g. "renamed topic"). +- **Composer:** `Menü` / emoji / auto-grow input / attach / mic-send. + +**Theme:** dark by default (reference is dark); Material 3; optional dynamic +color. Accent = user's chosen brand color (default indigo, like the reference). + +## 10.7 SQLDelight schema (cache) + +- `channels(chat_id PK, name, kind, parent_chat_id, is_default, last_preview, + last_ts, unread)`. +- `messages(id PK, chat_id, thread_id, role, text, reasoning, model, tokens, + ts, status[pending|sent|read], media_json)`. +- `media(media_id PK, local_path, kind, mime, size, ts)`. +- `meta(key PK, value)` — sync cursor, settings, device_id, server url, pinned + cert fingerprint. + +## 10.8 Onboarding / Connect screen + +- First launch → **Connect**: server URL + token (or scan QR). "Test connection" + does a real `hello` (not just a TCP probe — per hermes desktop guidance, the + auth leg must be exercised). On success → save (secure storage) → main. +- States: connecting / connected / reconnecting / degraded / auth-failed — each + with honest copy and a way out. \ No newline at end of file diff --git a/docs/11-desktop-app.md b/docs/11-desktop-app.md new file mode 100644 index 0000000..f74a64b --- /dev/null +++ b/docs/11-desktop-app.md @@ -0,0 +1,75 @@ +# 11 — Desktop App (Kotlin + Compose Multiplatform) + +The desktop app is **the Android app, tweaked for a big screen**. It reuses the +entire `app/shared` module (protocol, network, repositories, state, most UI) and +only adds desktop platform services + a wider default layout. + +## 11.1 What's shared vs desktop-specific + +| Layer | Shared (commonMain) | Desktop-specific (desktopMain) | +|---|---|---| +| Protocol / WS client | ✅ | — | +| Repositories / state | ✅ | — | +| Most Compose UI | ✅ | layout tweaks, keyboard shortcuts | +| Push | — | **tray icon + OS notifications** (no FCM) | +| Media playback | `MediaPlayer` interface | **desktop player** actual | +| File picker | `MediaPicker` interface | **desktop file dialog** actual | +| Window | — | resizable window, always-on-top, global hotkey | +| Secure store | `SecureStore` interface | keyring / encrypted file actual | + +## 11.2 `desktopMain` platform services + +- **Push → tray + OS notifications.** Desktop is assumed reachable (persistent + WS), so no FCM. A **system tray icon** shows connection state + unread count; + OS notifications (Java Desktop / `SystemTray` + a cross-platform notifier) + fire for background chats / approvals / cron. Clicking focuses the window and + deep-links to the chat. +- **Media playback → `MediaPlayer` actual.** ExoPlayer is Android-only. Desktop + uses a `libmpv`/`mpv`-backed Compose surface for video (and audio), with a + WebView-based fallback if `mpv` isn't available. Same `MediaPlayer` interface + the Android `ExoPlayer` actual implements, so UI code is identical. +- **File picker → `MediaPicker` actual.** A native file dialog (Compose + Desktop `FileChooser` / AWT `JFileChooser`) returning local file paths, then + the same `media.upload` chunked path. +- **Window.** Resizable, remembers size/position, optional always-on-top, a + global hotkey to focus/show. Minimize-to-tray option. +- **Secure store → `SecureStore` actual.** OS keychain (macOS Keychain, Linux + Secret Service / encrypted file, Windows Credential Manager) or an encrypted + file for the token + pinned cert. + +## 11.3 Layout (big-screen tweaks) + +- **Two-pane is the default** (persistent left channel rail + chat), since there + is width. Single-pane still available via the same toggle. +- **Wider chat column** with comfortable max line length; optional **third + pane** (inspector: session info, model, tokens, tool detail, channel settings). +- **Larger type scale** and spacing for desktop; hover states; mouse + keyboard + first. +- **Keyboard shortcuts:** + - `/` focus the composer and open the command menu. + - `Ctrl/Cmd+K` command palette (all slash commands + actions). + - `Ctrl/Cmd+N` new channel; `Ctrl/Cmd+T` new thread. + - `Ctrl/Cmd+F` search (with the all/this-chat scope toggle). + - `↑/↓` navigate channel list; `Enter` open. + - `Esc` close sheet/dialog (one cancel gesture does exactly one thing). +- **Multi-window (stretch):** open a channel in a separate window / pop-out. + +## 11.4 Distribution + +- **Compose Desktop** → native binaries via **jpackage** (or a plain app image): + Linux (.deb/.AppImage), macOS (.dmg), Windows (.msi/.exe). +- Bundles the JRE. Auto-update is a stretch goal (v1: manual download). +- The desktop app connects to the **same** gateway WS server as the phone (the + user's home server / Tailscale). It does **not** spawn its own backend (unlike + hermes's existing Electron desktop, which spawns `hermes serve`) — our + desktop is a pure client of the messaging gateway, matching the Android app. + +## 11.5 Parity checklist (same functionality as Android) + +- [ ] Streaming, reasoning block, tool cards (3-level verbosity), commentary. +- [ ] Channels + threads + new channel + cron target. +- [ ] Search (all / this-chat). +- [ ] Media attach + live playback (via desktop player). +- [ ] Slash command menu + autocomplete + interactive pickers. +- [ ] Push (tray + OS notifications) + outbox/sync. +- [ ] Pairing/Connect screen (URL + token, WSS cert pin). \ No newline at end of file diff --git a/docs/12-toolchain.md b/docs/12-toolchain.md new file mode 100644 index 0000000..d2712ca --- /dev/null +++ b/docs/12-toolchain.md @@ -0,0 +1,138 @@ +# 12 — Toolchain Setup + +First-time setup on a machine (verified baseline: CachyOS/Arch, `pacman`, +`uv` present, ADB present, no JDK/SDK/Gradle). + +## 12.1 JDK 17 + +```bash +pacman -S jdk17-openjdk +java -version # expect 17.x +``` +(Compose Multiplatform + current AGP are happy on JDK 17. Use 17 to match the +Android toolchain; 21 also works but 17 is the safe floor.) + +## 12.2 Android SDK + +```bash +# cmdline-tools +mkdir -p ~/android-sdk/cmdline-tools +cd ~/android-sdk/cmdline-tools +curl -O https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip +unzip commandlinetools-linux-*.zip && mv cmdline-tools latest +rm commandlinetools-linux-*.zip + +export ANDROID_HOME=$HOME/android-sdk +export PATH=$PATH:$ANDROID_HOME/cmdline-tools/latest/bin:$ANDROID_HOME/platform-tools + +sdkmanager --licenses +sdkmanager "platform-tools" "platforms;android-34" "build-tools;34.0.0" +``` +Persist `ANDROID_HOME`/`PATH` in `~/.bashrc`. ADB is already installed system-wide; +`platform-tools` from the SDK is fine too (whichever is first on `PATH`). + +Create `app/local.properties`: +``` +sdk.dir=/home//android-sdk +``` + +## 12.3 Gradle + +No system install — use the project wrapper: +```bash +cd app +./gradlew tasks # first run downloads the wrapper distribution +``` +(The wrapper version is pinned in `app/gradle/wrapper/gradle-wrapper.properties`.) + +## 12.4 hermes environment (for the plugin + running the gateway) + +```bash +cd hermes-agent +uv sync # creates .venv with all core deps (websockets, httpx, …) +source .venv/bin/activate +hermes --version # sanity +``` +- Run the gateway with the plugin: + ```bash + # install the plugin (dev: symlink) + mkdir -p ~/.hermes/plugins + ln -s "$PWD/../gateway-plugin" ~/.hermes/plugins/android + hermes gateway status # should list "android" + hermes gateway # run + ``` +- Tests use hermes's hermetic runner (never bare `pytest`): + ```bash + scripts/run_tests.sh tests/gateway/test_android.py + ``` + +## 12.5 Firebase (FCM) — primary push + +1. Create a Firebase project (console.firebase.google.com). +2. Add an **Android app** (package = `androidApp` applicationId, e.g. + `dev.iris.app`). Download `google-services.json` → `app/androidApp/`. +3. Create a **service account** (Project settings → Service accounts → Generate + new private key) → download the JSON. Store its path in + `ANDROID_FCM_SERVICE_ACCOUNT` (in `~/.hermes/.env`). +4. The app's `FirebaseMessagingService` obtains the FCM token at runtime and + registers it via `hello` / `fcm.register`. + +> Skip Firebase → set `ANDROID_PUSH_BACKEND=ntfy` and configure `NTFY_TOPIC` / +> `NTFY_SERVER_URL` (self-host ntfy or use ntfy.sh). See `08-push.md`. + +## 12.6 Environment variables (summary) + +**Secrets (`~/.hermes/.env`):** +``` +ANDROID_TOKEN=<64-hex> +ANDROID_PUSH_BACKEND=fcm # or ntfy +ANDROID_FCM_SERVICE_ACCOUNT=/path/to/service-account.json +# ANDROID_FCM_SERVER_KEY= # fallback if no service account +# NTFY_TOPIC=iris-push # when ntfy +# NTFY_SERVER_URL=https://ntfy.sh +# ANDROID_WS_CERT=/path/cert.pem # WSS +# ANDROID_WS_KEY=/path/key.pem +``` + +**Behavioral (`~/.hermes/config.yaml`):** +```yaml +gateway: + platforms: + android: + enabled: true + extra: + host: 127.0.0.1 # 0.0.0.0 for LAN + port: 8790 + home_channel: android:default + push_backend: fcm + outbox_retention_hours: 72 + max_upload_bytes: 104857600 # 100 MB +display: + platforms: + android: + show_reasoning: true + reasoning_style: code + streaming: true + tool_progress: all # gateway sends full data; app controls display +``` + +## 12.7 Verify the stack (smoke test) + +```bash +# 1. gateway up with plugin +hermes gateway status | grep -i android + +# 2. a raw WS client can pair + echo +python - <<'PY' +import asyncio, json, websockets +async def main(): + async with websockets.connect("ws://127.0.0.1:8790/ws") as ws: + await ws.send(json.dumps({"v":1,"type":"hello","payload":{ + "token":"","device_id":"test","device_name":"probe", + "caps":{"min_protocol":1}}})) + print("recv:", await ws.recv()) +asyncio.run(main()) +PY +``` +Expect a `hello.ack`. If you get `error {code:"auth"}`, the token/host/port is +wrong. \ No newline at end of file diff --git a/docs/13-testing.md b/docs/13-testing.md new file mode 100644 index 0000000..fd85be4 --- /dev/null +++ b/docs/13-testing.md @@ -0,0 +1,131 @@ +# 13 — Testing & Debugging (incl. ADB on-device) + +Three layers: **Python plugin tests**, **Kotlin unit/UI tests**, and **on-device +E2E via ADB**. Plus a **WS test-client harness** to drive the real gateway +without the app (critical for verifying frame shapes early). + +## 13.1 Python plugin tests + +- Location: `gateway-plugin/tests/` (and, for hermes-integration tests, mirror + into the hermes `tests/gateway/test_android.py` pattern when running under + hermes's suite). +- **Run with hermes's hermetic runner** (never bare `pytest`): + ```bash + cd hermes-agent + scripts/run_tests.sh tests/gateway/test_android.py + scripts/run_tests.sh # full suite (CI parity) + ``` +- Coverage to write (behavioral, not change-detector — per hermes test policy): + - `register(ctx)` produces a valid `PlatformEntry` (name, cron env var, + parse_target_ref). + - `check_requirements` / `validate_config` / `is_connected` truth table. + - `_parse_target_ref`: `android:`, `android::`, non-android + → None. + - **Reasoning split:** given a `show_reasoning`-style final text, `send()` + emits `message {reasoning, text}` correctly; no-prefix → no reasoning field. + - **Outbox:** frame with no subscriber → written with a cursor; `sync` replays + the right range; retention prunes. + - **Pairing:** valid token → `hello.ack`; wrong token → `error auth` + close. + Constant-time compare used. + - **Media:** upload start→binary→end reassembles + sha256 verified; over-limit + → `media_too_large`; pull serves only allowed paths. + - **Push backend selection:** fcm vs ntfy chosen by config; `configured()` + reflects missing creds. + - **Channel directory:** create/rename/set_default; cron target resolution. +- **No `~/.hermes` writes in tests** — use the `_isolate_hermes_home` fixture + pattern (temp `HERMES_HOME`). Profile tests also mock `Path.home()`. + +## 13.2 WS test-client harness (do this FIRST, in M1/M2) + +A small Python script (`gateway-plugin/tests/ws_probe.py`) that connects to the +**real running gateway** and drives a turn, printing every frame. This is how we +**empirically confirm** the exact frame shapes (especially tool-progress vs +commentary classification and the reasoning prefix) before/while building the +Kotlin client. + +```bash +hermes gateway & # with the android plugin +python gateway-plugin/tests/ws_probe.py --token \ + --send "list the files and summarize" +# prints: hello.ack, typing, message.start, message.update…, tool.start, tool.end, +# commentary, message.stop {reasoning,…}, … +``` +Use it to lock `04-wire-protocol.md` against reality and to debug the adapter +without waiting for the app. + +## 13.3 Kotlin unit / UI tests + +- **Protocol codec:** round-trip every frame type (serialize → deserialize → + equal); unknown `type`/fields ignored (forward-compat). +- **Repositories:** merge-don't-clobber on `channel.*`; optimistic send + rollback + on `error`; sync dedupe by `message_id`; cursor monotonic. +- **`GatewayClient`:** reconnect/backoff; `message.update` coalescing (latest + wins); request/response correlation by `id`. +- **ViewModels:** state transitions (connecting→connected→reconnecting); tool + verbosity filtering (everything/truncated/nothing); reasoning present/absent. +- **Compose UI tests:** ReasoningBlock collapse/expand + copy; ToolCard + spinner→done; composer auto-grow cap; channel list active highlight. +- Run: `./gradlew :shared:testDebugUnitTest` (and `:shared:testDesktopTest`). + +## 13.4 On-device E2E via ADB (the MIX 2S, API 29) + +Device: `a5ca2a4b` (Xiaomi MIX 2S). Workflow: + +```bash +# install + launch +cd app +./gradlew :androidApp:installDebug +adb shell am start -n dev.iris.app/.MainActivity + +# logs (filter our tags + WS + FCM + ExoPlayer) +adb logcat -c +adb logcat | grep -Ei "iris|GatewayClient|Firebase|ExoPlayer|MediaCodec" + +# screenshots for visual checks +adb exec-out screencap -p > /tmp/shot.png + +# clear app data between pairing attempts +adb shell pm clear dev.iris.app + +# push a file to the app's cache (for media tests) / pull logs +adb shell run-as dev.iris.app ls files +adb logcat -d > /tmp/logcat.txt +``` + +**E2E scenarios (script where possible):** +1. **Pair:** connect screen → enter URL+token → `hello.ack` → main. (Verify auth + leg, not just TCP.) +2. **Text round-trip:** send "hello" → streamed reply appears (message.start → + updates → stop). +3. **Reasoning:** ask a reasoning-model question → ReasoningBlock shows above the + answer; copy button works. +4. **Tools:** trigger a tool (e.g. "list files") → ToolCard shows; toggle + verbosity in Settings → rendering changes. +5. **Intermediate:** a multi-step prompt → commentary bubble appears dimmed. +6. **Channels:** create "Cron Reports" → appears in list; set as cron target. +7. **Cron delivery:** create a cron job `deliver=android:android:chan_` → it + fires → lands in that channel (not default). +8. **Search:** "search everywhere" vs "this chat" → correct scoping; tap → jump. +9. **Media (in):** attach a photo + a video → agent receives (vision) → reply. +10. **Media (out):** ask the agent to send an image/video → `media.offer` → + inline player plays it live. +11. **Push:** background the app (`adb shell am start` another app / lock) → + trigger a message → FCM notification appears → tap → syncs + deep-links. +12. **Reconnect/sync:** kill the WS (stop gateway briefly) → restart → app + reconnects → `sync` catches up (no lost/dup messages). + +## 13.5 Debugging tips + +- **Gateway side:** `~/.hermes/logs/gateway.log` (and `hermes logs --follow`). + Our plugin logs under the `android` adapter name; secrets redacted. +- **WS framing bugs:** use the `ws_probe.py` harness — it isolates the protocol + from the app. +- **Streaming jitter:** the consumer edits at intervals; if updates look chunky, + check `display.platforms.android.streaming` and the consumer's edit interval. +- **Media pull stalls:** check chunk size + backpressure; confirm the file is + within hermes delivery roots (`validate_media_delivery_path`). +- **FCM not arriving:** confirm the token registered (`devices.db`), the service + account can mint a token, and the app's `onNewToken` re-registered after + `pm clear`. +- **Profile leaks:** if tokens look wrong under multiple profiles, verify the + scope-aware secret read (`_get_scoped_secret`) is used. \ No newline at end of file diff --git a/docs/14-milestones.md b/docs/14-milestones.md new file mode 100644 index 0000000..58a67ad --- /dev/null +++ b/docs/14-milestones.md @@ -0,0 +1,138 @@ +# 14 — Milestones (M0–M7) + +Phased delivery. Each milestone ends with a **demo** (on-device where noted) and +has explicit **acceptance criteria**. Work top-to-bottom; don't skip M0/M1. + +--- + +## M0 — Toolchain & scaffolding +**Goal:** everything builds; the plugin is discoverable; the repo is safe. +- [ ] Install JDK 17, Android SDK, set `ANDROID_HOME` (`12-toolchain.md`). +- [ ] `cd hermes-agent && uv sync` (hermes venv works). +- [ ] Create monorepo scaffold (`02-monorepo.md`): `gateway-plugin/`, `app/` + (CMP: `shared`, `androidApp`, `desktopApp`), root `.gitignore` + (**excludes `hermes-agent/`**), root `README.md`. +- [ ] `git init` + a pre-commit/CI guard that fails if `hermes-agent/` is staged. +- [ ] CMP project builds empty: `./gradlew :androidApp:assembleDebug`, + `./gradlew :desktopApp:run` (blank window). +- [ ] Plugin skeleton: `plugin.yaml` + `adapter.py` with `register(ctx)` + a + no-op `AndroidAdapter` → `hermes gateway status` lists **android**. +- **Demo:** `hermes gateway status` shows `android`; `./gradlew + :androidApp:installDebug` installs a blank app on the MIX 2S. +- **Accept:** blank app installs + launches on-device; plugin visible in + `hermes gateway status`; `hermes-agent/` is git-ignored (verify with + `git status --ignored`). + +## M1 — Gateway core loop (text round-trip) +**Goal:** pair + send a text message + get a (non-streaming) reply. +- [ ] WS server (`ws_server.py`): bind, `hello` auth (constant-time), + `hello.ack`, heartbeat, connection registry. +- [ ] `AndroidAdapter.send()` → `message` frame; inbound `message.send` → + `MessageEvent` → `handle_message`. +- [ ] Pairing store + `ANDROID_TOKEN`; QR payload in `interactive_setup`. +- [ ] App: Connect screen (URL+token, real `hello` test), `GatewayClient` + (connect + reconnect), ChatScreen sends + renders `message`. +- [ ] `ws_probe.py` harness drives a real turn. +- **Demo (on-device):** pair the phone, send "hello", see the agent's reply. +- **Accept:** text round-trip works on-device; wrong token is rejected; + reconnect after gateway restart re-pairs. + +## M2 — Streaming + reasoning + tools + commentary +**Goal:** the "agent transparency" features. +- [ ] Map consumer `send`/`edit_message` → `message.start/update/stop`. +- [ ] Reasoning: set `show_reasoning` for android; adapter splits prefix → + `reasoning` field. **Verify format with `ws_probe.py`.** +- [ ] Tool events: classify tool-progress `send()`s → structured + `tool.start/progress/end` (turn-state machine). **Verify with probe.** +- [ ] Commentary → `commentary` frames. Typing → `typing`. +- [ ] App: live bubble (coalesced updates), `ReasoningBlock` (collapse + copy), + `ToolCard` with **Everything/Truncated/Nothing** setting, dimmed + `commentary` bubble. +- **Demo (on-device):** a multi-step prompt streams, shows reasoning above the + answer, tool cards (toggle verbosity), and an intermediate beat. +- **Accept:** all four render correctly; tool verbosity setting changes + rendering; reasoning copy button works; frame shapes match `04-wire-protocol`. + +## M3 — Channels/threads + cron + search +**Goal:** organization + cron delegation + search. +- [ ] Channel directory (SQLite): default channel ensured; `channel.create/ + rename/set_default/delete` + `channel.*` frames. +- [ ] Threads: toggle in default chat; `thread_id` lanes; `create_handoff_thread`. +- [ ] `parse_target_ref_fn` + `cron_deliver_env_var` → cron + `deliver=android:[:]` works. +- [ ] `search.py` FTS5 bridge; `search` frame (all / this-chat) → results. +- [ ] App: channel list (drawer/rail), thread toggle + topic switcher, "new + channel" + "set as cron target", SearchScreen with scope toggle + jump. +- **Demo (on-device):** create "Cron Reports"; create a cron job delivering to + it; it fires into that channel; search finds a message (both scopes). +- **Accept:** cron output lands in the chosen channel (not default); threads + isolate context; search scoping correct; channel list reconciles on events. + +## M4 — Media +**Goal:** attach + receive + play media. +- [ ] Inbound: `media.upload` chunked → `cache_*_from_bytes` → `media_urls`; + size limit + sha256 + MIME re-sniff. +- [ ] Outbound: `send_*` → `media.offer`; `media.pull` chunked; delivery-path + security. +- [ ] App: SAF pickers + preview chips + upload; `media.pull` → cache; + **ExoPlayer** inline (audio mini-player, video fullscreen/PiP); image/doc + viewers. +- **Demo (on-device):** attach a photo + video (agent sees them); ask agent to + send an image/video → plays live in-app. +- **Accept:** both directions work; over-limit rejected; playback is live; + only allowed files are servable. + +## M5 — Push + offline (FCM + ntfy) +**Goal:** reach the phone when backgrounded; catch up on reconnect. +- [ ] Outbox (SQLite) + sync cursor; `sync`/`sync.done`; retention prune. +- [ ] `push.py`: `FcmBackend` (HTTP v1 + service account, httpx) + + `NtfyBackend`; selected by `ANDROID_PUSH_BACKEND`. +- [ ] Fire push on no-live-subscriber; data payload for silent sync. +- [ ] App: FCM service (foreground banner + background foreground-service sync), + `onNewToken` → `fcm.register`; per-chat notification channels; deep-link. + ntfy listener fallback. +- [ ] In-app `notification` banners (channel_renamed, approval, cron, …). +- **Demo (on-device):** background the app → trigger a message → notification + appears → tap → syncs + opens the chat. Repeat with ntfy backend. +- **Accept:** push arrives when backgrounded (both backends); reconnect syncs + with no loss/dup; banners show for foreground events. + +## M6 — Desktop app +**Goal:** the same app on a big screen. +- [ ] `desktopMain`: tray + OS notifications; `MediaPlayer` actual (mpv/WebView); + `MediaPicker` actual (file dialog); `SecureStore` actual; window mgmt. +- [ ] Two-pane default layout; keyboard shortcuts; optional inspector pane. +- [ ] Parity pass vs Android feature checklist (`11-desktop-app.md`). +- [ ] jpackage builds (Linux first; macOS/Windows as available). +- **Demo:** desktop app pairs to the same gateway; full feature parity; tray + notifications; media plays. +- **Accept:** all Android features work on desktop; tray + shortcuts work; + native binary launches. + +## M7 — Polish + E2E + docs +**Goal:** ship-quality. +- [ ] Telegram-style layout pass (per reference image): header, bubbles, date + separators, ✓✓, model/token footer, banner, bottom bar. +- [ ] Theming (dark default, accent), onboarding/pairing UX, empty/loading/ + reconnecting/degraded states with honest copy. +- [ ] Full E2E suite (`13-testing.md` scenarios 1–12) automated where possible. +- [ ] Docs: `docs/protocol/frames.schema.json` finalized; `docs/setup.md` + (user-facing pairing + FCM/ntfy + remote access); root README. +- [ ] Security hardening checklist (`09-pairing-security.md`) verified. +- **Demo:** end-to-end on phone + desktop simultaneously; cron into a channel; + push; media; search. +- **Accept:** all feature-checklist items pass on-device; E2E green; docs + complete; `hermes-agent/` still never committed. + +--- + +## Sequencing notes +- **M1/M2 depend on the `ws_probe.py` harness** to lock frame shapes early — + build it in M1. +- **M3 (cron) and M5 (push) both touch the outbox** — build the outbox in M3, + extend for push in M5. +- **M6 (desktop) reuses M1–M5 shared code** — do it after the Android features + are stable so the shared module is settled. +- Parallelizable: plugin (Python) and app (Kotlin) can be worked on + concurrently once the protocol (`04-wire-protocol.md`) is agreed; the probe + harness is the integration seam. \ No newline at end of file diff --git a/docs/15-hermes-reference.md b/docs/15-hermes-reference.md new file mode 100644 index 0000000..554f5f6 --- /dev/null +++ b/docs/15-hermes-reference.md @@ -0,0 +1,136 @@ +# 15 — hermes-agent Source Reference Map + +A cheat-sheet of the **exact hermes-agent files/lines** to read for each +integration point. Paths are relative to `hermes-agent/` (the read-only +reference). This lets a coder jump straight to the right code instead of +re-deriving the architecture. + +> ⚠️ Read-only. We **install** our plugin into `~/.hermes/plugins/android`; we +> never edit these files. + +## Plugin / platform registration +| What | Where | +|---|---| +| How to add a platform (Plugin Path) | `gateway/platforms/ADDING_A_PLATFORM.md` | +| Canonical plugin-platform example | `plugins/platforms/irc/adapter.py` (esp. `register(ctx)` at :953, `_env_enablement` :677, `_standalone_send` :743, `_get_scoped_secret` :42) | +| ntfy plugin example (push-ish) | `plugins/platforms/ntfy/adapter.py`, `plugin.yaml` | +| `register_platform()` (PluginContext) | `hermes_cli/plugins.py:2774` | +| `PlatformEntry` dataclass (all fields) | `gateway/platform_registry.py:63` | +| `Platform` enum | `gateway/config.py` | +| Plugin discovery (`PluginManager`) | `hermes_cli/plugins.py` | + +## Base adapter contract +| What | Where | +|---|---| +| `BasePlatformAdapter` (ABC) | `gateway/platforms/base.py:2890` | +| `MessageEvent` (inbound) | `gateway/platforms/base.py:2300` | +| `MessageType` | `gateway/platforms/base.py:2278` | +| `SendResult` | `gateway/platforms/base.py:2466` | +| `build_source(...)` (SessionSource) | `gateway/platforms/base.py:7047` | +| `handle_message(event)` | `gateway/platforms/base.py:5981` | +| `send()` / `edit_message()` / `delete_message()` | `base.py:3920 / 3976 / 4005` | +| `send_typing` / `stop_typing` | `base.py:4298 / 4307` | +| Media send: `send_image/video/document/voice/animation/image_file/multiple_images` | `base.py:4396 / 4632 / 4659 / 4486 / 4415 / 4339 / 7834(tg)` | +| `extract_images` / `extract_media` / `extract_local_files` | `base.py:4439 / 4884 / 5020` | +| Media cache helpers: `cache_image/audio/video/document_from_bytes` | `base.py:854 / 1005 / 1122 / 2126` | +| Inbound media size: `get_inbound_media_max_bytes` / `validate_inbound_media_size` | `base.py:758 / 779` | +| Media delivery security: `validate_media_delivery_path` + roots/recency/denied | `base.py:1684 / 1312-1480` | +| Interactive: `send_slash_confirm` / `send_clarify` / `send_private_notice` | `base.py:4169 / 4204 / 4278` | +| `create_handoff_thread` | `base.py:3949` | +| Streaming hooks: `supports_draft_streaming` / `send_draft` / `render_message_event` / `format_tool_event` | `base.py:3215 / 3274 / 3316 / 3337` | +| Scoped secret read pattern (profile-safe) | `plugins/platforms/irc/adapter.py:42` | +| Scoped lock (profile-safe bind) | `gateway/status.py` (`acquire_scoped_lock`) | + +## Streaming (legacy callback path — what the main gateway uses) +| What | Where | +|---|---| +| `GatewayStreamConsumer` (the sink) | `gateway/stream_consumer.py:156` | +| `on_delta` / `on_commentary` / `on_segment_break` / `finish` | `stream_consumer.py:611 / 518 / 514 / 623` | +| Consumer `run()` loop (edit cadence) | `stream_consumer.py:781` | +| Structured events (ACP-only, NOT main gateway) | `gateway/stream_events.py`, `gateway/stream_dispatch.py:40` | +| Callback wiring (agent → consumer) | `gateway/run.py:5642-5704` (`tool_progress_callback`, `stream_delta_callback`, `interim_assistant_callback`, `status_callback`, `event_callback`) | +| `send_progress_messages` (tool progress → `send`) | `gateway/run.py:4603` | +| Progress metadata | `gateway/run.py:28089` | + +## Reasoning display +| What | Where | +|---|---| +| Reasoning prepended to final response | `gateway/run.py:20089-20127` | +| `show_reasoning` / `reasoning_style` defaults + per-platform | `gateway/display_config.py:33-181` | +| `resolve_display_setting()` | `gateway/display_config.py:187` | +| `last_reasoning` in agent result | `gateway/run.py:6471` | +| Think-tag filtering in consumer | `stream_consumer.py:175-185, 627+` | + +## Display settings (tool progress, interim, streaming, etc.) +| What | Where | +|---|---| +| All overrideable display keys + defaults | `gateway/display_config.py:33` (`tool_progress`, `show_reasoning`, `reasoning_style`, `tool_preview_length`, `streaming`, `interim_assistant_messages`, `long_running_notifications`, `cleanup_progress`, `live_status`) | +| Per-platform tiers | `gateway/display_config.py:81-181` | + +## Slash commands +| What | Where | +|---|---| +| `COMMAND_REGISTRY` / `CommandDef` (all commands) | `hermes_cli/commands.py:144+` | +| `GATEWAY_KNOWN_COMMANDS` / `is_gateway_known_command` / `resolve_command` | `hermes_cli/commands.py` | +| Gateway command dispatch (alias, access, hooks) | `gateway/run.py:16974-17099` | +| `send_model_picker` / `send_choice_picker` (Telegram ref) | `plugins/platforms/telegram/adapter.py:6350 / 6424` | +| Picker invocation from gateway | `gateway/slash_commands.py:1859, 2157, 3622-3657` | +| Button-callback id conventions (`cl:`, `appr:`, `sc:`) | `gateway/platforms/ADDING_A_PLATFORM.md` (Interactive UX) | + +## Cron delivery +| What | Where | +|---|---| +| Resolve a delivery target (`platform:chat_id[:thread_id]`) | `cron/scheduler.py:2148` (`_resolve_single_delivery_target`) | +| `_resolve_delivery_targets` / routing tokens (`all`) | `cron/scheduler.py:2296 / 2278` | +| `cron_deliver_env_var` handling (home channel) | `cron/scheduler.py:1903+` | +| `deliver` param normalization | `cron/scheduler.py:2251` | +| `send_message` tool target resolution (`resolve_send_target`, `prepare_send_message_platforms`) | `tools/send_message_tool.py` | +| `parse_target_ref_fn` usage | `tools/send_message_tool.py` (`_parse_target_ref`) | +| Cron mirror delivery (default off) | `cron/scheduler.py:1520` | +| `cronjob` tool schema (`deliver` description) | `tools/cronjob_tools.py` | + +## Search (FTS5) +| What | Where | +|---|---| +| Session store (SQLite + FTS5) | `hermes_state.py` | +| Search implementation | `hermes_state_search.py` | +| Schema | `hermes_state_schema.py` | + +## Config / env / profiles +| What | Where | +|---|---| +| `get_hermes_home()` / `display_hermes_home()` (profile-safe paths) | `hermes_constants.py` | +| `DEFAULT_CONFIG` / `OPTIONAL_ENV_VARS` | `hermes_cli/config.py` | +| Gateway config load (`load_gateway_config`, `_apply_env_overrides`) | `gateway/config.py` | +| Profile override (`_apply_profile_override`) | `hermes_cli/main.py` | +| Secret scope (multiplex fail-closed) | `agent/secret_scope.py` | +| PII redaction | `agent/redact.py` | + +## Dependencies (confirm zero new deps) +| What | Where | +|---|---| +| `websockets==15.0.1` (core) | `pyproject.toml:111` | +| `httpx[socks]==0.28.1` (core) | `pyproject.toml:44` | +| `aiohttp` (messaging extra, NOT core) | `pyproject.toml:185` | +| Dependency pinning policy | `pyproject.toml:19-39`, root `AGENTS.md` | + +## Testing +| What | Where | +|---|---| +| Hermetic test runner (use this, not bare pytest) | `scripts/run_tests.sh` | +| `_isolate_hermes_home` fixture | `tests/conftest.py` | +| Example platform tests | `tests/gateway/test_*.py` (e.g. `test_google_chat.py`, `test_line_plugin.py`) | +| Stream-event tests (dispatcher) | `tests/gateway/test_stream_events.py` | + +## Existing desktop app (for reference only — we do NOT reuse its backend) +| What | Where | +|---|---| +| Electron desktop app | `apps/desktop/` (its own `AGENTS.md`, `DESIGN.md`) | +| Shared JSON-RPC WS client (tui_gateway protocol) | `apps/shared/src/json-rpc-gateway.ts` | +| tui_gateway WS transport (mobile-client-ready) | `tui_gateway/ws.py` | +| tui_gateway method catalog | `tui_gateway/server.py` | + +> Note: the existing desktop app talks to the **`tui_gateway`** backend +> (`hermes serve`), a *different* process from the messaging gateway. Our apps +> talk to the **messaging gateway** via our own plugin + protocol. We borrow +> naming conventions only. \ No newline at end of file diff --git a/docs/16-open-questions.md b/docs/16-open-questions.md new file mode 100644 index 0000000..2ec93e4 --- /dev/null +++ b/docs/16-open-questions.md @@ -0,0 +1,75 @@ +# 16 — Decisions & Open Questions + +## Locked decisions (from planning, 2026-08-19) + +| # | Decision | Choice | Rationale | +|---|---|---|---| +| 1 | Desktop app tech | **Compose Multiplatform** | Desktop = "the Android app, tweaked"; share protocol/state/UI. | +| 2 | Push backend | **Both — FCM primary, ntfy fallback** | FCM is standard/reliable; ntfy for self-hosters with no Firebase. `ANDROID_PUSH_BACKEND`. | +| 3 | Media transport | **Over the WebSocket** | One transport, zero new Python deps; chunked binary frames. | +| 4 | Phone default layout | **User-toggleable, single-pane default** | App-like on phones; auto two-pane on large screens; desktop defaults two-pane. | + +## Additional decisions made during planning + +| Decision | Choice | Note | +|---|---|---| +| Connection point | **Messaging gateway** (platform plugin `android`), not `tui_gateway` | Makes cron/`send_message`/slash/coexistence native. | +| Plugin style | **Community plugin** (`register(ctx)`) | Zero hermes-core changes. | +| Python deps | **None new** (`websockets` + `httpx` are core) | Respects hermes pinning policy. | +| Tool events | **Structured frames; app controls verbosity** | Gateway sends full data; app = everything/truncated/nothing. | +| Reasoning | **Adapter splits the `show_reasoning` prefix** | Clean `reasoning` field → collapsible block above message. | +| Channels/threads | **Map onto `chat_id`/`thread_id`** | Existing gateway primitives; cron targets them. | +| Offline | **SQLite outbox + sync cursor** | Catch-up on reconnect; push on disconnect. | +| Default chat id | `android:default` | Home channel + cron default. | +| WS port | `8790` (default) | Configurable. | +| minSdk | 26 (test device API 29) | Broad coverage. | +| Frame routing | **Broadcast to all connected devices** (no per-chat subscribe) | Single-user model; simpler. | +| Initial channel load | **`history` frame** (paginated) | `sync` only replays outbox; `history` loads full messages. | +| Slash menu | **`commands.catalog` + `commands.complete`** frames | Gateway serves catalog; app renders bottom sheet + autocomplete. | +| Agent lifecycle | **`agent.busy`/`agent.idle`** events + **`agent.stop`/`agent.steer`** requests | App shows thinking indicator; user can abort or steer mid-turn. | +| Local DB (KMP) | **SQLDelight** (not Room) | Room is Android-only; SQLDelight works in commonMain for both platforms. | +| Voice input | **Record → upload as audio media** (no client-side STT) | Agent's STT (if configured) handles transcription. | + +## Open questions (resolve during implementation) + +These are **implementation-time** details, not blockers. Each has a default we +will proceed with unless you say otherwise. + +1. **Tool-progress vs commentary classification (M2).** The exact signal that + distinguishes a tool-progress `send()` from a regular `send()`/commentary in + the legacy path. *Default:* per-chat turn-state machine + line-format + heuristic, **verified empirically** with the `ws_probe.py` harness against the + live gateway. If a clean metadata marker exists, prefer it. +2. **Reasoning prefix format stability (M2).** We split on the `code`-style + `💭 **Reasoning:**\n```\n…\n```\n\n` prefix. *Default:* set + `reasoning_style: code` for android and split on that; fallback = no + reasoning field (full text) if the prefix isn't found. Verify in M2. +3. **Per-device tokens vs shared token (M1/M5).** *Default (v1):* shared + `ANDROID_TOKEN` + optional `ANDROID_ALLOWED_USERS` device allowlist. + Per-device revocable tokens are a stretch. +4. **Desktop video backend (M6).** *Default:* `libmpv`/`mpv`-backed Compose + surface, WebView fallback. Confirm `mpv` availability on target OSes during + M6. +5. **Remote access default (M1).** *Default:* document Tailscale as the + recommended remote path; WSS + reverse proxy as alternatives. No public bind + by default. +6. **Streaming cadence (M2).** If live updates look chunky, tune + `display.platforms.android.streaming` / consumer edit interval. *Default:* + follow global streaming config. +7. **App package name / branding.** *Default:* applicationId `dev.iris.app`, + app name "Iris". Confirm final product name + package + icon. +8. **ntfy in-app listener (M5).** *Default:* a foreground service maintaining the + ntfy subscription (no extra native lib) — or a lightweight ntfy client lib if + one is acceptable. Decide in M5. +9. **Auto-update for desktop (M7).** *Default:* out of scope (manual download). + Revisit post-v1. +10. **iOS port.** Out of scope for v1 (protocol is transport-agnostic, so it's a + future port). No action now. + +## Explicit non-goals (v1) + +- Multi-user / group chat (personal 1-user agent). +- End-to-end encryption (transport WSS only). +- Standalone-cron delivery while the gateway process is fully down (best-effort + push only). +- iOS. \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..f90ac1e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,67 @@ +# Iris × Hermes — Implementation Reference Library + +A coder-facing reference library for building a **native Android + Desktop +experience** for [hermes-agent](https://github.com/NousResearch/hermes-agent), +connected through a **gateway platform plugin**. + +This folder is the single source of truth for *what to build and why*. Read it +top-to-bottom once, then use the numbered docs as a lookup while implementing. + +> ⚠️ **READ FIRST — two hard rules** +> 1. **`hermes-agent/` (sibling of this folder) is a read-only research +> reference. It must NEVER be committed, pushed, or shipped.** It is +> git-ignored at the repo root. We only *install* our plugin into a live +> hermes install (`~/.hermes/plugins/`); we never modify hermes core. +> 2. **ADB is installed and a device is connected** (`a5ca2a4b`, Xiaomi MIX 2S, +> Android 10 / API 29). Use it to install/launch/debug the app on-device. + +--- + +## Reading order + +| # | File | When to read | +|---|------|--------------| +| 0 | [`00-overview.md`](00-overview.md) | Always first. Vision, scope, disclaimers, locked decisions. | +| 1 | [`01-architecture.md`](01-architecture.md) | Before touching code. System shape + rationale. | +| 2 | [`02-monorepo.md`](02-monorepo.md) | When scaffolding the repo. | +| 3 | [`03-gateway-plugin.md`](03-gateway-plugin.md) | When building the Python plugin. | +| 4 | [`04-wire-protocol.md`](04-wire-protocol.md) | When implementing either side of the WS. | +| 5 | [`05-streaming.md`](05-streaming.md) | Streaming / reasoning / tools / intermediate. | +| 6 | [`06-channels-cron-search.md`](06-channels-cron-search.md) | Channels, threads, cron delivery, search. | +| 7 | [`07-media.md`](07-media.md) | Media upload/download + playback. | +| 8 | [`08-push.md`](08-push.md) | Push (FCM + ntfy), outbox, sync. | +| 9 | [`09-pairing-security.md`](09-pairing-security.md) | Pairing, auth, security model. | +| 10 | [`10-android-app.md`](10-android-app.md) | When building the Android app. | +| 11 | [`11-desktop-app.md`](11-desktop-app.md) | When building the Desktop app. | +| 12 | [`12-toolchain.md`](12-toolchain.md) | First time on a machine (JDK/SDK/uv/Firebase). | +| 13 | [`13-testing.md`](13-testing.md) | Writing tests + on-device ADB workflow. | +| 14 | [`14-milestones.md`](14-milestones.md) | Planning work / tracking progress. | +| 15 | [`15-hermes-reference.md`](15-hermes-reference.md) | **Cheat-sheet** of hermes-agent source to read. | +| 16 | [`16-open-questions.md`](16-open-questions.md) | Decisions made + open items. | + +Machine-readable / diagrams: +- [`protocol/frames.schema.json`](protocol/frames.schema.json) — wire-frame schema. +- [`diagrams/architecture.mmd`](diagrams/architecture.mmd) — mermaid architecture. + +--- + +## The three deliverables (one monorepo) + +1. **`gateway-plugin/`** — a Python hermes **platform plugin** named `android`. + Runs inside the `hermes gateway` process. Opens a WebSocket server the apps + connect to. Implements the full `BasePlatformAdapter` contract. **Zero new + Python dependencies, zero hermes-core changes.** +2. **`app/androidApp`** — native Kotlin + Jetpack Compose client. +3. **`app/desktopApp`** — Kotlin + Compose Multiplatform client that *shares* + the Android app's code and is "tweaked" for a big screen. + +The Android and Desktop clients live in **one Compose Multiplatform Gradle +project** (`app/`) with a shared KMP module (`app/shared`). + +--- + +## Status + +- **Phase:** Planning complete → ready to implement (Milestone M0). +- **Owner decisions locked:** see [`16-open-questions.md`](16-open-questions.md). +- **Last updated:** 2026-08-19. \ No newline at end of file diff --git a/docs/diagrams/architecture.mmd b/docs/diagrams/architecture.mmd new file mode 100644 index 0000000..92ca9c1 --- /dev/null +++ b/docs/diagrams/architecture.mmd @@ -0,0 +1,54 @@ +%% Iris x Hermes — architecture (mermaid) +%% Render with any mermaid viewer (e.g. https://mermaid.live or `mmdc`). +flowchart TB + subgraph HOST["User's machine (home server / PC)"] + subgraph GW["hermes gateway (ONE process)"] + AGENT["Agent core
(run_agent.py)"] + SESS["Sessions
(SQLite + FTS5)"] + CRON["Cron scheduler"] + subgraph PLUGIN["android PLATFORM PLUGIN"] + ADAPTER["AndroidAdapter
(BasePlatformAdapter)"] + OUTBOX["Outbox (SQLite)
+ sync cursor"] + PUSH["push.py
FcmBackend / NtfyBackend"] + MEDIA["media.py
cache + chunk stream"] + PAIR["pairing.py
token + device registry"] + SEARCH["search.py
FTS5 bridge"] + end + WSS["WebSocket SERVER
(websockets) ws://host:8790/ws"] + end + AGENT -->|legacy stream callbacks| ADAPTER + CRON -->|deliver=android:chat:thread| ADAPTER + ADAPTER <--> WSS + ADAPTER <--> OUTBOX + ADAPTER <--> PUSH + ADAPTER <--> MEDIA + ADAPTER <--> PAIR + ADAPTER <--> SEARCH + ADAPTER <--> SESS + end + + subgraph CLOUD["Cloud relays"] + FCM["Google FCM"] + NTFY["ntfy (self-host / ntfy.sh)"] + end + + subgraph DEVICES["Clients"] + PHONE["ANDROID APP
(Kotlin / Compose)
WS client + ExoPlayer + FCM"] + DESKTOP["DESKTOP APP
(Compose Multiplatform)
WS client + tray + desktop player"] + end + + WSS <-->|WSS JSON + binary media| PHONE + WSS <-->|WSS JSON + binary media| DESKTOP + PUSH -->|FCM HTTP v1| FCM + PUSH -->|ntfy publish| NTFY + FCM -->|wake (data msg)| PHONE + NTFY -->|subscribe| PHONE + + classDef host fill:#eef,stroke:#333; + classDef plugin fill:#efe,stroke:#333; + classDef device fill:#fee,stroke:#333; + classDef cloud fill:#ffe,stroke:#333; + class GW,AGENT,SESS,CRON,OUTBOX,PUSH,MEDIA,PAIR,SEARCH,WSS host; + class ADAPTER plugin; + class PHONE,DESKTOP device; + class FCM,NTFY cloud; \ No newline at end of file diff --git a/docs/protocol/frames.schema.json b/docs/protocol/frames.schema.json new file mode 100644 index 0000000..5583475 --- /dev/null +++ b/docs/protocol/frames.schema.json @@ -0,0 +1,107 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Iris x Hermes wire protocol", + "description": "Machine-readable description of the JSON frames exchanged over the gateway WebSocket. Mirrors gateway-plugin/protocol.py and docs/04-wire-protocol.md. Media travels as binary WS frames referenced by header/end frames.", + "protocol_version": 1, + "envelope": { + "type": "object", + "required": ["v", "type"], + "properties": { + "v": { "type": "integer", "const": 1, "description": "Protocol version." }, + "id": { "type": "integer", "description": "Request id; present on requests and their responses/acks. Absent on pure events." }, + "type": { "type": "string", "description": "Frame type (see frame_types)." }, + "chat_id": { "type": "string", "description": "Optional chat scope (e.g. android:default, android:chan_7)." }, + "thread_id": { "type": "string", "description": "Optional thread scope within a chat_id." }, + "payload": { "type": "object", "description": "Type-specific payload." } + } + }, + "frame_types": { + "server_to_app": { + "hello.ack": { + "description": "Pairing succeeded.", + "payload": { + "server_caps": { "type": "object", "properties": { "streaming": {"type":"boolean"}, "reasoning": {"type":"boolean"}, "tools": {"type":"boolean"}, "media": {"type":"boolean"}, "search": {"type":"boolean"}, "push": {"type":"string","enum":["fcm","ntfy","none"]}, "pickers": {"type":"boolean"} } }, + "sync_cursor": { "type": "integer" }, + "channels": { "type": "array", "items": { "$ref": "#/definitions/channel" } } + } + }, + "message": { + "description": "A final / standalone message.", + "payload": { + "message_id": { "type": "string" }, + "role": { "type": "string", "enum": ["user", "assistant", "system", "cron"] }, + "text": { "type": "string" }, + "reasoning": { "type": "string", "description": "Optional; render ABOVE text." }, + "media": { "type": "array", "items": { "$ref": "#/definitions/media_ref" } }, + "reply_to": { "type": "string" }, + "model": { "type": "string" }, + "tokens": { "type": "integer" }, + "ts": { "type": "integer", "description": "epoch millis" } + } + }, + "message.start": { "payload": { "message_id": { "type": "string" }, "role": { "type": "string" } } }, + "message.update": { "payload": { "message_id": { "type": "string" }, "text": { "type": "string", "description": "Full current text (app replaces)." } } }, + "message.stop": { "payload": { "message_id": { "type": "string" }, "final_text": { "type": "string" }, "reasoning": { "type": "string" }, "model": { "type": "string" }, "tokens": { "type": "integer" } } }, + "commentary": { "description": "Intermediate assistant beat.", "payload": { "message_id": { "type": "string" }, "text": { "type": "string" } } }, + "tool.start": { "payload": { "index": { "type": "integer" }, "name": { "type": "string" }, "preview": { "type": "string" }, "args": { "type": "object" } } }, + "tool.progress": { "payload": { "index": { "type": "integer" }, "name": { "type": "string" }, "note": { "type": "string" } } }, + "tool.end": { "payload": { "index": { "type": "integer" }, "name": { "type": "string" }, "ok": { "type": "boolean" }, "duration": { "type": "number" }, "output_preview": { "type": "string" } } }, + "typing": { "payload": { "on": { "type": "boolean" } } }, + "notification": { "payload": { "kind": { "type": "string", "enum": ["channel_renamed", "channel_created", "cron", "approval", "clarify", "generic"] }, "title": { "type": "string" }, "body": { "type": "string" }, "ts": { "type": "integer" } } }, + "picker.model": { "payload": { "picker_id": { "type": "string" }, "current_model": { "type": "string" }, "current_provider": { "type": "string" }, "providers": { "type": "array", "items": { "type": "object", "properties": { "id": {"type":"string"}, "label": {"type":"string"}, "models": { "type": "array", "items": { "type": "object", "properties": { "id": {"type":"string"}, "label": {"type":"string"} } } } } } } } }, + "picker.choice": { "payload": { "picker_id": { "type": "string" }, "title": { "type": "string" }, "choices": { "type": "array", "items": { "type": "object", "properties": { "value": {"type":"string"}, "label": {"type":"string"}, "is_current": {"type":"boolean"} } } } } }, + "picker.clarify": { "payload": { "picker_id": { "type": "string" }, "question": { "type": "string" }, "choices": { "type": "array", "items": { "type": "object" } } } }, + "picker.approval": { "payload": { "picker_id": { "type": "string" }, "command": { "type": "string" }, "description": { "type": "string" } } }, + "picker.confirm": { "payload": { "picker_id": { "type": "string" }, "title": { "type": "string" }, "message": { "type": "string" } } }, + "channel.list": { "payload": { "channels": { "type": "array", "items": { "$ref": "#/definitions/channel" } } } }, + "channel.created": { "payload": { "$ref": "#/definitions/channel" } }, + "channel.renamed": { "payload": { "chat_id": { "type": "string" }, "name": { "type": "string" } } }, + "channel.deleted": { "payload": { "chat_id": { "type": "string" } } }, + "history": { "description": "Response to history request; page of messages oldest→newest.", "payload": { "messages": { "type": "array", "items": { "type": "object", "properties": { "message_id": {"type":"string"}, "role": {"type":"string"}, "text": {"type":"string"}, "reasoning": {"type":"string"}, "model": {"type":"string"}, "tokens": {"type":"integer"}, "ts": {"type":"integer"} } } }, "has_more": { "type": "boolean" }, "oldest_message_id": { "type": "string" } } }, + "commands.catalog": { "description": "Full slash-command catalog.", "payload": { "commands": { "type": "array", "items": { "type": "object", "properties": { "name": {"type":"string"}, "description": {"type":"string"}, "args_hint": {"type":"string"}, "category": {"type":"string"} } } } } }, + "commands.complete": { "description": "Autocomplete matches for a typed prefix.", "payload": { "prefix": { "type": "string" }, "matches": { "type": "array", "items": { "type": "object", "properties": { "name": {"type":"string"}, "description": {"type":"string"}, "args_hint": {"type":"string"} } } } } }, + "agent.busy": { "description": "Agent is processing; app shows thinking indicator.", "payload": { "reason": { "type": "string", "enum": ["processing", "tool", "waiting_input", "cron"] } } }, + "agent.idle": { "description": "Agent turn complete; clear thinking indicator.", "payload": {} }, + "search.results": { "payload": { "query": { "type": "string" }, "scope": { "type": "string", "enum": ["all", "chat"] }, "hits": { "type": "array", "items": { "type": "object", "properties": { "message_id": {"type":"string"}, "chat_id": {"type":"string"}, "thread_id": {"type":["string","null"]}, "role": {"type":"string"}, "snippet": {"type":"string"}, "ts": {"type":"integer"} } } } } }, + "media.offer": { "description": "Agent-sent media available; app pulls bytes.", "payload": { "$ref": "#/definitions/media_ref" } }, + "status": { "payload": { "state": { "type": "string", "enum": ["online", "restarting", "degraded"] }, "session": { "type": "object" } } }, + "error": { "payload": { "code": { "type": "string", "enum": ["auth", "not_found", "rate_limited", "media_too_large", "unsupported", "internal"] }, "message": { "type": "string" } } }, + "pong": { "payload": { "ts": { "type": "integer" } } }, + "sync.done": { "payload": { "cursor": { "type": "integer" } } }, + "media.pull.end": { "payload": { "ok": { "type": "boolean" } } } + }, + "app_to_server": { + "hello": { "description": "First frame; auth + caps.", "payload": { "token": { "type": "string" }, "device_id": { "type": "string" }, "device_name": { "type": "string" }, "caps": { "type": "object", "properties": { "min_protocol": {"type":"integer"}, "media": {"type":"boolean"}, "push": {"type":"string"} } }, "fcm_token": { "type": "string" }, "ntfy_topic": { "type": "string" } } }, + "message.send": { "payload": { "text": { "type": "string" }, "reply_to": { "type": "string" }, "media_refs": { "type": "array", "items": { "type": "string" } } } }, + "media.upload.start": { "payload": { "media_ref": { "type": "string" }, "kind": { "$ref": "#/definitions/kind" }, "mime": { "type": "string" }, "size": { "type": "integer" }, "filename": { "type": "string" } } }, + "media.upload.end": { "payload": { "media_ref": { "type": "string" }, "sha256": { "type": "string" } } }, + "media.pull": { "payload": { "media_id": { "type": "string" } } }, + "picker.select": { "payload": { "picker_id": { "type": "string" }, "value": { "type": "string" } } }, + "channel.create": { "payload": { "name": { "type": "string" }, "kind": { "type": "string", "enum": ["channel", "thread"] }, "parent_chat_id": { "type": ["string", "null"] } } }, + "channel.rename": { "payload": { "name": { "type": "string" } } }, + "channel.set_default": { "payload": {} }, + "channel.delete": { "payload": {} }, + "search": { "payload": { "query": { "type": "string" }, "scope": { "type": "string", "enum": ["all", "chat"] }, "chat_id": { "type": "string" }, "thread_id": { "type": "string" } } }, + "history": { "description": "Load a page of messages (initial open / scroll-up).", "payload": { "before_message_id": { "type": "string" }, "limit": { "type": "integer", "minimum": 1, "maximum": 200 } } }, + "commands.catalog": { "description": "Fetch full slash-command catalog.", "payload": {} }, + "commands.complete": { "description": "Autocomplete for typed /prefix.", "payload": { "prefix": { "type": "string" } } }, + "agent.stop": { "description": "Abort current agent turn.", "payload": {} }, + "agent.steer": { "description": "Inject steering message mid-turn.", "payload": { "text": { "type": "string" } } }, + "read.receipt": { "description": "User viewed message; server stores + broadcasts to other devices.", "payload": { "chat_id": { "type": "string" }, "message_id": { "type": "string" } } }, + "sync": { "description": "Reconnect catch-up; replays undelivered outbox frames only (not full history).", "payload": { "cursor": { "type": "integer" } } }, + "fcm.register": { "payload": { "fcm_token": { "type": "string" }, "ntfy_topic": { "type": "string" } } }, + "ping": { "payload": { "ts": { "type": "integer" } } } + } + }, + "definitions": { + "kind": { "type": "string", "enum": ["image", "audio", "video", "document", "voice"] }, + "channel": { "type": "object", "properties": { "chat_id": {"type":"string"}, "name": {"type":"string"}, "kind": {"type":"string","enum":["default","channel","thread"]}, "parent_chat_id": {"type":["string","null"]}, "is_default": {"type":"boolean"} } }, + "media_ref": { "type": "object", "properties": { "media_id": {"type":"string"}, "kind": { "$ref": "#/definitions/kind" }, "mime": {"type":"string"}, "size": {"type":"integer"}, "filename": {"type":"string"} } } + }, + "reliability": { + "ordering": "Per-connection (TCP/WS). message.update for a message_id is monotonic; app may coalesce to latest.", + "never_dropped": ["message", "message.stop", "tool.end", "notification", "picker.*", "channel.*", "agent.busy", "agent.idle", "history", "commands.catalog", "commands.complete", "search.results", "error"], + "coalescable_under_backpressure": ["message.update", "tool.progress"], + "offline": "Undelivered frames go to the outbox; replayed by sync. Terminal frames always outboxed." + } +} diff --git a/gateway-plugin/__init__.py b/gateway-plugin/__init__.py new file mode 100644 index 0000000..9094c78 --- /dev/null +++ b/gateway-plugin/__init__.py @@ -0,0 +1,3 @@ +from .adapter import register + +__all__ = ["register"] \ No newline at end of file diff --git a/gateway-plugin/adapter.py b/gateway-plugin/adapter.py new file mode 100644 index 0000000..2535f48 --- /dev/null +++ b/gateway-plugin/adapter.py @@ -0,0 +1,471 @@ +""" +Android Platform Adapter for Hermes Agent (Iris x Hermes). + +A plugin-based gateway adapter that runs a WebSocket server *inside* the +``hermes gateway`` process. The native Android / Desktop app connects to it +with a pairing token and talks to the agent over a single WS transport +(chat, streaming, tools, media, pairing, push-token). + +Zero new Python dependencies: ``websockets`` and ``httpx`` are hermes core +deps. Zero hermes-core changes. + +Milestone M0: this is a *skeleton* adapter. It registers the ``android`` +platform, resolves its configuration, and implements the abstract adapter +contract as no-ops so that ``hermes gateway status`` lists ``android``. The +WebSocket server, pairing, streaming, media, outbox, push, and search are +wired in later milestones (see ``docs/14-milestones.md``). + +Configuration in config.yaml:: + + gateway: + platforms: + android: + enabled: true + extra: + host: 127.0.0.1 + port: 8790 + home_channel: android:default + push_backend: fcm + outbox_retention_hours: 72 + max_upload_bytes: 104857600 + +Or via environment variables (overrides config.yaml; secrets live in .env): + ANDROID_TOKEN, ANDROID_WS_HOST, ANDROID_WS_PORT, ANDROID_HOME_CHANNEL, + ANDROID_PUSH_BACKEND, ANDROID_FCM_SERVICE_ACCOUNT, NTFY_TOPIC, ... +""" + +import logging +import os +import time +import uuid +from typing import Any, Dict, List, Optional + +from agent.secret_scope import UnscopedSecretError as _UnscopedSecretError +from agent.secret_scope import get_secret as _scoped_get_secret + + +def _get_scoped_secret(name, default=None): + """Scope-aware credential read with the default-profile startup fallback. + + Secondary profiles construct their adapters under a profile secret scope + -- the scope is authoritative and a scoped miss returns ``default`` (no + cross-profile borrow from ``os.environ``, which may hold another + profile's value). The DEFAULT profile's adapter constructs and sends + *unscoped* under multiplexing, where a bare ``get_secret`` would raise + ``UnscopedSecretError`` and crash this path; there ``os.environ`` is that + profile's own value, so fall back to it. Same pattern as the IRC + ``IRC_SERVER_PASSWORD`` read (``plugins/platforms/irc/adapter.py``). + """ + try: + val = _scoped_get_secret(name, default) + except _UnscopedSecretError: + val = os.getenv(name) + return val if val is not None else default + + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Lazy import: BasePlatformAdapter and friends live in the main repo. +# We import at module level (as the bundled plugins do) but guard the heavy +# gateway imports so the plugin can be discovered before the gateway is fully +# initialised. +# --------------------------------------------------------------------------- + +from gateway.platforms.base import ( # noqa: E402 + BasePlatformAdapter, + SendResult, + MessageEvent, + MessageType, +) +from gateway.config import Platform # noqa: E402 + + +# --------------------------------------------------------------------------- +# Defaults +# --------------------------------------------------------------------------- + +DEFAULT_HOST = "127.0.0.1" +DEFAULT_PORT = 8790 +DEFAULT_HOME_CHANNEL = "android:default" +DEFAULT_PUSH_BACKEND = "fcm" +DEFAULT_OUTBOX_RETENTION_HOURS = 72 +DEFAULT_MAX_UPLOAD_BYTES = 100 * 1024 * 1024 # 100 MB + + +def _truthy(value: Optional[str]) -> bool: + return (value or "").strip().lower() in {"1", "true", "yes", "on"} + + +# --------------------------------------------------------------------------- +# Passive / config probes (called from status displays -- no side effects) +# --------------------------------------------------------------------------- + +def check_requirements() -> bool: + """PASSIVE dependency probe: ``websockets`` importable + token set. + + Must be side-effect free (called from ``hermes setup`` / ``status`` / + dashboard readiness). Never installs. + """ + try: + import websockets # noqa: F401 (core dep) + except Exception: + return False + return bool(_get_scoped_secret("ANDROID_TOKEN")) + + +def validate_config(config) -> bool: + """Given a PlatformConfig, is the platform properly configured?""" + extra = getattr(config, "extra", {}) or {} + token = _get_scoped_secret("ANDROID_TOKEN") or extra.get("token", "") + return bool(token) + + +def is_connected(config) -> bool: + """Is the platform configured (env or config.yaml)?""" + return validate_config(config) + + +# --------------------------------------------------------------------------- +# Env-driven auto-configuration (seeds PlatformConfig.extra pre-adapter) +# --------------------------------------------------------------------------- + +def _env_enablement() -> Optional[dict]: + """Seed ``PlatformConfig.extra`` from env vars during gateway config load. + + Called by the platform registry's env-enablement hook BEFORE adapter + construction, so ``gateway status`` and ``get_connected_platforms()`` + reflect env-only configuration without instantiating the adapter. + Returns ``None`` when the platform isn't minimally configured (no token); + the caller then skips auto-enabling. + + The special ``home_channel`` key in the returned dict is handled by the + core hook -- it becomes a proper ``HomeChannel`` dataclass on the + ``PlatformConfig`` rather than being merged into ``extra``. + """ + token = _get_scoped_secret("ANDROID_TOKEN", "") + if not token: + return None + + seed: Dict[str, Any] = { + "host": os.getenv("ANDROID_WS_HOST", "").strip() or DEFAULT_HOST, + "port": _parse_port(os.getenv("ANDROID_WS_PORT", "")), + "push_backend": ( + os.getenv("ANDROID_PUSH_BACKEND", "").strip().lower() + or DEFAULT_PUSH_BACKEND + ), + } + home = os.getenv("ANDROID_HOME_CHANNEL", "").strip() or DEFAULT_HOME_CHANNEL + seed["home_channel"] = { + "chat_id": home, + "name": os.getenv("ANDROID_HOME_CHANNEL_NAME", "").strip() or "Default", + } + return seed + + +def _parse_port(raw: str) -> int: + try: + return int((raw or "").strip()) + except (ValueError, TypeError): + return DEFAULT_PORT + + +# --------------------------------------------------------------------------- +# Target parsing: "android:[:]" +# --------------------------------------------------------------------------- + +def _parse_target_ref(target_ref: str) -> Optional[tuple]: + """Parse a raw target string into ``(chat_id, thread_id)`` or ``None``. + + Recognises the native syntax ``android:[:]``. Returns + ``None`` for anything else so the target proceeds to channel-directory + resolution. + """ + if not target_ref or not target_ref.startswith("android:"): + return None + body = target_ref[len("android:"):] + if not body: + return None + if ":" in body: + chat_id, thread_id = body.split(":", 1) + thread_id = thread_id or None + else: + chat_id, thread_id = body, None + chat_id = chat_id.strip() + if not chat_id: + return None + return (chat_id, thread_id) + + +# --------------------------------------------------------------------------- +# Standalone (out-of-process) send -- best-effort, stretch for v1 +# --------------------------------------------------------------------------- + +async def _standalone_send( + pconfig, + chat_id: str, + message: str, + *, + thread_id: Optional[str] = None, + media_files: Optional[List[str]] = None, + force_document: bool = False, +) -> Dict[str, Any]: + """Out-of-process delivery for cron jobs that run separately from the + gateway. + + The outbox is served by the *running* gateway, so standalone delivery + while the gateway process is fully down is best-effort only (see + ``docs/00-overview.md`` "Out of scope"). For M0 this is a stub that + reports the gateway is required; the real implementation lands with the + outbox (M3/M5). + """ + return { + "error": ( + "android standalone send: the running gateway is required to serve " + "the outbox (standalone delivery is best-effort only)" + ) + } + + +# --------------------------------------------------------------------------- +# Interactive setup (hermes gateway setup flow) -- full version in M1 +# --------------------------------------------------------------------------- + +def interactive_setup() -> None: + """Prompt for the pairing token / host / port / push backend. + + M0: minimal. M1 adds token generation, QR payload, and a live ``hello`` + connectivity test. + """ + try: + from hermes_cli.config import ( + get_env_value, + save_env_value, + prompt, + print_info, + print_success, + print_warning, + ) + except Exception: + print("android: setup helpers unavailable; set ANDROID_TOKEN in ~/.hermes/.env") + return + + print_info("📱 Android / Desktop (Iris x Hermes)") + token = get_env_value("ANDROID_TOKEN") or "" + if not token: + generated = uuid.uuid4().hex + uuid.uuid4().hex # 64 hex chars + save_env_value("ANDROID_TOKEN", generated) + print_success(f"Generated pairing token: {generated}") + print_warning("Keep this secret -- the app presents it on connect.") + else: + print_info("Existing ANDROID_TOKEN found (not shown).") + + host = prompt("WS bind host", default=get_env_value("ANDROID_WS_HOST") or DEFAULT_HOST) + save_env_value("ANDROID_WS_HOST", host or DEFAULT_HOST) + port = prompt("WS port", default=str(_parse_port(get_env_value("ANDROID_WS_PORT") or ""))) + save_env_value("ANDROID_WS_PORT", str(_parse_port(port))) + backend = prompt("Push backend (fcm/ntfy)", default=get_env_value("ANDROID_PUSH_BACKEND") or DEFAULT_PUSH_BACKEND) + save_env_value("ANDROID_PUSH_BACKEND", (backend or DEFAULT_PUSH_BACKEND).strip().lower()) + + print_success("Android configuration saved to ~/.hermes/.env") + print_info("Restart the gateway for changes to take effect: hermes gateway restart") + + +# --------------------------------------------------------------------------- +# Android Adapter +# --------------------------------------------------------------------------- + +class AndroidAdapter(BasePlatformAdapter): + """WebSocket-backed adapter for the native Iris Android / Desktop app. + + M0: skeleton. Implements the abstract adapter contract as no-ops and + resolves configuration. The WebSocket server, connection registry, + pairing, streaming, media, outbox, push, and search are added in later + milestones. + """ + + def __init__(self, config, **kwargs): + platform = Platform("android") + super().__init__(config=config, platform=platform) + + extra = getattr(config, "extra", {}) or {} + + # Connection settings (env vars override config.yaml) + self.host = os.getenv("ANDROID_WS_HOST", "").strip() or extra.get("host", DEFAULT_HOST) + self.port = _parse_port(os.getenv("ANDROID_WS_PORT", "") or str(extra.get("port", DEFAULT_PORT))) + self.token = _get_scoped_secret("ANDROID_TOKEN") or extra.get("token", "") + self.home_channel = extra.get("home_channel", DEFAULT_HOME_CHANNEL) + self.push_backend = ( + os.getenv("ANDROID_PUSH_BACKEND", "").strip().lower() + or extra.get("push_backend", DEFAULT_PUSH_BACKEND) + ) + self.outbox_retention_hours = int( + extra.get("outbox_retention_hours", DEFAULT_OUTBOX_RETENTION_HOURS) + ) + self.max_upload_bytes = int( + extra.get("max_upload_bytes", DEFAULT_MAX_UPLOAD_BYTES) + ) + + # TLS (optional) + self.ws_cert = _get_scoped_secret("ANDROID_WS_CERT") or extra.get("ws_cert", "") + self.ws_key = _get_scoped_secret("ANDROID_WS_KEY") or extra.get("ws_key", "") + + # Auth + allowed = os.getenv("ANDROID_ALLOWED_USERS", "").strip() + self.allowed_users: List[str] = ( + [u.strip() for u in allowed.split(",") if u.strip()] if allowed else [] + ) + self.allow_all = _truthy(os.getenv("ANDROID_ALLOW_ALL_USERS")) + + # Runtime state (populated by the WS server in M1) + self._ws_server = None + self._connections: Dict[str, Any] = {} + self._connected = False + + @property + def name(self) -> str: + return "Android" + + # ── Connection lifecycle ────────────────────────────────────────────── + + async def connect(self, *, is_reconnect: bool = False) -> bool: + """Bring the platform up. + + M0: no WebSocket server yet -- just validate config and mark + connected so ``hermes gateway status`` reflects the platform. M1 + starts the ``websockets`` server here. + """ + if not self.token: + logger.error("android: ANDROID_TOKEN must be set") + self._set_fatal_error( + "config_missing", + "ANDROID_TOKEN must be set", + retryable=False, + ) + return False + + # Prevent two profiles from binding the same port/identity. + try: + from gateway.status import acquire_scoped_lock + lock_key = f"{self.host}:{self.port}" + if not acquire_scoped_lock("android", lock_key): + logger.error("android: %s:%s already in use by another profile", self.host, self.port) + self._set_fatal_error( + "lock_conflict", + "WS port in use by another profile", + retryable=False, + ) + return False + self._lock_key = lock_key + except ImportError: + self._lock_key = None # status module not available (e.g. tests) + + # M1: start the websockets server on host:port (TLS if cert/key set). + self._connected = True + self._mark_connected() + logger.info("android: connected (skeleton; WS server starts in M1) on %s:%s", self.host, self.port) + return True + + async def disconnect(self) -> None: + """Tear down the platform.""" + try: + from gateway.status import release_scoped_lock + if getattr(self, "_lock_key", None): + release_scoped_lock("android", self._lock_key) + except ImportError: + pass + # M1: stop the server and close all device sockets. + self._connected = False + self._mark_disconnected() + logger.info("android: disconnected") + + # ── Outbound (agent -> app) ─────────────────────────────────────────── + + async def send( + self, + chat_id: str, + content: str, + reply_to: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> SendResult: + """Send a message to a chat. + + M0: no live devices yet -- log and report success with a minted id. + M1: broadcast a ``message`` frame to connected devices, else fall to + the outbox + fire push. + """ + message_id = f"msg_{uuid.uuid4().hex}" + logger.debug("android: send to %s (%d chars) [skeleton no-op]", chat_id, len(content or "")) + return SendResult(success=True, message_id=message_id) + + async def send_typing(self, chat_id: str, metadata: Optional[Dict[str, Any]] = None) -> None: + """Send a typing indicator. M0: no-op (M1 emits a ``typing`` frame).""" + return None + + async def send_image( + self, + chat_id: str, + image_url: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> SendResult: + """Send an image. M0: not implemented (M4).""" + return SendResult(success=False, error="android: media not implemented yet (M4)") + + # ── Chat info ───────────────────────────────────────────────────────── + + async def get_chat_info(self, chat_id: str) -> Dict[str, Any]: + """Return ``{name, type, chat_id}`` for a chat. + + M0: the channel directory is not persisted yet, so report the home + channel name for the default chat and a generic name otherwise. + """ + name = "Default" if chat_id in (self.home_channel, DEFAULT_HOME_CHANNEL) else (chat_id or "chat") + return {"name": name, "type": "channel", "chat_id": chat_id} + + +# --------------------------------------------------------------------------- +# Plugin entry point +# --------------------------------------------------------------------------- + +def register(ctx): + """Plugin entry point: called by the Hermes plugin system.""" + ctx.register_platform( + name="android", + label="Android", + adapter_factory=lambda cfg: AndroidAdapter(cfg), + check_fn=check_requirements, + validate_config=validate_config, + is_connected=is_connected, + required_env=["ANDROID_TOKEN"], + install_hint="No extra packages needed (websockets + httpx are core deps)", + setup_fn=interactive_setup, + # Env-driven auto-configuration: seeds PlatformConfig.extra with + # host/port/push_backend + home_channel so env-only setups show up in + # gateway status without instantiating the adapter. + env_enablement_fn=_env_enablement, + # Cron home-channel delivery support (deliver=android:[:]). + cron_deliver_env_var="ANDROID_HOME_CHANNEL", + # Out-of-process cron delivery (best-effort; outbox is gateway-served). + standalone_sender_fn=_standalone_send, + # Native target syntax: "android:[:]". + parse_target_ref_fn=_parse_target_ref, + # Auth env vars for _is_user_authorized() integration. + allowed_users_env="ANDROID_ALLOWED_USERS", + allow_all_env="ANDROID_ALLOW_ALL_USERS", + # WS has no message-size limit. + max_message_length=0, + # Display. + emoji="📱", + pii_safe=False, + allow_update_command=True, + # LLM guidance. + platform_hint=( + "You are chatting with the user through their native Iris app " + "(Android/Desktop). It renders Markdown, inline code, images, " + "audio and video, and shows your reasoning and tool activity. " + "Conversations are organized into channels and optional threads. " + "Keep formatting rich but readable." + ), + ) \ No newline at end of file diff --git a/gateway-plugin/media.py b/gateway-plugin/media.py new file mode 100644 index 0000000..4f82ffa --- /dev/null +++ b/gateway-plugin/media.py @@ -0,0 +1,16 @@ +"""Inbound media cache + outbound chunked streaming. + +Inbound: ``media.upload`` (chunked binary frames) -> ``cache_*_from_bytes`` +-> a ``media_ref`` the adapter attaches to the ``MessageEvent``. Enforces +size limit + sha256 + MIME re-sniff. + +Outbound: ``send_*`` -> stage the file in the media cache, mint a +``media_id``, emit ``media.offer {media_id, mime, size, filename, kind}``; +serve bytes on ``media.pull`` as chunked binary frames. Delivery-path +security via ``validate_media_delivery_path``. + +Reuses hermes ``cache_image/audio/video/document_from_bytes`` where possible. +All paths under ``get_hermes_home()/"android"/media``. + +Milestone M4. +""" \ No newline at end of file diff --git a/gateway-plugin/outbox.py b/gateway-plugin/outbox.py new file mode 100644 index 0000000..2101dc0 --- /dev/null +++ b/gateway-plugin/outbox.py @@ -0,0 +1,10 @@ +"""SQLite offline outbox + monotonic sync cursor. + +Undelivered frames are appended per ``chat_id`` with a monotonic cursor so a +reconnecting app can ``sync {cursor}`` the delta without re-reading full +history. Retention prunes old entries (``outbox_retention_hours``). + +Storage: ``get_hermes_home()/"android"/outbox.db``. + +Milestone M3 (built), extended in M5 (push integration). +""" \ No newline at end of file diff --git a/gateway-plugin/pairing.py b/gateway-plugin/pairing.py new file mode 100644 index 0000000..3d36337 --- /dev/null +++ b/gateway-plugin/pairing.py @@ -0,0 +1,10 @@ +"""Pairing: token generation/verification + device registry. + +Token generation (64-hex) and constant-time verification. Device registry +(SQLite) tracks ``device_id``, name, caps, fcm_token, ntfy_topic, last_seen, +created. QR payload for the pairing flow (``interactive_setup``). + +Storage: ``get_hermes_home()/"android"/devices.db``. + +Milestone M1. +""" \ No newline at end of file diff --git a/gateway-plugin/plugin.yaml b/gateway-plugin/plugin.yaml new file mode 100644 index 0000000..f7c1cfa --- /dev/null +++ b/gateway-plugin/plugin.yaml @@ -0,0 +1,67 @@ +name: android-platform +label: Android +kind: platform +version: 0.1.0 +description: > + Native Android / Desktop client gateway adapter for Hermes Agent. + Runs a WebSocket server inside the gateway; the app connects with a + pairing token. Supports streaming, reasoning, structured tool events, + channels/threads, media, FTS5 search, and FCM/ntfy push. +author: Iris x Hermes +# ``requires_env`` / ``optional_env`` entries are surfaced in the +# ``hermes config`` / ``hermes gateway setup`` UI via the platform-plugin +# env var injector in ``hermes_cli/config.py``. +requires_env: + - name: ANDROID_TOKEN + description: "Shared pairing token the app presents on connect" + prompt: "Android pairing token" + password: true +optional_env: + - name: ANDROID_WS_HOST + description: "WS bind host (default 127.0.0.1; use 0.0.0.0 for LAN)" + prompt: "WS host" + password: false + - name: ANDROID_WS_PORT + description: "WS port (default 8790)" + prompt: "WS port" + password: false + - name: ANDROID_HOME_CHANNEL + description: "Default chat id for cron/notification delivery (default android:default)" + prompt: "Home channel" + password: false + - name: ANDROID_ALLOWED_USERS + description: "Comma-separated allowed device_ids (empty = token-only auth)" + prompt: "Allowed device ids" + password: false + - name: ANDROID_ALLOW_ALL_USERS + description: "Allow any paired device (dev only)" + prompt: "Allow all devices? (true/false)" + password: false + - name: ANDROID_PUSH_BACKEND + description: "Push backend: fcm (default) or ntfy" + prompt: "Push backend" + password: false + - name: ANDROID_FCM_SERVICE_ACCOUNT + description: "Path to Firebase service-account JSON (FCM HTTP v1)" + prompt: "FCM service account path" + password: true + - name: ANDROID_FCM_SERVER_KEY + description: "Legacy FCM server key (fallback if no service account)" + prompt: "FCM server key" + password: true + - name: NTFY_TOPIC + description: "ntfy topic for push (when ANDROID_PUSH_BACKEND=ntfy)" + prompt: "ntfy topic" + password: false + - name: NTFY_SERVER_URL + description: "ntfy server URL (default https://ntfy.sh)" + prompt: "ntfy server URL" + password: false + - name: ANDROID_WS_CERT + description: "TLS cert path for WSS (optional)" + prompt: "WSS cert" + password: false + - name: ANDROID_WS_KEY + description: "TLS key path for WSS (optional)" + prompt: "WSS key" + password: false \ No newline at end of file diff --git a/gateway-plugin/protocol.py b/gateway-plugin/protocol.py new file mode 100644 index 0000000..9be6b4e --- /dev/null +++ b/gateway-plugin/protocol.py @@ -0,0 +1,22 @@ +"""Frame schemas -- the single source of truth for the wire protocol. + +Every frame the plugin sends/receives is modelled here as a dataclass + +constants. ``docs/protocol/frames.schema.json`` is generated/mirrored from +this module, and the Kotlin side mirrors these shapes (see +``docs/04-wire-protocol.md``). + +Milestone M1: hello/hello.ack, message, error, ping/pong. +Milestone M2: message.start/update/stop, reasoning, tool.*, commentary, typing. +Milestone M3: channel.*, search, sync. +Milestone M4: media.*. +Milestone M5: notification, fcm.register, read.receipt. +""" + +PROTOCOL_VERSION = 1 + +# Frame type constants (the ``type`` field of every frame). +TYPE_HELLO = "hello" +TYPE_HELLO_ACK = "hello.ack" +TYPE_ERROR = "error" +TYPE_PING = "ping" +TYPE_PONG = "pong" \ No newline at end of file diff --git a/gateway-plugin/push.py b/gateway-plugin/push.py new file mode 100644 index 0000000..7ae9cce --- /dev/null +++ b/gateway-plugin/push.py @@ -0,0 +1,15 @@ +"""Push backends: FCM (primary) + ntfy (fallback). + +``PushBackend`` interface with two implementations: + - ``FcmBackend``: FCM HTTP v1 via ``httpx`` + a Firebase service account + (``ANDROID_FCM_SERVICE_ACCOUNT``), or a legacy server key + (``ANDROID_FCM_SERVER_KEY``). + - ``NtfyBackend``: reuses hermes ntfy publish (``NTFY_TOPIC`` / + ``NTFY_SERVER_URL``). + +Selected by ``ANDROID_PUSH_BACKEND`` (``fcm`` default, ``ntfy`` fallback). +Fired when a frame has no live subscriber; data payload drives a silent sync +on the device. + +Milestone M5. +""" \ No newline at end of file diff --git a/gateway-plugin/search.py b/gateway-plugin/search.py new file mode 100644 index 0000000..a1ff3ed --- /dev/null +++ b/gateway-plugin/search.py @@ -0,0 +1,8 @@ +"""FTS5 session search bridge. + +Bridges the ``search {query, scope, chat_id?, thread_id?}`` frame to the +hermes session store (SQLite + FTS5, ``hermes_state_search.py``) and returns +``search.results``. Scope: "all" (everywhere) or "chat" (this chat/channel). + +Milestone M3. +""" \ No newline at end of file diff --git a/gateway-plugin/tests/README.md b/gateway-plugin/tests/README.md new file mode 100644 index 0000000..6a26aed --- /dev/null +++ b/gateway-plugin/tests/README.md @@ -0,0 +1,7 @@ +# Tests for the android gateway plugin. + +Run via hermes's hermetic runner (never bare pytest):: + + scripts/run_tests.sh tests/gateway/test_android.py + +See ``docs/13-testing.md`` for the scenario list. \ No newline at end of file diff --git a/gateway-plugin/ws_server.py b/gateway-plugin/ws_server.py new file mode 100644 index 0000000..94b21c3 --- /dev/null +++ b/gateway-plugin/ws_server.py @@ -0,0 +1,24 @@ +"""WebSocket server, connection registry, and frame routing. + +Runs on the gateway's asyncio loop (started in ``AndroidAdapter.connect()``). +Uses the ``websockets`` core dep (v15): ``websockets.serve(handler, host, +port, ssl=ctx)``. + +Per-connection handler: + 1. Await first frame; must be ``hello {token, device_id, device_name, + caps, fcm_token?}``. Verify token (constant-time) + allowlist. On + failure: send ``error {code:"auth"}`` and close. + 2. On success: register in the connection registry (``device_id -> + {ws, caps, fcm_token}``), send ``hello.ack {server_caps, sync_cursor, + channels[]}``. + 3. Loop: decode frames, dispatch to adapter inbound handlers. + 4. On close: deregister; if no devices remain, ensure pending outbox + frames have push fired. + +Routing: ``emit(chat_id, frame)`` broadcasts to ALL connected devices +(single-user model). Heartbeat via WS ping/pong + app-level ping/pong. +Backpressure: bounded per-connection send queue; coalesce ``message.update`` +under pressure, never drop ``message``/``tool.end``/``notification``. + +Milestone M1. +""" \ No newline at end of file diff --git a/scripts/guard_hermes_agent.sh b/scripts/guard_hermes_agent.sh new file mode 100755 index 0000000..7a2b703 --- /dev/null +++ b/scripts/guard_hermes_agent.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Guard: hermes-agent/ is a read-only research reference and must NEVER be +# committed, pushed, or shipped. This fails the commit/CI if any path under +# hermes-agent/ is staged. +# +# Usage: +# - pre-commit hook: scripts/guard_hermes_agent.sh --staged +# - CI / manual: scripts/guard_hermes_agent.sh --staged +# +# Exit 0 = clean, exit 1 = hermes-agent/ paths detected. + +set -euo pipefail + +MODE="${1:---staged}" + +if [[ "$MODE" == "--staged" ]]; then + # Every path currently in the index. Using `git ls-files --cached` (rather + # than `git diff --cached`) makes this robust on the very first commit, + # where no HEAD exists yet and `git diff --cached` is unreliable. We never + # want hermes-agent/ in the index at all, so blocking on its presence is + # both correct and safe. + mapfile -t BAD < <(git ls-files --cached | grep -E '^hermes-agent/' || true) +else + # Fallback: any tracked file under hermes-agent/. + mapfile -t BAD < <(git ls-files | grep -E '^hermes-agent/' || true) +fi + +if [[ ${#BAD[@]} -gt 0 ]]; then + echo "ERROR: the following hermes-agent/ paths are staged. This directory is a" >&2 + echo "read-only research reference and must NEVER be committed (see .gitignore" >&2 + echo "and docs/00-overview.md 'Disclaimers'). Unstage them with:" >&2 + echo " git restore --staged " >&2 + printf ' %s\n' "${BAD[@]}" >&2 + exit 1 +fi + +exit 0 \ No newline at end of file