# GitHub Release Monitor — Project Plan ## 1. Overview A CLI tool that tracks GitHub releases or tags from user-defined repositories and exposes them as a subscribable RSS feed. Some repos (e.g., Zammad) only use git tags without creating GitHub releases — both modes are supported. - **Language**: Python 3.12 - **Storage**: SQLite (single file, embedded via `sqlite3` stdlib) - **API**: GitHub REST API via `httpx` - **Output**: RSS 2.0 XML feed served over HTTP (stdlib `http.server`) ## 2. Architecture ``` ┌──────────────────────────────────────────────────┐ │ CLI Interface │ │ (add, remove, update, list, check, serve) │ ├──────────────────────────────────────────────────┤ │ Core Logic │ │ ┌──────────────┐ ┌──────────────┐ ┌────────┐ │ │ │ config.py │ │ db.py │ │github.py│ │ │ │ (defaults, │ │ (sqlite3) │ │(httpx) │ │ │ │ env vars) │ │ │ │ │ │ │ └──────────────┘ └──────┬───────┘ └───┬────┘ │ │ │ │ │ │ ┌────────▼──────────────▼────┐ │ │ │ rss.py │ │ │ │ (xml.etree + markdown) │ │ │ └──────────────┬─────────────┘ │ │ │ │ │ ┌──────────────▼─────────────┐ │ │ │ server.py │ │ │ │ (http.server, stdlib) │ │ │ │ (/feed.xml endpoint) │ │ │ └─────────────────────────────┘ │ └──────────────────────────────────────────────────┘ ``` ## 3. File Structure ``` github_release_monitor/ ├── plan.md # This file ├── pyproject.toml # Project metadata, dependencies ├── README.md ├── src/ │ ├── __init__.py │ ├── cli.py # CLI entry point (argparse) │ ├── config.py # Centralized config (defaults, env vars, paths) │ ├── db.py # SQLite schema & queries │ ├── github.py # GitHub API interaction │ ├── rss.py # RSS feed generation │ └── server.py # HTTP server for feed ├── tests/ │ ├── __init__.py │ ├── test_db.py │ ├── test_github.py │ ├── test_rss.py │ ├── test_server.py │ └── test_integration.py # End-to-end tests └── .gitignore ``` ## 4. Database Schema ### `repos` table | Column | Type | Constraints | Description | |------------|--------------|--------------------------------------|--------------------------------------| | id | INTEGER | PRIMARY KEY AUTO | Internal identifier | | owner_repo | TEXT | NOT NULL UNIQUE | GitHub repo identifier (`owner/repo`)| | mode | TEXT | NOT NULL DEFAULT 'release' CHECK(mode IN ('release', 'tag')) | Track releases or tags | | created_at | DATETIME | DEFAULT NOW() | When the repo was added | The `owner_repo` field serves as both the unique identifier and the display name. The `mode` field determines whether to fetch GitHub releases (`release`) or git tags (`tag`). Default is `release`. ### `entries` table | Column | Type | Constraints | Description | |--------------|--------------|--------------------|----------------------------------------| | id | INTEGER | PRIMARY KEY AUTO | Internal identifier | | repo_id | INTEGER | FK → repos.id | Which repository this entry belongs to | | kind | TEXT | NOT NULL CHECK(kind IN ('release', 'tag')) | Whether this is a release or tag | | tag_name | TEXT | NOT NULL | Git tag / release tag | | title | TEXT | NOT NULL | Release title or tag name | | body | TEXT | DEFAULT '' | Release notes (empty for tags) | | published_at | DATETIME | NOT NULL | Release date or commit date | | html_url | TEXT | NOT NULL | Link to release or commit on GitHub | | fetched_at | DATETIME | DEFAULT NOW() | When we fetched this entry | **Unique constraint**: `(repo_id, tag_name)` to avoid duplicates. ## 5. CLI Commands ``` ghrel # Main command ├── --version # Show version ├── --verbose / -v # Enable verbose/debug output ├── --quiet / -q # Suppress non-essential output ├── --db-path # Override database path (default: ~/.config/ghrel/repos.db) ├── add [--tags] # Add a repository to track ├── remove # Remove a repository ├── update [--tags] [--release] # Update tracking mode ├── list # List all tracked repositories ├── check [--since N] # Fetch latest releases/tags for all repos └── serve [--port 8080] [--host 127.0.0.1] # Start HTTP server ``` ### `add [--tags]` - Accepts a single `owner/repo` argument (e.g., `golang/go`) - `--tags` flag: track git tags instead of releases (default: releases) - Validates the format with a regex: `^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$` - Pings the GitHub API to verify the repo exists (with token validation) - Inserts into `repos` table; errors on duplicate `owner_repo` - On success: `Added golang/go (releases)` or `Added zammad/zammad (tags)` ### `remove ` - Deletes from `repos` table by `owner_repo` - Cascading delete removes associated entries - On success: `Removed golang/go` ### `update [--tags] [--release]` - Updates the tracking mode for an existing repo - `--tags`: switch to tracking tags - `--release`: switch to tracking releases - On success: `Updated zammad/zammad to track tags` ### `list` - Displays a table of tracked repos: ``` REPO MODE ENTRIES LAST CHECKED golang/go release 12 2025-01-15 10:30 zammad/zammad tag 8 2025-01-15 10:30 ``` ### `check [--since N]` - Iterates over all repos in the database - For each repo, fetches latest releases or tags depending on the repo's `mode` - If `--since N` is provided (e.g., `--since 24h`), only fetches entries published in the last N hours - Upserts into `entries` table (insert if new, skip if already known by `tag_name`) - Reports results: `Checked 5 repos: 3 new entries found` - Global rate-limit-aware queue: if one repo hits the rate limit, remaining repos wait ### `serve [--port 8080] [--host 127.0.0.1]` - Starts an HTTP server using stdlib `http.server` - Default host: `127.0.0.1` (localhost only, not exposed to network) - Serves RSS feed at `/feed.xml` - Serves a small index page at `/` with a link to the feed - Health check endpoint at `/health` returns `200 OK` ### Error format All errors go to stderr with non-zero exit code: ``` Error: ``` ## 6. GitHub API Integration ### Approach: Direct HTTP calls via `httpx` - **Authentication**: Read `GH_TOKEN` from environment variable. If set, use it for authenticated requests (5000/h vs 60/h). If not set, fall back to unauthenticated. - **Token validation**: On first API call after startup, check `GET /rate_limit` to verify the token is valid. If `401`, print a clear error: `Error: Invalid or missing GH_TOKEN. Set it with: export GH_TOKEN=your_token` - **Pagination**: Fetch first page only (30 items). Sufficient for tracking recent entries. - **Rate limiting**: Track `X-RateLimit-Remaining` header globally across all repos. If rate limited, pause all requests until `X-RateLimit-Reset` time. No per-repo retries — a single global wait prevents cascading 403s. ### Two API endpoints depending on `mode` **Releases** (`mode = 'release'`): - `GET https://api.github.com/repos/{owner}/{repo}/releases?per_page=30` - Data extracted: `tag_name`, `name` (title), `body` (markdown), `published_at`, `html_url` **Tags** (`mode = 'tag'`): - `GET https://api.github.com/repos/{owner}/{repo}/tags?per_page=30` - Data extracted: `name` (used as both `tag_name` and `title`), `zipball_url` → derive `html_url` as `https://github.com/{owner}/{repo}/tags/{name}` - Tags have no `body` — store empty string - Tags have no `published_at` — fetch commit date via `GET /repos/{owner}/{repo}/git/ref/tags/{name}` and use `object.commit.author.date` from the resolved commit object ## 7. RSS Feed Specification RSS 2.0 generated with stdlib `xml.etree.ElementTree`. RSS 2.0 is simple enough that `feedgen` is unnecessary. ### Feed-level - **title**: "GitHub Release Monitor" - **description**: "Latest releases from tracked GitHub repositories" - **link**: `http://{host}:{port}/` (base URL of the serving instance) - **lastBuildDate**: Current time in RFC 822 format ### Per-item - **title**: `[{owner/repo}] {title}` - **description**: First 300 characters of `body`, with markdown stripped to plain text. Append `…` if truncated. For tags (no body), use: `Tag {tag_name}` - **link**: `html_url` from the entry - **pubDate**: `published_at` in RFC 822 format - **guid**: `github.com/{owner}/{repo}/releases/tag/{tag_name}` (unique identifier, same format for both releases and tags) Items sorted by `published_at` descending (newest first). **Capped at 50 items** to prevent feed bloat. ### Markdown stripping Use `markdown` library to convert markdown to HTML, then `bleach` to strip all HTML tags and produce plain text. ## 8. Configuration (`config.py`) Centralized configuration module: | Setting | Default | Override | |------------------|----------------------------|-------------------| | `db_path` | `~/.config/ghrel/repos.db` | `--db-path` flag | | `gh_token` | `GH_TOKEN` env var | — | | `server_host` | `127.0.0.1` | `--host` flag | | `server_port` | `8080` | `--port` flag | | `rss_max_items` | `50` | — | | `api_per_page` | `30` | — | On database creation, set file permissions to `0600` (owner read/write only) to avoid leaking tracked repo list. ## 9. Error Handling - **GitHub API errors** (404, 403, etc.): Log warning, skip the repo, continue with others - **Network errors**: Retry with exponential backoff (3 attempts, 1s / 2s / 4s), then skip - **Rate limit exceeded**: Log warning with reset time, pause all requests globally - **Invalid repo URL**: Validate format on `add`, show clear error: `Error: Invalid repo format 'foo'. Expected 'owner/repo'` - **Invalid token**: Clear error on first API call: `Error: GH_TOKEN is invalid (401 Unauthorized)` - **Database errors**: Wrap in user-friendly messages: `Error: Failed to open database: ` - **Port in use**: Clear error: `Error: Port 8080 is already in use` All errors go to stderr. Exit codes: `0` success, `1` general error, `2` usage error. ## 10. Testing Strategy Write tests **alongside implementation**, not after. Each module gets tests before moving to the next. | Test file | What it covers | |----------------------|----------------------------------------------------------| | `test_db.py` | Schema creation, CRUD, constraints, `mode` CHECK, `0600` permissions | | `test_github.py` | API mocking for both releases and tags, parsing, rate limiting, token validation | | `test_rss.py` | Feed generation, markdown stripping, truncation, XML escaping, item cap, tag entries (no body) | | `test_server.py` | HTTP server, feed endpoint, health check, response headers | | `test_integration.py`| Full flow: `add` → `check` → `serve` → fetch feed → verify XML (both modes) | | `test_cli.py` | CLI smoke tests: exit codes, stderr output, flag parsing, `--tags` flag | Use `pytest` with `respx` for mocking `httpx`. In-memory SQLite for DB tests. `pytest-httpserver` or `http.server` in a thread for integration tests. ## 11. Implementation Order 1. **Project scaffolding**: `pyproject.toml`, directory structure, `.gitignore` 2. **Config module** (`config.py`): Defaults, env vars, DB path management 3. **Database layer** (`db.py`): Schema, connection, CRUD functions + `test_db.py` 4. **CLI foundation** (`cli.py`): `add`, `remove`, `list` commands + `test_cli.py` 5. **GitHub fetcher** (`github.py`): API calls for releases and tags, parsing, rate limiting, token validation + `test_github.py` 6. **CLI `check` command**: Integration of fetcher with DB upsert 7. **RSS generator** (`rss.py`): xml.etree generation, markdown stripping + `test_rss.py` 8. **HTTP server** (`server.py`): stdlib http.server, `/feed.xml`, `/health` + `test_server.py` 9. **CLI `serve` command**: Wire up server 10. **Integration tests** (`test_integration.py`): End-to-end flow 11. **README**: Usage documentation ## 12. Dependencies | Package | Purpose | |--------------|----------------------------------------------| | `httpx` | HTTP client for GitHub API | | `markdown` | Convert markdown to HTML | | `bleach` | Strip HTML tags for plain text | | `pytest` | Testing framework | | `respx` | Mocking HTTP requests for tests | No heavy frameworks. `http.server` and `xml.etree.ElementTree` are stdlib. ## 13. Future Considerations (Out of Scope for MVP) - Webhook-based real-time updates instead of polling - Docker container for easy deployment - Configuration file (`ghrel.toml`) instead of CLI-only management - Support for GitLab releases - Email notifications for new releases - Web UI for managing tracked repos - Scheduled background checks (cron / systemd timer integration)