ARIA/github_release_monitor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
07f7370b6042c9995d45095a6ebf14d7c1c9083d
github_release_monitor/plan.md
T
ARIA 2a5f11bd9b Initial commit: GitHub Release Monitor CLI tool 
CLI tool that tracks GitHub releases or tags from user-defined repositories
and exposes them as a subscribable RSS 2.0 feed.

Features:
- Track releases or tags per repository
- SQLite storage with automatic deduplication
- RSS 2.0 feed with markdown-stripped release notes
- Built-in HTTP server for local feed access
- Rate-limit-aware GitHub API client
- 64 passing tests
2026-07-28 11:56:10 +02:00

275 lines
15 KiB
Markdown
Raw Blame History

# GitHub Release Monitor — Project Plan
## 1. Overview
A CLI tool that tracks GitHub releases or tags from user-defined repositories and exposes them as a subscribable RSS feed. Some repos (e.g., Zammad) only use git tags without creating GitHub releases — both modes are supported.
- **Language**: Python 3.12
- **Storage**: SQLite (single file, embedded via `sqlite3` stdlib)
- **API**: GitHub REST API via `httpx`
- **Output**: RSS 2.0 XML feed served over HTTP (stdlib `http.server`)
## 2. Architecture
```
┌──────────────────────────────────────────────────┐
│ CLI Interface │
│ (add, remove, update, list, check, serve) │
├──────────────────────────────────────────────────┤
│ Core Logic │
│ ┌──────────────┐ ┌──────────────┐ ┌────────┐ │
│ │ config.py │ │ db.py │ │github.py│ │
│ │ (defaults, │ │ (sqlite3) │ │(httpx) │ │
│ │ env vars) │ │ │ │ │ │
│ └──────────────┘ └──────┬───────┘ └───┬────┘ │
│ │ │ │
│ ┌────────▼──────────────▼────┐ │
│ │ rss.py │ │
│ │ (xml.etree + markdown) │ │
│ └──────────────┬─────────────┘ │
│ │ │
│ ┌──────────────▼─────────────┐ │
│ │ server.py │ │
│ │ (http.server, stdlib) │ │
│ │ (/feed.xml endpoint) │ │
│ └─────────────────────────────┘ │
└──────────────────────────────────────────────────┘
```
## 3. File Structure
```
github_release_monitor/
├── plan.md # This file
├── pyproject.toml # Project metadata, dependencies
├── README.md
├── src/
│ ├── __init__.py
│ ├── cli.py # CLI entry point (argparse)
│ ├── config.py # Centralized config (defaults, env vars, paths)
│ ├── db.py # SQLite schema & queries
│ ├── github.py # GitHub API interaction
│ ├── rss.py # RSS feed generation
│ └── server.py # HTTP server for feed
├── tests/
│ ├── __init__.py
│ ├── test_db.py
│ ├── test_github.py
│ ├── test_rss.py
│ ├── test_server.py
│ └── test_integration.py # End-to-end tests
└── .gitignore
```
## 4. Database Schema
### `repos` table
| Column | Type | Constraints | Description |
|------------|--------------|--------------------------------------|--------------------------------------|
| id | INTEGER | PRIMARY KEY AUTO | Internal identifier |
| owner_repo | TEXT | NOT NULL UNIQUE | GitHub repo identifier (`owner/repo`)|
| mode | TEXT | NOT NULL DEFAULT 'release' CHECK(mode IN ('release', 'tag')) | Track releases or tags |
| created_at | DATETIME | DEFAULT NOW() | When the repo was added |
The `owner_repo` field serves as both the unique identifier and the display name. The `mode` field determines whether to fetch GitHub releases (`release`) or git tags (`tag`). Default is `release`.
### `entries` table
| Column | Type | Constraints | Description |
|--------------|--------------|--------------------|----------------------------------------|
| id | INTEGER | PRIMARY KEY AUTO | Internal identifier |
| repo_id | INTEGER | FK → repos.id | Which repository this entry belongs to |
| kind | TEXT | NOT NULL CHECK(kind IN ('release', 'tag')) | Whether this is a release or tag |
| tag_name | TEXT | NOT NULL | Git tag / release tag |
| title | TEXT | NOT NULL | Release title or tag name |
| body | TEXT | DEFAULT '' | Release notes (empty for tags) |
| published_at | DATETIME | NOT NULL | Release date or commit date |
| html_url | TEXT | NOT NULL | Link to release or commit on GitHub |
| fetched_at | DATETIME | DEFAULT NOW() | When we fetched this entry |
**Unique constraint**: `(repo_id, tag_name)` to avoid duplicates.
## 5. CLI Commands
```
ghrel # Main command
├── --version # Show version
├── --verbose / -v # Enable verbose/debug output
├── --quiet / -q # Suppress non-essential output
├── --db-path <path> # Override database path (default: ~/.config/ghrel/repos.db)
├── add <owner/repo> [--tags] # Add a repository to track
├── remove <owner/repo> # Remove a repository
├── update <owner/repo> [--tags] [--release] # Update tracking mode
├── list # List all tracked repositories
├── check [--since N] # Fetch latest releases/tags for all repos
└── serve [--port 8080] [--host 127.0.0.1] # Start HTTP server
```
### `add <owner/repo> [--tags]`
- Accepts a single `owner/repo` argument (e.g., `golang/go`)
- `--tags` flag: track git tags instead of releases (default: releases)
- Validates the format with a regex: `^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$`
- Pings the GitHub API to verify the repo exists (with token validation)
- Inserts into `repos` table; errors on duplicate `owner_repo`
- On success: `Added golang/go (releases)` or `Added zammad/zammad (tags)`
### `remove <owner/repo>`
- Deletes from `repos` table by `owner_repo`
- Cascading delete removes associated entries
- On success: `Removed golang/go`
### `update <owner/repo> [--tags] [--release]`
- Updates the tracking mode for an existing repo
- `--tags`: switch to tracking tags
- `--release`: switch to tracking releases
- On success: `Updated zammad/zammad to track tags`
### `list`
- Displays a table of tracked repos:
```
REPO MODE ENTRIES LAST CHECKED
golang/go release 12 2025-01-15 10:30
zammad/zammad tag 8 2025-01-15 10:30
```
### `check [--since N]`
- Iterates over all repos in the database
- For each repo, fetches latest releases or tags depending on the repo's `mode`
- If `--since N` is provided (e.g., `--since 24h`), only fetches entries published in the last N hours
- Upserts into `entries` table (insert if new, skip if already known by `tag_name`)
- Reports results: `Checked 5 repos: 3 new entries found`
- Global rate-limit-aware queue: if one repo hits the rate limit, remaining repos wait
### `serve [--port 8080] [--host 127.0.0.1]`
- Starts an HTTP server using stdlib `http.server`
- Default host: `127.0.0.1` (localhost only, not exposed to network)
- Serves RSS feed at `/feed.xml`
- Serves a small index page at `/` with a link to the feed
- Health check endpoint at `/health` returns `200 OK`
### Error format
All errors go to stderr with non-zero exit code:
```
Error: <message>
```
## 6. GitHub API Integration
### Approach: Direct HTTP calls via `httpx`
- **Authentication**: Read `GH_TOKEN` from environment variable. If set, use it for authenticated requests (5000/h vs 60/h). If not set, fall back to unauthenticated.
- **Token validation**: On first API call after startup, check `GET /rate_limit` to verify the token is valid. If `401`, print a clear error: `Error: Invalid or missing GH_TOKEN. Set it with: export GH_TOKEN=your_token`
- **Pagination**: Fetch first page only (30 items). Sufficient for tracking recent entries.
- **Rate limiting**: Track `X-RateLimit-Remaining` header globally across all repos. If rate limited, pause all requests until `X-RateLimit-Reset` time. No per-repo retries — a single global wait prevents cascading 403s.
### Two API endpoints depending on `mode`
**Releases** (`mode = 'release'`):
- `GET https://api.github.com/repos/{owner}/{repo}/releases?per_page=30`
- Data extracted: `tag_name`, `name` (title), `body` (markdown), `published_at`, `html_url`
**Tags** (`mode = 'tag'`):
- `GET https://api.github.com/repos/{owner}/{repo}/tags?per_page=30`
- Data extracted: `name` (used as both `tag_name` and `title`), `zipball_url` → derive `html_url` as `https://github.com/{owner}/{repo}/tags/{name}`
- Tags have no `body` — store empty string
- Tags have no `published_at` — fetch commit date via `GET /repos/{owner}/{repo}/git/ref/tags/{name}` and use `object.commit.author.date` from the resolved commit object
## 7. RSS Feed Specification
RSS 2.0 generated with stdlib `xml.etree.ElementTree`. RSS 2.0 is simple enough that `feedgen` is unnecessary.
### Feed-level
- **title**: "GitHub Release Monitor"
- **description**: "Latest releases from tracked GitHub repositories"
- **link**: `http://{host}:{port}/` (base URL of the serving instance)
- **lastBuildDate**: Current time in RFC 822 format
### Per-item
- **title**: `[{owner/repo}] {title}`
- **description**: First 300 characters of `body`, with markdown stripped to plain text. Append `…` if truncated. For tags (no body), use: `Tag {tag_name}`
- **link**: `html_url` from the entry
- **pubDate**: `published_at` in RFC 822 format
- **guid**: `github.com/{owner}/{repo}/releases/tag/{tag_name}` (unique identifier, same format for both releases and tags)
Items sorted by `published_at` descending (newest first). **Capped at 50 items** to prevent feed bloat.
### Markdown stripping
Use `markdown` library to convert markdown to HTML, then `bleach` to strip all HTML tags and produce plain text.
## 8. Configuration (`config.py`)
Centralized configuration module:
| Setting | Default | Override |
|------------------|----------------------------|-------------------|
| `db_path` | `~/.config/ghrel/repos.db` | `--db-path` flag |
| `gh_token` | `GH_TOKEN` env var | — |
| `server_host` | `127.0.0.1` | `--host` flag |
| `server_port` | `8080` | `--port` flag |
| `rss_max_items` | `50` | — |
| `api_per_page` | `30` | — |
On database creation, set file permissions to `0600` (owner read/write only) to avoid leaking tracked repo list.
## 9. Error Handling
- **GitHub API errors** (404, 403, etc.): Log warning, skip the repo, continue with others
- **Network errors**: Retry with exponential backoff (3 attempts, 1s / 2s / 4s), then skip
- **Rate limit exceeded**: Log warning with reset time, pause all requests globally
- **Invalid repo URL**: Validate format on `add`, show clear error: `Error: Invalid repo format 'foo'. Expected 'owner/repo'`
- **Invalid token**: Clear error on first API call: `Error: GH_TOKEN is invalid (401 Unauthorized)`
- **Database errors**: Wrap in user-friendly messages: `Error: Failed to open database: <reason>`
- **Port in use**: Clear error: `Error: Port 8080 is already in use`
All errors go to stderr. Exit codes: `0` success, `1` general error, `2` usage error.
## 10. Testing Strategy
Write tests **alongside implementation**, not after. Each module gets tests before moving to the next.
| Test file | What it covers |
|----------------------|----------------------------------------------------------|
| `test_db.py` | Schema creation, CRUD, constraints, `mode` CHECK, `0600` permissions |
| `test_github.py` | API mocking for both releases and tags, parsing, rate limiting, token validation |
| `test_rss.py` | Feed generation, markdown stripping, truncation, XML escaping, item cap, tag entries (no body) |
| `test_server.py` | HTTP server, feed endpoint, health check, response headers |
| `test_integration.py`| Full flow: `add` → `check` → `serve` → fetch feed → verify XML (both modes) |
| `test_cli.py` | CLI smoke tests: exit codes, stderr output, flag parsing, `--tags` flag |
Use `pytest` with `respx` for mocking `httpx`. In-memory SQLite for DB tests. `pytest-httpserver` or `http.server` in a thread for integration tests.
## 11. Implementation Order
1. **Project scaffolding**: `pyproject.toml`, directory structure, `.gitignore`
2. **Config module** (`config.py`): Defaults, env vars, DB path management
3. **Database layer** (`db.py`): Schema, connection, CRUD functions + `test_db.py`
4. **CLI foundation** (`cli.py`): `add`, `remove`, `list` commands + `test_cli.py`
5. **GitHub fetcher** (`github.py`): API calls for releases and tags, parsing, rate limiting, token validation + `test_github.py`
6. **CLI `check` command**: Integration of fetcher with DB upsert
7. **RSS generator** (`rss.py`): xml.etree generation, markdown stripping + `test_rss.py`
8. **HTTP server** (`server.py`): stdlib http.server, `/feed.xml`, `/health` + `test_server.py`
9. **CLI `serve` command**: Wire up server
10. **Integration tests** (`test_integration.py`): End-to-end flow
11. **README**: Usage documentation
## 12. Dependencies
| Package | Purpose |
|--------------|----------------------------------------------|
| `httpx` | HTTP client for GitHub API |
| `markdown` | Convert markdown to HTML |
| `bleach` | Strip HTML tags for plain text |
| `pytest` | Testing framework |
| `respx` | Mocking HTTP requests for tests |
No heavy frameworks. `http.server` and `xml.etree.ElementTree` are stdlib.
## 13. Future Considerations (Out of Scope for MVP)
- Webhook-based real-time updates instead of polling
- Docker container for easy deployment
- Configuration file (`ghrel.toml`) instead of CLI-only management
- Support for GitLab releases
- Email notifications for new releases
- Web UI for managing tracked repos
- Scheduled background checks (cron / systemd timer integration)
Reference in New Issue View Git Blame Copy Permalink