github_release_monitor/plan.md

GitHub Release Monitor — Project Plan

1. Overview

A CLI tool that tracks GitHub releases or tags from user-defined repositories and exposes them as a subscribable RSS feed. Some repos (e.g., Zammad) only use git tags without creating GitHub releases — both modes are supported.

• Language: Python 3.12
• Storage: SQLite (single file, embedded via sqlite3 stdlib)
• API: GitHub REST API via httpx
• Output: RSS 2.0 XML feed served over HTTP (stdlib http.server)

2. Architecture

┌──────────────────────────────────────────────────┐
│                 CLI Interface                     │
│    (add, remove, update, list, check, serve)     │
├──────────────────────────────────────────────────┤
│                  Core Logic                       │
│  ┌──────────────┐  ┌──────────────┐ ┌────────┐  │
│  │  config.py    │  │  db.py       │ │github.py│  │
│  │  (defaults,   │  │  (sqlite3)   │ │(httpx)  │  │
│  │   env vars)   │  │              │ │        │  │
│  └──────────────┘  └──────┬───────┘ └───┬────┘  │
│                           │              │       │
│                  ┌────────▼──────────────▼────┐  │
│                  │         rss.py              │  │
│                  │  (xml.etree + markdown)     │  │
│                  └──────────────┬─────────────┘  │
│                                 │                 │
│                  ┌──────────────▼─────────────┐  │
│                  │      server.py              │  │
│                  │ (http.server, stdlib)       │  │
│                  │ (/feed.xml endpoint)        │  │
│                  └─────────────────────────────┘  │
└──────────────────────────────────────────────────┘

3. File Structure

github_release_monitor/
├── plan.md                  # This file
├── pyproject.toml           # Project metadata, dependencies
├── README.md
├── src/
│   ├── __init__.py
│   ├── cli.py               # CLI entry point (argparse)
│   ├── config.py            # Centralized config (defaults, env vars, paths)
│   ├── db.py                # SQLite schema & queries
│   ├── github.py            # GitHub API interaction
│   ├── rss.py               # RSS feed generation
│   └── server.py            # HTTP server for feed
├── tests/
│   ├── __init__.py
│   ├── test_db.py
│   ├── test_github.py
│   ├── test_rss.py
│   ├── test_server.py
│   └── test_integration.py  # End-to-end tests
└── .gitignore

4. Database Schema

repos table

Column	Type	Constraints	Description
id	INTEGER	PRIMARY KEY AUTO	Internal identifier
owner_repo	TEXT	NOT NULL UNIQUE	GitHub repo identifier (owner/repo)
mode	TEXT	NOT NULL DEFAULT 'release' CHECK(mode IN ('release', 'tag'))	Track releases or tags
created_at	DATETIME	DEFAULT NOW()	When the repo was added

The owner_repo field serves as both the unique identifier and the display name. The mode field determines whether to fetch GitHub releases (release) or git tags (tag). Default is release.

entries table

Column	Type	Constraints	Description
id	INTEGER	PRIMARY KEY AUTO	Internal identifier
repo_id	INTEGER	FK → repos.id	Which repository this entry belongs to
kind	TEXT	NOT NULL CHECK(kind IN ('release', 'tag'))	Whether this is a release or tag
tag_name	TEXT	NOT NULL	Git tag / release tag
title	TEXT	NOT NULL	Release title or tag name
body	TEXT	DEFAULT ''	Release notes (empty for tags)
published_at	DATETIME	NOT NULL	Release date or commit date
html_url	TEXT	NOT NULL	Link to release or commit on GitHub
fetched_at	DATETIME	DEFAULT NOW()	When we fetched this entry

Unique constraint: (repo_id, tag_name) to avoid duplicates.

5. CLI Commands

ghrel                          # Main command
├── --version                  # Show version
├── --verbose / -v             # Enable verbose/debug output
├── --quiet / -q               # Suppress non-essential output
├── --db-path <path>           # Override database path (default: ~/.config/ghrel/repos.db)
├── add <owner/repo> [--tags]  # Add a repository to track
├── remove <owner/repo>        # Remove a repository
├── update <owner/repo> [--tags] [--release]  # Update tracking mode
├── list                       # List all tracked repositories
├── check [--since N]          # Fetch latest releases/tags for all repos
└── serve [--port 8080] [--host 127.0.0.1]  # Start HTTP server

add <owner/repo> [--tags]

• Accepts a single owner/repo argument (e.g., golang/go)
• --tags flag: track git tags instead of releases (default: releases)
• Validates the format with a regex: ^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$
• Pings the GitHub API to verify the repo exists (with token validation)
• Inserts into repos table; errors on duplicate owner_repo
• On success: Added golang/go (releases) or Added zammad/zammad (tags)

remove <owner/repo>

• Deletes from repos table by owner_repo
• Cascading delete removes associated entries
• On success: Removed golang/go

update <owner/repo> [--tags] [--release]

• Updates the tracking mode for an existing repo
• --tags: switch to tracking tags
• --release: switch to tracking releases
• On success: Updated zammad/zammad to track tags

list

• Displays a table of tracked repos:

  REPO            MODE      ENTRIES  LAST CHECKED
  golang/go       release   12       2025-01-15 10:30
  zammad/zammad   tag       8        2025-01-15 10:30
  

check [--since N]

• Iterates over all repos in the database
• For each repo, fetches latest releases or tags depending on the repo's mode
• If --since N is provided (e.g., --since 24h), only fetches entries published in the last N hours
• Upserts into entries table (insert if new, skip if already known by tag_name)
• Reports results: Checked 5 repos: 3 new entries found
• Global rate-limit-aware queue: if one repo hits the rate limit, remaining repos wait

serve [--port 8080] [--host 127.0.0.1]

• Starts an HTTP server using stdlib http.server
• Default host: 127.0.0.1 (localhost only, not exposed to network)
• Serves RSS feed at /feed.xml
• Serves a small index page at / with a link to the feed
• Health check endpoint at /health returns 200 OK

Error format

All errors go to stderr with non-zero exit code:

Error: <message>

6. GitHub API Integration

Approach: Direct HTTP calls via httpx

• Authentication: Read GH_TOKEN from environment variable. If set, use it for authenticated requests (5000/h vs 60/h). If not set, fall back to unauthenticated.
• Token validation: On first API call after startup, check GET /rate_limit to verify the token is valid. If 401, print a clear error: Error: Invalid or missing GH_TOKEN. Set it with: export GH_TOKEN=your_token
• Pagination: Fetch first page only (30 items). Sufficient for tracking recent entries.
• Rate limiting: Track X-RateLimit-Remaining header globally across all repos. If rate limited, pause all requests until X-RateLimit-Reset time. No per-repo retries — a single global wait prevents cascading 403s.

Two API endpoints depending on mode

Releases (mode = 'release'):

• GET https://api.github.com/repos/{owner}/{repo}/releases?per_page=30
• Data extracted: tag_name, name (title), body (markdown), published_at, html_url

Tags (mode = 'tag'):

• GET https://api.github.com/repos/{owner}/{repo}/tags?per_page=30
• Data extracted: name (used as both tag_name and title), zipball_url → derive html_url as https://github.com/{owner}/{repo}/tags/{name}
• Tags have no body — store empty string
• Tags have no published_at — fetch commit date via GET /repos/{owner}/{repo}/git/ref/tags/{name} and use object.commit.author.date from the resolved commit object

7. RSS Feed Specification

RSS 2.0 generated with stdlib xml.etree.ElementTree. RSS 2.0 is simple enough that feedgen is unnecessary.

Feed-level

• title: "GitHub Release Monitor"
• description: "Latest releases from tracked GitHub repositories"
• link: http://{host}:{port}/ (base URL of the serving instance)
• lastBuildDate: Current time in RFC 822 format

Per-item

• title: [{owner/repo}] {title}
• description: First 300 characters of body, with markdown stripped to plain text. Append … if truncated. For tags (no body), use: Tag {tag_name}
• link: html_url from the entry
• pubDate: published_at in RFC 822 format
• guid: github.com/{owner}/{repo}/releases/tag/{tag_name} (unique identifier, same format for both releases and tags)

Items sorted by published_at descending (newest first). Capped at 50 items to prevent feed bloat.

Markdown stripping

Use markdown library to convert markdown to HTML, then bleach to strip all HTML tags and produce plain text.

8. Configuration (config.py)

Centralized configuration module:

Setting	Default	Override
db_path	~/.config/ghrel/repos.db	--db-path flag
gh_token	GH_TOKEN env var	—
server_host	127.0.0.1	--host flag
server_port	8080	--port flag
rss_max_items	50	—
api_per_page	30	—

On database creation, set file permissions to 0600 (owner read/write only) to avoid leaking tracked repo list.

9. Error Handling

• GitHub API errors (404, 403, etc.): Log warning, skip the repo, continue with others
• Network errors: Retry with exponential backoff (3 attempts, 1s / 2s / 4s), then skip
• Rate limit exceeded: Log warning with reset time, pause all requests globally
• Invalid repo URL: Validate format on add, show clear error: Error: Invalid repo format 'foo'. Expected 'owner/repo'
• Invalid token: Clear error on first API call: Error: GH_TOKEN is invalid (401 Unauthorized)
• Database errors: Wrap in user-friendly messages: Error: Failed to open database: <reason>
• Port in use: Clear error: Error: Port 8080 is already in use

All errors go to stderr. Exit codes: 0 success, 1 general error, 2 usage error.

10. Testing Strategy

Write tests alongside implementation, not after. Each module gets tests before moving to the next.

Test file	What it covers
test_db.py	Schema creation, CRUD, constraints, mode CHECK, 0600 permissions
test_github.py	API mocking for both releases and tags, parsing, rate limiting, token validation
test_rss.py	Feed generation, markdown stripping, truncation, XML escaping, item cap, tag entries (no body)
test_server.py	HTTP server, feed endpoint, health check, response headers
test_integration.py	Full flow: add → check → serve → fetch feed → verify XML (both modes)
test_cli.py	CLI smoke tests: exit codes, stderr output, flag parsing, --tags flag

Use pytest with respx for mocking httpx. In-memory SQLite for DB tests. pytest-httpserver or http.server in a thread for integration tests.

11. Implementation Order

1. Project scaffolding: pyproject.toml, directory structure, .gitignore
2. Config module (config.py): Defaults, env vars, DB path management
3. Database layer (db.py): Schema, connection, CRUD functions + test_db.py
4. CLI foundation (cli.py): add, remove, list commands + test_cli.py
5. GitHub fetcher (github.py): API calls for releases and tags, parsing, rate limiting, token validation + test_github.py
6. CLI check command: Integration of fetcher with DB upsert
7. RSS generator (rss.py): xml.etree generation, markdown stripping + test_rss.py
8. HTTP server (server.py): stdlib http.server, /feed.xml, /health + test_server.py
9. CLI serve command: Wire up server
10. Integration tests (test_integration.py): End-to-end flow
11. README: Usage documentation

12. Dependencies

Package	Purpose
httpx	HTTP client for GitHub API
markdown	Convert markdown to HTML
bleach	Strip HTML tags for plain text
pytest	Testing framework
respx	Mocking HTTP requests for tests

No heavy frameworks. http.server and xml.etree.ElementTree are stdlib.

13. Future Considerations (Out of Scope for MVP)

• Webhook-based real-time updates instead of polling
• Docker container for easy deployment
• Configuration file (ghrel.toml) instead of CLI-only management
• Support for GitLab releases
• Email notifications for new releases
• Web UI for managing tracked repos
• Scheduled background checks (cron / systemd timer integration)