github_release_monitor/plan.md

# GitHub Release Monitor — Project Plan

## 1. Overview

A CLI tool that tracks GitHub releases or tags from user-defined repositories and exposes them as a subscribable RSS feed. Some repos (e.g., Zammad) only use git tags without creating GitHub releases — both modes are supported.

- **Language**: Python 3.12
- **Storage**: SQLite (single file, embedded via `sqlite3` stdlib)
- **API**: GitHub REST API via `httpx`
- **Output**: RSS 2.0 XML feed served over HTTP (stdlib `http.server`)

## 2. Architecture

```
┌──────────────────────────────────────────────────┐
│                 CLI Interface                     │
│    (add, remove, update, list, check, serve)     │
├──────────────────────────────────────────────────┤
│                  Core Logic                       │
│  ┌──────────────┐  ┌──────────────┐ ┌────────┐  │
│  │  config.py    │  │  db.py       │ │github.py│  │
│  │  (defaults,   │  │  (sqlite3)   │ │(httpx)  │  │
│  │   env vars)   │  │              │ │        │  │
│  └──────────────┘  └──────┬───────┘ └───┬────┘  │
│                           │              │       │
│                  ┌────────▼──────────────▼────┐  │
│                  │         rss.py              │  │
│                  │  (xml.etree + markdown)     │  │
│                  └──────────────┬─────────────┘  │
│                                 │                 │
│                  ┌──────────────▼─────────────┐  │
│                  │      server.py              │  │
│                  │ (http.server, stdlib)       │  │
│                  │ (/feed.xml endpoint)        │  │
│                  └─────────────────────────────┘  │
└──────────────────────────────────────────────────┘
```

## 3. File Structure

```
github_release_monitor/
├── plan.md                  # This file
├── pyproject.toml           # Project metadata, dependencies
├── README.md
├── src/
│   ├── __init__.py
│   ├── cli.py               # CLI entry point (argparse)
│   ├── config.py            # Centralized config (defaults, env vars, paths)
│   ├── db.py                # SQLite schema & queries
│   ├── github.py            # GitHub API interaction
│   ├── rss.py               # RSS feed generation
│   └── server.py            # HTTP server for feed
├── tests/
│   ├── __init__.py
│   ├── test_db.py
│   ├── test_github.py
│   ├── test_rss.py
│   ├── test_server.py
│   └── test_integration.py  # End-to-end tests
└── .gitignore
```

## 4. Database Schema

### `repos` table
| Column     | Type         | Constraints                          | Description                          |
|------------|--------------|--------------------------------------|--------------------------------------|
| id         | INTEGER      | PRIMARY KEY AUTO                     | Internal identifier                  |
| owner_repo | TEXT         | NOT NULL UNIQUE                      | GitHub repo identifier (`owner/repo`)|
| mode       | TEXT         | NOT NULL DEFAULT 'release' CHECK(mode IN ('release', 'tag')) | Track releases or tags |
| created_at | DATETIME     | DEFAULT NOW()                        | When the repo was added              |

The `owner_repo` field serves as both the unique identifier and the display name. The `mode` field determines whether to fetch GitHub releases (`release`) or git tags (`tag`). Default is `release`.

### `entries` table
| Column       | Type         | Constraints        | Description                            |
|--------------|--------------|--------------------|----------------------------------------|
| id           | INTEGER      | PRIMARY KEY AUTO   | Internal identifier                    |
| repo_id      | INTEGER      | FK → repos.id      | Which repository this entry belongs to |
| kind         | TEXT         | NOT NULL CHECK(kind IN ('release', 'tag')) | Whether this is a release or tag |
| tag_name     | TEXT         | NOT NULL           | Git tag / release tag                  |
| title        | TEXT         | NOT NULL           | Release title or tag name              |
| body         | TEXT         | DEFAULT ''         | Release notes (empty for tags)         |
| published_at | DATETIME     | NOT NULL           | Release date or commit date            |
| html_url     | TEXT         | NOT NULL           | Link to release or commit on GitHub    |
| fetched_at   | DATETIME     | DEFAULT NOW()      | When we fetched this entry             |

**Unique constraint**: `(repo_id, tag_name)` to avoid duplicates.

## 5. CLI Commands

```
ghrel                          # Main command
├── --version                  # Show version
├── --verbose / -v             # Enable verbose/debug output
├── --quiet / -q               # Suppress non-essential output
├── --db-path <path>           # Override database path (default: ~/.config/ghrel/repos.db)
├── add <owner/repo> [--tags]  # Add a repository to track
├── remove <owner/repo>        # Remove a repository
├── update <owner/repo> [--tags] [--release]  # Update tracking mode
├── list                       # List all tracked repositories
├── check [--since N]          # Fetch latest releases/tags for all repos
└── serve [--port 8080] [--host 127.0.0.1]  # Start HTTP server
```

### `add <owner/repo> [--tags]`
- Accepts a single `owner/repo` argument (e.g., `golang/go`)
- `--tags` flag: track git tags instead of releases (default: releases)
- Validates the format with a regex: `^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$`
- Pings the GitHub API to verify the repo exists (with token validation)
- Inserts into `repos` table; errors on duplicate `owner_repo`
- On success: `Added golang/go (releases)` or `Added zammad/zammad (tags)`

### `remove <owner/repo>`
- Deletes from `repos` table by `owner_repo`
- Cascading delete removes associated entries
- On success: `Removed golang/go`

### `update <owner/repo> [--tags] [--release]`
- Updates the tracking mode for an existing repo
- `--tags`: switch to tracking tags
- `--release`: switch to tracking releases
- On success: `Updated zammad/zammad to track tags`

### `list`
- Displays a table of tracked repos:
  ```
  REPO            MODE      ENTRIES  LAST CHECKED
  golang/go       release   12       2025-01-15 10:30
  zammad/zammad   tag       8        2025-01-15 10:30
  ```

### `check [--since N]`
- Iterates over all repos in the database
- For each repo, fetches latest releases or tags depending on the repo's `mode`
- If `--since N` is provided (e.g., `--since 24h`), only fetches entries published in the last N hours
- Upserts into `entries` table (insert if new, skip if already known by `tag_name`)
- Reports results: `Checked 5 repos: 3 new entries found`
- Global rate-limit-aware queue: if one repo hits the rate limit, remaining repos wait

### `serve [--port 8080] [--host 127.0.0.1]`
- Starts an HTTP server using stdlib `http.server`
- Default host: `127.0.0.1` (localhost only, not exposed to network)
- Serves RSS feed at `/feed.xml`
- Serves a small index page at `/` with a link to the feed
- Health check endpoint at `/health` returns `200 OK`

### Error format
All errors go to stderr with non-zero exit code:
```
Error: <message>
```

## 6. GitHub API Integration

### Approach: Direct HTTP calls via `httpx`

- **Authentication**: Read `GH_TOKEN` from environment variable. If set, use it for authenticated requests (5000/h vs 60/h). If not set, fall back to unauthenticated.
- **Token validation**: On first API call after startup, check `GET /rate_limit` to verify the token is valid. If `401`, print a clear error: `Error: Invalid or missing GH_TOKEN. Set it with: export GH_TOKEN=your_token`
- **Pagination**: Fetch first page only (30 items). Sufficient for tracking recent entries.
- **Rate limiting**: Track `X-RateLimit-Remaining` header globally across all repos. If rate limited, pause all requests until `X-RateLimit-Reset` time. No per-repo retries — a single global wait prevents cascading 403s.

### Two API endpoints depending on `mode`

**Releases** (`mode = 'release'`):
- `GET https://api.github.com/repos/{owner}/{repo}/releases?per_page=30`
- Data extracted: `tag_name`, `name` (title), `body` (markdown), `published_at`, `html_url`

**Tags** (`mode = 'tag'`):
- `GET https://api.github.com/repos/{owner}/{repo}/tags?per_page=30`
- Data extracted: `name` (used as both `tag_name` and `title`), `zipball_url` → derive `html_url` as `https://github.com/{owner}/{repo}/tags/{name}`
- Tags have no `body` — store empty string
- Tags have no `published_at` — fetch commit date via `GET /repos/{owner}/{repo}/git/ref/tags/{name}` and use `object.commit.author.date` from the resolved commit object

## 7. RSS Feed Specification

RSS 2.0 generated with stdlib `xml.etree.ElementTree`. RSS 2.0 is simple enough that `feedgen` is unnecessary.

### Feed-level
- **title**: "GitHub Release Monitor"
- **description**: "Latest releases from tracked GitHub repositories"
- **link**: `http://{host}:{port}/` (base URL of the serving instance)
- **lastBuildDate**: Current time in RFC 822 format

### Per-item
- **title**: `[{owner/repo}] {title}`
- **description**: First 300 characters of `body`, with markdown stripped to plain text. Append `…` if truncated. For tags (no body), use: `Tag {tag_name}`
- **link**: `html_url` from the entry
- **pubDate**: `published_at` in RFC 822 format
- **guid**: `github.com/{owner}/{repo}/releases/tag/{tag_name}` (unique identifier, same format for both releases and tags)

Items sorted by `published_at` descending (newest first). **Capped at 50 items** to prevent feed bloat.

### Markdown stripping
Use `markdown` library to convert markdown to HTML, then `bleach` to strip all HTML tags and produce plain text.

## 8. Configuration (`config.py`)

Centralized configuration module:

| Setting          | Default                    | Override          |
|------------------|----------------------------|-------------------|
| `db_path`        | `~/.config/ghrel/repos.db` | `--db-path` flag  |
| `gh_token`       | `GH_TOKEN` env var         | —                 |
| `server_host`    | `127.0.0.1`                | `--host` flag     |
| `server_port`    | `8080`                     | `--port` flag     |
| `rss_max_items`  | `50`                       | —                 |
| `api_per_page`   | `30`                       | —                 |

On database creation, set file permissions to `0600` (owner read/write only) to avoid leaking tracked repo list.

## 9. Error Handling

- **GitHub API errors** (404, 403, etc.): Log warning, skip the repo, continue with others
- **Network errors**: Retry with exponential backoff (3 attempts, 1s / 2s / 4s), then skip
- **Rate limit exceeded**: Log warning with reset time, pause all requests globally
- **Invalid repo URL**: Validate format on `add`, show clear error: `Error: Invalid repo format 'foo'. Expected 'owner/repo'`
- **Invalid token**: Clear error on first API call: `Error: GH_TOKEN is invalid (401 Unauthorized)`
- **Database errors**: Wrap in user-friendly messages: `Error: Failed to open database: <reason>`
- **Port in use**: Clear error: `Error: Port 8080 is already in use`

All errors go to stderr. Exit codes: `0` success, `1` general error, `2` usage error.

## 10. Testing Strategy

Write tests **alongside implementation**, not after. Each module gets tests before moving to the next.

| Test file            | What it covers                                           |
|----------------------|----------------------------------------------------------|
| `test_db.py`         | Schema creation, CRUD, constraints, `mode` CHECK, `0600` permissions |
| `test_github.py`     | API mocking for both releases and tags, parsing, rate limiting, token validation |
| `test_rss.py`        | Feed generation, markdown stripping, truncation, XML escaping, item cap, tag entries (no body) |
| `test_server.py`     | HTTP server, feed endpoint, health check, response headers |
| `test_integration.py`| Full flow: `add` → `check` → `serve` → fetch feed → verify XML (both modes) |
| `test_cli.py`        | CLI smoke tests: exit codes, stderr output, flag parsing, `--tags` flag |

Use `pytest` with `respx` for mocking `httpx`. In-memory SQLite for DB tests. `pytest-httpserver` or `http.server` in a thread for integration tests.

## 11. Implementation Order

1. **Project scaffolding**: `pyproject.toml`, directory structure, `.gitignore`
2. **Config module** (`config.py`): Defaults, env vars, DB path management
3. **Database layer** (`db.py`): Schema, connection, CRUD functions + `test_db.py`
4. **CLI foundation** (`cli.py`): `add`, `remove`, `list` commands + `test_cli.py`
5. **GitHub fetcher** (`github.py`): API calls for releases and tags, parsing, rate limiting, token validation + `test_github.py`
6. **CLI `check` command**: Integration of fetcher with DB upsert
7. **RSS generator** (`rss.py`): xml.etree generation, markdown stripping + `test_rss.py`
8. **HTTP server** (`server.py`): stdlib http.server, `/feed.xml`, `/health` + `test_server.py`
9. **CLI `serve` command**: Wire up server
10. **Integration tests** (`test_integration.py`): End-to-end flow
11. **README**: Usage documentation

## 12. Dependencies

| Package      | Purpose                                      |
|--------------|----------------------------------------------|
| `httpx`      | HTTP client for GitHub API                   |
| `markdown`   | Convert markdown to HTML                     |
| `bleach`     | Strip HTML tags for plain text               |
| `pytest`     | Testing framework                            |
| `respx`      | Mocking HTTP requests for tests              |

No heavy frameworks. `http.server` and `xml.etree.ElementTree` are stdlib.

## 13. Future Considerations (Out of Scope for MVP)

- Webhook-based real-time updates instead of polling
- Docker container for easy deployment
- Configuration file (`ghrel.toml`) instead of CLI-only management
- Support for GitLab releases
- Email notifications for new releases
- Web UI for managing tracked repos
- Scheduled background checks (cron / systemd timer integration)