Initial commit: MCQ benchmark with checkpoint/resume support

2026-07-14 14:32:24 +02:00
commit fd40133142
5 changed files with 704 additions and 0 deletions
@@ -0,0 +1,54 @@
+# MCQ Benchmark
+
+LLM benchmark for evaluating models on multiple-choice questions from a parquet dataset (financial reasoning).
+
+## Features
+
+- **Checkpoint/Resume** — progress is saved after each question. Stop and resume anytime.
+- **Live Status** — real-time progress with running accuracy percentage.
+- **Configurable** — YAML config for model endpoint, parameters, and benchmark settings.
+
+## Setup
+
+1. Copy the example config:
+   ```bash
+   cp config.example.yaml config.yaml
+   ```
+
+2. Edit `config.yaml` with your model endpoint and parameters.
+
+## Usage
+
+```bash
+# Run benchmark (prompts to resume if checkpoint exists)
+python3 benchmark.py
+
+# Auto-resume from latest checkpoint
+python3 benchmark.py --resume
+
+# Start fresh, ignoring existing checkpoint
+python3 benchmark.py --fresh
+
+# Use custom config
+python3 benchmark.py --config my_config.yaml
+```
+
+## Output
+
+Results are saved to `runs/<model_name>_YYYYMMDD_HHMMSS.json` with full details:
+- Overall accuracy and timing
+- Per-question results (correct, incorrect, timeouts)
+- Model configuration used
+
+Checkpoints (`*.checkpoint.json`) are stored in `runs/` and auto-deleted on completion.
+
+## Live Output Example
+
+```
+  [3048/5074] what is AAPL expecting in return for Q4 2016 based on this sheet:...  CORRECT  [67.4%]
+  [3049/5074] calculate the total revenue for the fiscal year 2015...  WRONG (expected: B, got: A)  [67.4%]
+```
+
+## License
+
+Apache 2.0