54 lines
1.4 KiB
Markdown
54 lines
1.4 KiB
Markdown
# MCQ Benchmark
|
|
|
|
LLM benchmark for evaluating models on multiple-choice questions from a parquet dataset (financial reasoning).
|
|
|
|
## Features
|
|
|
|
- **Checkpoint/Resume** — progress is saved after each question. Stop and resume anytime.
|
|
- **Live Status** — real-time progress with running accuracy percentage.
|
|
- **Configurable** — YAML config for model endpoint, parameters, and benchmark settings.
|
|
|
|
## Setup
|
|
|
|
1. Copy the example config:
|
|
```bash
|
|
cp config.example.yaml config.yaml
|
|
```
|
|
|
|
2. Edit `config.yaml` with your model endpoint and parameters.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Run benchmark (prompts to resume if checkpoint exists)
|
|
python3 benchmark.py
|
|
|
|
# Auto-resume from latest checkpoint
|
|
python3 benchmark.py --resume
|
|
|
|
# Start fresh, ignoring existing checkpoint
|
|
python3 benchmark.py --fresh
|
|
|
|
# Use custom config
|
|
python3 benchmark.py --config my_config.yaml
|
|
```
|
|
|
|
## Output
|
|
|
|
Results are saved to `runs/<model_name>_YYYYMMDD_HHMMSS.json` with full details:
|
|
- Overall accuracy and timing
|
|
- Per-question results (correct, incorrect, timeouts)
|
|
- Model configuration used
|
|
|
|
Checkpoints (`*.checkpoint.json`) are stored in `runs/` and auto-deleted on completion.
|
|
|
|
## Live Output Example
|
|
|
|
```
|
|
[3048/5074] what is AAPL expecting in return for Q4 2016 based on this sheet:... CORRECT [67.4%]
|
|
[3049/5074] calculate the total revenue for the fiscal year 2015... WRONG (expected: B, got: A) [67.4%]
|
|
```
|
|
|
|
## License
|
|
|
|
Apache 2.0 |