# MCQ Benchmark

LLM benchmark for evaluating models on multiple-choice questions from a parquet dataset (financial reasoning).

## Features

- **Checkpoint/Resume** — progress is saved after each question. Stop and resume anytime.
- **Live Status** — real-time progress with running accuracy percentage.
- **Configurable** — YAML config for model endpoint, parameters, and benchmark settings.

## Setup

1. Copy the example config:
   ```bash
   cp config.example.yaml config.yaml
   ```

2. Edit `config.yaml` with your model endpoint and parameters.

## Usage

```bash
# Run benchmark (prompts to resume if checkpoint exists)
python3 benchmark.py

# Auto-resume from latest checkpoint
python3 benchmark.py --resume

# Start fresh, ignoring existing checkpoint
python3 benchmark.py --fresh

# Use custom config
python3 benchmark.py --config my_config.yaml
```

## Output

Results are saved to `runs/<model_name>_YYYYMMDD_HHMMSS.json` with full details:
- Overall accuracy and timing
- Per-question results (correct, incorrect, timeouts)
- Model configuration used

Checkpoints (`*.checkpoint.json`) are stored in `runs/` and auto-deleted on completion.

## Live Output Example

```
  [3048/5074] what is AAPL expecting in return for Q4 2016 based on this sheet:...  CORRECT  [67.4%]
  [3049/5074] calculate the total revenue for the fiscal year 2015...  WRONG (expected: B, got: A)  [67.4%]
```

## License

Apache 2.0