Free tool · MIT licensed

Does your multiple-choice eval depend on option order?

A single Python file, standard library only. It shows every question to your model once per cyclic rotation of the options and reports accuracy with an interval, how often the chosen answer changes, and accuracy by the slot of the correct answer.

Download option_order_check.py Download demo.jsonl

Install with pip

pip install option-order-check
option-order-check --demo --provider mock

Also on PyPI, with the source on GitHub (MIT).

Or try the single file, no API key

python option_order_check.py demo.jsonl --provider mock

The mock model has a deliberate lean toward one answer slot so you can see what a biased model looks like in the output.

Run it on your own model

Your file is JSONL, one question per line, with the correct option as a 0-based index:

{"question": "Which gas do plants absorb?", "options": ["Oxygen", "Carbon dioxide", "Helium", "Neon"], "answer": 1}

# OpenAI
export OPENAI_API_KEY=...
python option_order_check.py mytests.jsonl --provider openai --model gpt-4o-mini

# Local model through any OpenAI-compatible server (Ollama shown)
python option_order_check.py mytests.jsonl --provider openai --model llama3 \
    --base-url http://localhost:11434/v1 --api-key-env NONE

How to read the output

• Flip rate: the share of questions where the chosen answer changes with option order. It should be low. A high flip rate at decent accuracy means your score depends on where answers are placed.
• Accuracy by correct slot: if accuracy is much higher when the answer is in one slot, the model has a position preference. See the audit for what a large gap looks like (Qwen2.5-0.5B: 26.0% versus 60.3%).
• Correct under every ordering: a stricter, more honest score than mean accuracy.