Likelihood Ranking doesn't Scale Like Prompting in LLMs
Abstract Overview
This paper compares two evaluation protocols for multiple-choice question answering in large language models: direct prompted answering and likelihood ranking of declarative statements derived from the same question-answer pairs. The study covers 95 open-weight decoder-only models ranging from 0.1B to 104B parameters across 10 MCQA datasets, measuring Accuracy on Prompt Execution (APX) and Accuracy on Perplexity Score (APS). The authors observe a systematic divergence between the two protocols: prompted answering improves sharply with model scale and instruction tuning, whereas declarative-statement likelihood ranking improves slowly and remains within a narrow range. The findings suggest that prompted evaluation reflects a learned task-conditioned answer-selection policy rather than solely indicating sharper underlying likelihood preferences.
Novelty
The study introduces and evaluates declarative-statement likelihood ranking as a complementary alternative to standard MCQA likelihood scoring that removes the task-conditioned answer-selection interface. Across large-scale experiments, it demonstrates that likelihood-based rankings over declarative statements do not exhibit the same scaling trajectories as prompted answer selection.
Results
APS shows weak scaling behavior and minimal gains from instruction tuning across datasets and model families, whereas APX improves steeply with scale and eventually exceeds APS by a substantial margin. Instruction-tuned models reach the prompting break-even point at smaller parameter counts than base models, while standard question-conditioned likelihood MCQA scales similarly to prompted answering.
Key Points
- The evaluation tests 95 open-weight decoder-only models across 10 MCQA datasets to contrast prompted answering (APX) with declarative-statement likelihood ranking (APS).
- Declarative-statement likelihood accuracy scales weakly across model sizes, whereas prompted accuracy rises rapidly and benefits strongly from instruction tuning.
- The divergence indicates that likelihood preferences over controlled declarative statements and task-conditioned answer selection capture distinct, non-interchangeable evaluation signals.