Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation

• URL: http://arxiv.org/abs/2609.15467v1
• Date: Mon, 14 Sep 2026 12:28:09 GMT
• Title: Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation
• Abstract summary: Adding answer options can lower multiple-choice scores without improving assessment validity.<n>Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adding answer options can lower multiple-choice scores without improving assessment validity. Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections. Each question retains its stem, five original options and source key, and receives five options copied from other questions in the same section. Sentence-embedding retrieval proposes candidates; a language model selects existing identifiers. Deterministic verification reconstructs all 60,000 additions. A 25-model calibration exposes scoring and generation-budget effects. Five evaluations produce source-key accuracies of 34.8%-81.4%. On 981 shared questions, one API-served model falls from 93.7% with five choices to 83.1% with ten; 102 of 115 lost correct responses select borrowed options. The decrease is 24.4 percentage points on heuristically flagged negative stems and 5.9 points elsewhere. A completed human-checked audit of 200 sampled questions, with undocumented reviewer tool use, yields 47 and 31 multiple-answer judgments across the two record sets, 25 of the latter unresolved. These records support concern about ambiguity, while their dependence and incomplete reviewer-method documentation limit validation. Because order and labels also change, the paired comparison measures augmentation as implemented. The contribution is a traceable construction and an analysis of its validity limits, not evidence that lower ten-choice scores measure knowledge better.

Related papers

• From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection [12.159476982130286]
  Speech deepfakes can mimic a speaker's voice convincingly enough to deceive listeners and automated systems.<n>Most detectors end with one score per utterance, but it says little about why a borderline item should be trusted, deferred, or reviewed.<n>We answer this question with an auditable decision record that carries four aligned cues into a late calibration step.
  arXiv  Detail & Related papers  (2026-09-08T15:38:57Z)
• Abstention Errors and Segment Support in CUAD: An Auditable Contract-Extraction Case Study [0.0]
  This paper presents a retrospective, reproducible evaluation of a fixed public RoBERTa checkpoint on CUAD's published 102-contract test split.<n>Of 30,464 returned candidate strings, 19,562 occur on questions with no annotated answer.<n>The system answers 1,335 of 2,938 such questions, a 45.4% false-positive rate.
  arXiv  Detail & Related papers  (2026-09-07T21:26:53Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models [49.47300047353726]
  We introduce MemToC, a benchmark for post-tool-return arbitration with executable tools.<n>MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions.<n>Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers.
  arXiv  Detail & Related papers  (2026-08-26T18:22:03Z)
• When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs [42.32694162421078]
  We introduce CALVER, a training-free symbolic verifier that scores structured traces against Pearl's causal criteria.<n>On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%.
  arXiv  Detail & Related papers  (2026-08-04T11:45:46Z)
• Auditable Release Control for Pedagogical Leakage in LLM Tutors [0.0]
  Large language model tutors can be correct and helpful yet disclose an answer or decisive reasoning before that disclosure is authorized.<n>We formalize this state- and action-dependent failure as pedagogical leakage-aware authorization and introduce complete-mediation boundary.<n>A global A1 yields 0 majority and 54 any-utility flags, outperforming fitted Q on automatic safety and utility.
  arXiv  Detail & Related papers  (2026-08-01T08:24:30Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Evaluating Commercial AI Chatbots as News Intermediaries [85.32040752972836]
  Best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier.<n>Every model achieves its lowest accuracy on Hindi.<n> retrieval, not reasoning, failures drive over 70% of all errors.
  arXiv  Detail & Related papers  (2026-05-21T17:42:07Z)
• When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models [2.064612766965483]
  We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions)<n>Our experiments reveal several concerning findings.
  arXiv  Detail & Related papers  (2026-03-26T23:04:20Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.