Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoning

• URL: http://arxiv.org/abs/2608.27756v1
• Date: Thu, 27 Aug 2026 22:36:49 GMT
• Title: Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoning
• Abstract summary: Self-contained linguistic olympiad puzzles provide a controlled setting where all answers derive from expert-designed context examples.<n>We leverage this to introduce a diagnostic framework for analyzing individual context examples.<n>Using 53 UK Linguistics Olympiad puzzles, we generate two modified variants by deleting a single context example.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Determining whether large language models derive answers from context or prior knowledge remains a fundamental challenge. Self-contained linguistic olympiad puzzles provide a controlled setting where all answers derive solely from expert-designed context examples without external knowledge. Removing individual context examples can eliminate information needed for specific questions while leaving the rest of the puzzle unchanged. We leverage this to introduce a diagnostic framework for analyzing individual context examples. Using 53 UK Linguistics Olympiad puzzles, we generate two modified variants by deleting a single context example: (1) uniform random deletion, and (2) targeted deletion (inspired by error-correcting codes) to remove a structurally load-bearing example uniquely carrying necessary information. We formalize this impact using a Question Damage Score to classify puzzles as fragile or robust. Evaluating three frontier LLMs under instructions to abstain when information is insufficient, we find they rarely abstain, often continuing to produce correct answers after load-bearing context is removed. These findings motivate further investigation into context-based reasoning, prior knowledge, memorization, and linguistic inference. Beyond abstention, the framework enables fine-grained analyses of context reliance, including causal interventions, stopping-set analysis, targeted contamination studies, and mechanistic interpretability.

Related papers

• Causally Evaluating the Learnability of Formal Language Tasks [59.73001113068827]
  We rigorously investigate the relationship between data frequency and learnability.<n>We derive Kullback-Leibler divergence metrics to measure the learnability of specific sub-tasks.<n>Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions.
  arXiv  Detail & Related papers  (2026-06-08T17:58:36Z)
• Learning the Cue or Learning the Word? Analyzing Generalization in Metaphor Detection for Verbs [54.86138668387175]
  We analyze metaphor detection through RoBERTa, the shared backbone of many state-of-the-art systems, focusing on English verbs.<n>We introduce a controlled lexical hold-out setup where all instances of selected target lemmas are strictly excluded from fine-tuning, and compare predictions on these Held-out lemmas against Exposed lemmas.<n>While the model performs best on Exposed lemmas, it maintains robust performance on Held-out lemmas.
  arXiv  Detail & Related papers  (2026-04-15T10:48:09Z)
• Beyond Memorization: Testing LLM Reasoning on Unseen Theory of Computation Tasks [8.210112631285666]
  Large language models (LLMs) have demonstrated strong performance on formal language tasks.<n>We introduce a benchmark for deterministic finite automata (DFA) construction from regular languages.<n>We show that models achieve perfect accuracy on factual questions and 84-90% on seen tasks, but accuracy drops sharply on unseen problems.
  arXiv  Detail & Related papers  (2026-01-19T21:00:31Z)
• Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentations [65.11348389219887]
  We introduce Dialectic-RAG (DRAG), a modular approach that evaluates retrieved information by comparing, contrasting, and resolving conflicting perspectives.<n>We show the impact of our framework both as an in-context learning strategy and for constructing demonstrations to instruct smaller models.
  arXiv  Detail & Related papers  (2025-04-07T06:55:15Z)
• Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models [1.8759938228229573]
  We investigate the challenge of generating adversarial examples to test the robustness of text classification algorithms.<n>We focus on simulation of content moderation by setting realistic limits on the number of queries an attacker is allowed to attempt.
  arXiv  Detail & Related papers  (2024-10-28T11:46:30Z)
• InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification [60.10193972862099]
  This work proposes a framework to characterize and recover simplification-induced information loss in form of question-and-answer pairs. QA pairs are designed to help readers deepen their knowledge of a text.
  arXiv  Detail & Related papers  (2024-01-29T19:00:01Z)
• Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion [57.43781399856913]
  This work adopts the linguistic framework of Questions Under Discussion (QUD) for discourse analysis. We characterize relationships between sentences as free-form questions, in contrast to exhaustive fine-grained questions. We develop the first-of-its-kind QUD that derives a dependency structure of questions over full documents.
  arXiv  Detail & Related papers  (2022-10-12T03:53:12Z)
• A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions [35.24301299033675]
  We propose a framework for investigating fine-grained language understanding in visually grounded dialogues. We focus on OneCommon Corpus citepudagawa 2019natural,udagawa 2020annotated, a simple yet challenging common grounding dataset. We analyze their linguistic structures based on textitspatial expressions and provide comprehensive and reliable annotation for 600 dialogues.
  arXiv  Detail & Related papers  (2020-10-07T02:50:38Z)
• On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering [120.64104995052189]
  We present a dataset that takes a step towards addressing this problem in that it contains questions expressed in two languages. Measuring reasoning directly encourages generalization by penalizing answers that are coincidentally correct. The dataset reflects the scene-text version of the VQA problem, and the reasoning evaluation can be seen as a text-based version of a referring expression challenge.
  arXiv  Detail & Related papers  (2020-02-24T13:02:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.