The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

• URL: http://arxiv.org/abs/2607.12963v2
• Date: Wed, 15 Jul 2026 17:00:42 GMT
• Title: The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
• Abstract summary: Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
• Score: 76.36635039699112
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.

Related papers

• Context-Length Robustness in Question Answering Models: A Comparative Empirical Study [0.0]
  We present an empirical study of context-length robustness in large language models using two benchmarks: SQuAD and HotpotQA.<n>We evaluate model accuracy as a function of total context length by systematically increasing the amount of irrelevant context while preserving the answer-bearing signal.<n>Our results show a consistent degradation in performance as context length increases, with substantially larger drops observed on multi-hop reasoning tasks compared to single-span extraction tasks.
  arXiv  Detail & Related papers  (2026-03-16T17:14:05Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions [0.0]
  Large Language Models (LLMs) often exhibit inconsistent behavior when answering paraphrased questions.<n>We introduce RoParQ, a benchmark to evaluate cross-paraphrase consistency in closed-book multiple-choice QA.<n>We also propose XParaCon, a novel evaluation metric that quantifies a model's robustness.
  arXiv  Detail & Related papers  (2025-11-26T16:40:53Z)
• NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction [7.856998585396422]
  We present a systematic evaluation of state-of-the-art models for veracity prediction on numerical claims and evidence pairs.<n>Results indicate that even leading proprietary systems experience accuracy drops of up to 62% under certain perturbations.<n>These findings highlight critical limitations in numerical fact-checking and suggest that robustness remains an open challenge for current language models.
  arXiv  Detail & Related papers  (2025-11-13T05:09:52Z)
• Are vision language models robust to uncertain inputs? [5.249651874118556]
  We show that newer and larger vision language models exhibit improved robustness compared to earlier models, but still suffer from a tendency to strictly follow instructions.<n>For natural images such as ImageNet, this limitation can be overcome without pipeline modifications.<n>We propose a novel mechanism based on caption diversity to reveal a model's internal uncertainty.
  arXiv  Detail & Related papers  (2025-05-17T03:16:49Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [49.15931834209624]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world.<n>We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.<n>By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models [80.23791222509644]
  Inconsistent AI models are considered brittle and untrustworthy by human users. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks. We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
  arXiv  Detail & Related papers  (2023-03-28T16:57:12Z)
• Language model acceptability judgements are not always robust to context [30.868765627701457]
  We investigate the stability of language models' performance on targeted syntactic evaluations. We find that model judgements are generally robust when placed in randomly sampled linguistic contexts. We show that these changes in model performance are not explainable by simple features matching the context and the test inputs.
  arXiv  Detail & Related papers  (2022-12-18T00:11:06Z)
• Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures [62.562760228942054]
  Existing approaches to improve robustness against dataset biases mostly focus on changing the training objective. We propose to augment the input sentences in the training data with their corresponding predicate-argument structures. We show that without targeting a specific bias, our sentence augmentation improves the robustness of transformer models against multiple biases.
  arXiv  Detail & Related papers  (2020-10-23T16:22:05Z)
• Understanding Neural Abstractive Summarization Models via Uncertainty [54.37665950633147]
  seq2seq abstractive summarization models generate text in a free-form manner. We study the entropy, or uncertainty, of the model's token-level predictions. We show that uncertainty is a useful perspective for analyzing summarization and text generation models more broadly.
  arXiv  Detail & Related papers  (2020-10-15T16:57:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.