Context-Length Robustness in Question Answering Models: A Comparative Empirical Study

• URL: http://arxiv.org/abs/2603.15723v1
• Date: Mon, 16 Mar 2026 17:14:05 GMT
• Title: Context-Length Robustness in Question Answering Models: A Comparative Empirical Study
• Abstract summary: We present an empirical study of context-length robustness in large language models using two benchmarks: SQuAD and HotpotQA.<n>We evaluate model accuracy as a function of total context length by systematically increasing the amount of irrelevant context while preserving the answer-bearing signal.<n>Our results show a consistent degradation in performance as context length increases, with substantially larger drops observed on multi-hop reasoning tasks compared to single-span extraction tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly deployed in settings where relevant information is embedded within long and noisy contexts. Despite this, robustness to growing context length remains poorly understood across different question answering tasks. In this work, we present a controlled empirical study of context-length robustness in large language models using two widely used benchmarks: SQuAD and HotpotQA. We evaluate model accuracy as a function of total context length by systematically increasing the amount of irrelevant context while preserving the answer-bearing signal. This allows us to isolate the effect of context length from changes in task difficulty. Our results show a consistent degradation in performance as context length increases, with substantially larger drops observed on multi-hop reasoning tasks compared to single-span extraction tasks. In particular, HotpotQA exhibits nearly twice the accuracy degradation of SQuAD under equivalent context expansions. These findings highlight task-dependent differences in robustness and suggest that multi-hop reasoning is especially vulnerable to context dilution. We argue that context-length robustness should be evaluated explicitly when assessing model reliability, especially for applications involving long documents or retrieval-augmented generation.

Related papers

• The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context [76.36635039699112]
  Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
  arXiv  Detail & Related papers  (2026-07-14T17:01:12Z)
• Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench [6.490490294075993]
  We propose PredicateLongBench as a benchmark for evaluating large language models.<n>It asks models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints.<n>We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes.
  arXiv  Detail & Related papers  (2026-07-09T09:28:06Z)
• EntropyLong: Effective Long-Context Training via Predictive Uncertainty [32.39649991665767]
  Training long-context language models to capture long-range dependencies requires specialized data construction.<n>We propose Entropy, a novel data construction method that leverages predictive uncertainty to verify dependency quality.<n>Our approach identifies high-entropy positions in documents, retrieves semantically relevant contexts from large corpora, and verifies their utility by assessing whether they reduce prediction entropy.
  arXiv  Detail & Related papers  (2025-09-26T02:38:08Z)
• A Controllable Examination for Long-Context Language Models [62.845852724511964]
  This study introduces $textbfLongBioBench, a benchmark for evaluating long-context language models.<n>We show that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results.<n>Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence.
  arXiv  Detail & Related papers  (2025-06-03T14:23:06Z)
• What is Wrong with Perplexity for Long-context Language Modeling? [71.34933096461124]
  Long-context inputs are crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning.<n>Perplexity (PPL) has proven unreliable for assessing long-context capabilities.<n>We propose bfLongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them.
  arXiv  Detail & Related papers  (2024-10-31T09:39:28Z)
• GATEAU: Selecting Influential Samples for Long Context Alignment [59.579128690086385]
  GATEAU identifies influential samples enriched with long-range dependency relations.<n>Model trained on selected samples exhibits better instruction-following and long-context understanding capabilities.
  arXiv  Detail & Related papers  (2024-10-21T04:30:53Z)
• A Controlled Study on Long Context Extension and Generalization in LLMs [85.4758128256142]
  Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data.
  arXiv  Detail & Related papers  (2024-09-18T17:53:17Z)
• Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP [32.19010113355365]
  We argue that conflating different tasks by their context length is unproductive.<n>We propose to unpack the taxonomy of long-context based on the properties that make them more difficult with longer contexts.<n>We conclude that the most difficult and interesting settings, whose necessary information is very long and highly diffused within the input, is severely under-explored.
  arXiv  Detail & Related papers  (2024-06-29T11:09:47Z)
• Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA [71.04146366608904]
  Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. We propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning.
  arXiv  Detail & Related papers  (2024-06-25T09:42:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.