Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale

• URL: http://arxiv.org/abs/2603.11513v1
• Date: Thu, 12 Mar 2026 03:59:42 GMT
• Title: Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale
• Abstract summary: We evaluate five model sizes from 360M to 8B across three architecture families SmolLM2 Qwen2.5 and Llama 3.1.<n>We find that for models below 7B parameters the main limitation of RAG is context utilization rather than retrieval quality.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval augmented generation RAG is widely deployed to improve factual accuracy in language models yet it remains unclear whether smaller models of size 7B parameters or less can effectively utilize retrieved information. To investigate this question we evaluate five model sizes from 360M to 8B across three architecture families SmolLM2 Qwen2.5 and Llama 3.1 under four retrieval conditions including no retrieval BM25 dense retrieval using E5 large v2 and oracle retrieval where the retrieved passage is guaranteed to contain the answer. We introduce a parametric knowledge split that separates questions a model can already answer from those that require external knowledge which allows us to isolate utilization failure from retrieval quality failure. We find three main results. First even with oracle retrieval models of size 7B or smaller fail to extract the correct answer 85 to 100 percent of the time on questions they cannot answer alone which indicates a fundamental utilization bottleneck. Second adding retrieval context destroys 42 to 100 percent of answers the model previously knew suggesting a distraction effect driven by the presence of context rather than its quality. Third an error analysis of 2588 oracle failures shows that the dominant failure mode is irrelevant generation where the model ignores the provided context entirely. These patterns hold across multiple prompt templates and retrieval methods. The results indicate that for models below 7B parameters the main limitation of RAG is context utilization rather than retrieval quality and that deploying RAG at this scale can lead to a net negative trade off under standard evaluation conditions.

Related papers

• Metadata, Structure, or Strategy? A Decomposition of RAG Context Enrichment [3.2058241360543254]
  Retrieval-augmented generation systems increasingly enrich retrieved passages by attaching quality metadata, structuring them into explicit records, and adopting multi-hop retrieval strategies that accumulate evidence across steps.<n>These changes assume that richer context yields better answers, yet existing evaluations cannot test this because they vary all three factors at once.<n>We isolate each factor in a controlled experiment across six benchmarks, four models from three families, and five enrichment levels, totaling over 24,000 evaluated responses.<n>What determines answer quality is not how much metadata the context carries but whether the model can act on it for the given task.
  arXiv  Detail & Related papers  (2026-06-28T23:26:28Z)
• Benchmarking Deflection and Hallucination in Large Vision-Language Models [25.176271096443482]
  Existing benchmarks overlook conflicts between visual and textual evidence.<n>We introduce VLM-DeflectionBench, a benchmark of 2,775 samples spanning diverse multimodal retrieval settings.<n>Our results highlight the need to evaluate not only what models know, but how they behave when they do not.
  arXiv  Detail & Related papers  (2026-04-13T20:22:22Z)
• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models [28.80331720382804]
  We introduce Orion, a training framework that enables compact models to perform iterative retrieval through learned search strategies.<n>Orion combines synthetic trajectory generation and supervised fine-tuning to encourage diverse exploration patterns in models.<n>Despite using only 3% of the training data available, our 1.2B model achieves 77.6% success on SciFact.
  arXiv  Detail & Related papers  (2025-11-10T19:49:55Z)
• SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension [77.93156509994994]
  We show how to represent short chunks in a way that is conditioned on a broader context window to enhance retrieval performance.<n>Existing embedding models are not well-equipped to encode such situated context effectively.<n>Our method substantially outperforms state-of-the-art embedding models.
  arXiv  Detail & Related papers  (2025-08-03T23:59:31Z)
• Maximally-Informative Retrieval for State Space Model Generation [59.954191072042526]
  We introduce Retrieval In-Context Optimization (RICO) to minimize model uncertainty for a particular query at test-time.<n>Unlike traditional retrieval-augmented generation (RAG), which relies on externals for document retrieval, our approach leverages direct feedback from the model.<n>We show that standard top-$k$ retrieval with model gradients can approximate our optimization procedure, and provide connections to the leave-one-out loss.
  arXiv  Detail & Related papers  (2025-06-13T18:08:54Z)
• Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models [0.18846515534317265]
  General-purpose large language models (LLMs) often struggle with hallucinations, where generated responses deviate from authoritative sources.<n>This challenge highlights the need for systems that can integrate domain-specific knowledge while maintaining response accuracy, relevance, and faithfulness.<n>This research utilizes a descriptive dataset of Quranic surahs including the meanings, historical context, and qualities of the 114 surahs.<n>The models are evaluated using three key metrics set by human evaluators: context relevance, answer faithfulness, and answer relevance.
  arXiv  Detail & Related papers  (2025-03-20T13:26:30Z)
• Sufficient Context: A New Lens on Retrieval Augmented Generation Systems [19.238772793096473]
  Augmenting LLMs with context leads to improved performance across many applications.<n>We develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query.<n>By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance excel at answering queries when the context is sufficient.
  arXiv  Detail & Related papers  (2024-11-09T02:13:14Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context [55.564789967211844]
  An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant. Recent work has shown that retrieval augmentation can sometimes have a negative effect on performance.
  arXiv  Detail & Related papers  (2023-10-02T18:52:35Z)
• Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrieval [9.136948771060895]
  We evaluate two methods for further improvement in this setting. Both focus on combining rationales generated by a larger Language Model with longer contexts created from a multi-hop dense retrieval system. Our single best Reasoning model materially improves upon strong comparable prior baselines for unseen evaluation datasets.
  arXiv  Detail & Related papers  (2023-08-09T05:06:39Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.