Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

• URL: http://arxiv.org/abs/2607.21861v1
• Date: Thu, 23 Jul 2026 23:14:59 GMT
• Title: Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
• Abstract summary: We study baking documents directly into the weights of a 4-bit Gemma-4-e4b model via LoRA.<n>We find that once adapter capacity is adequate, training-data quality is the dominant lever on closed-book accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study baking documents directly into the weights of a 4-bit Gemma-4-e4b model via LoRA, so a system can answer questions about a corpus closed-book: no retrieval and no context-window budget. Across roughly 100 training runs from single documents to a 99-document corpus, we find that once adapter capacity is adequate, training-data quality is the dominant lever on closed-book accuracy, outweighing LoRA rank, learning rate, and two alternative architectures combined; capacity itself is a hard gate below which no data intervention helps. A single curation pass (shortening gold answers to canonical 1-6 word spans and dropping trivia) moved closed-book accuracy from 57.7% to 85.7% on a 15-document corpus, a larger jump than any architectural change. We confirm a capacity trend (rank must grow with corpus size) entangled with a coupling between rank and learning rate that we initially misdiagnosed. On a 15-document slice we add a real retrieval baseline: the internalized adapter (84.2% recall) beats a BM25-RAG pipeline with a base reader (58.9%) and even a realistic gold-chunk oracle (65.6%) at lower latency. We report the full arc, including three misdiagnoses, as a case study in debugging LLM training empirically.

Related papers

• PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents [85.82624962221026]
  We introduce PARSER, which decouples reading from reasoning.<n>A bank of lightweight subagents each bound to a single chunk read the entire document in parallel.<n>On multi-hop QA with contexts ranging from 7K to 896K tokens, PARSER with a 4B backbone outperforms the strongest sequential memory baseline.
  arXiv  Detail & Related papers  (2026-09-06T16:19:01Z)
• SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams [55.90739575113059]
  We argue the missing unit of reuse is the solving procedure shared by a cluster of related tasks, and build SkillGLoW around it.<n>Across four benchmarks (mathematical reasoning, terminal automation, software repair, and embodied control) and three models, the priors gain 17.2 points (hard) over the no-skill baseline on average.
  arXiv  Detail & Related papers  (2026-09-02T07:31:18Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model [0.0]
  We show that LoRA with r=16 recovers within 11.6 percentage points of full fine-tuning accuracy (59.6% vs. 71.2% exact-match) while training fewer than 1% of parameters and consuming 31% less peak GPU memory.<n>QLoRA with INT8 and NF4 quantization achieves comparable accuracy (52.8% and 53.2%) at dramatically lower memory cost (0.60 GB each)
  arXiv  Detail & Related papers  (2026-07-28T11:12:03Z)
• DocAtlas: Long-Document Understanding as Mutable-State Interaction [67.90400271544678]
  We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process.<n>Given a document and a question, DocAtlas exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence.<n>With GPT-5.4, DocAtlas reaches 71.4% on MMLongBench-Doc, exceeding the human-expert reference of 65.8%.
  arXiv  Detail & Related papers  (2026-07-21T09:45:34Z)
• What Survives Into Context: A Diagnostic for Budget-Constrained Multi-Hop RAG and When Submodular Evidence Packing Improves It [0.0]
  We argue that document recall is the wrong quantity to optimize in a fixed reader-context budget.<n>We introduce answer-in-context, a diagnostic that measures whether a gold answer survives as a contiguous span in the packed reader context.<n>We map the scope of this win honestly: it requires the conjunction of (i) multi-hop complementary structure, (ii) retrieval that surfaces the evidence, (iii) a binding but not extreme budget, and (iv) a reader weak enough that evidence density, not reading capacity, is the bottleneck.
  arXiv  Detail & Related papers  (2026-07-01T10:12:15Z)
• Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation [59.438696079345426]
  DICE is a training-free document-side strategy that splits documents into chunks, encodes them independently with a frozen model, and aggregates them back into a single vector.<n>On LongEmbed, DICE improves retrieval across four backbones, with the largest gains on slices beyond 4k tokens.<n>Across 12,779 filtered samples, DICE yields lower EDI than the single-vector baseline in 92.8% of cases.
  arXiv  Detail & Related papers  (2026-06-17T07:44:04Z)
• Towards Retrieving Interaction Spaces for Agentic Search [62.666902610940525]
  Recent direct corpus (DCI) work shows that agents can interact with the raw corpus through shell tools such as 5.4 interaction and file reads.<n>We argue that the role of retrieval for agentic search is not just to select documents that fit in the LLM context window, but to construct an interaction space.
  arXiv  Detail & Related papers  (2026-06-05T03:47:40Z)
• WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems [0.0]
  We characterize the compilation gap across 17 RepLiQA domains (6,800 questions)<n>We propose WiCER, an iterative algorithm inspired by counterexample-guided abstraction refinement (CEGAR) that closes this gap.<n>An ablation across all 17 topics confirms that targeted diagnosis (+0.95), not generic pinning (+0.16), drives the gains.
  arXiv  Detail & Related papers  (2026-05-08T00:25:16Z)
• The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptation [6.290180827867576]
  Hypernetwork-based methods internalize a document into an LLM's weights in a single forward pass.<n>When the document contradicts pretraining knowledge, accuracy collapses to 46.4% on the deepest facts.<n>We show the failure is a magnitude problem rather than a representational one.
  arXiv  Detail & Related papers  (2026-04-26T14:59:14Z)
• Knows: Agent-Native Structured Research Representations [5.037922001284019]
  We introduce Knows, a lightweight specification that binds structured claims, evidence, provenance, and verifiable relations to existing research artifacts.<n>Knows addresses the gap with a thin YAML sidecar that coexists with the original PDF, requiring no changes to the publication itself, and validated by a deterministic schema linter.<n>We evaluate Knows on 140 comprehension questions across 20 papers spanning 14 academic disciplines, comparing PDF-only, sidecar-only, and hybrid conditions.
  arXiv  Detail & Related papers  (2026-04-19T07:53:52Z)
• DeepBridge: A Unified and Production-Ready Framework for Multi-Dimensional Machine Learning Validation [0.0]
  DeepBridge is an 80Kline Python library that unifies validation, automatic compliance verification knowledge distillation, and synthetic data generation.<n>We demonstrate that DeepBridge: reduces validation time by 89% (17 min vs. 150 min with fragmented tools), automatically detects fairness violations with complete coverage/10 features vs. 2/10 from existing tools, generates audit-ready reports in minutes.
  arXiv  Detail & Related papers  (2025-12-18T01:32:32Z)
• Document Attribution: Examining Citation Relationships using Large Language Models [62.46146670035751]
  We propose a zero-shot approach that frames attribution as a straightforward textual entailment task.<n>We also explore the role of the attention mechanism in enhancing the attribution process.
  arXiv  Detail & Related papers  (2025-05-09T04:40:11Z)
• ReasonIR: Training Retrievers for Reasoning Tasks [139.54343970560103]
  ReasonIR-8B is the first retriever specifically trained for general reasoning tasks.<n>It achieves a new state-of-the-art of 29.9 nDCG@10 without reranker and 36.9 nDCG@10 with reranker on BRIGHT, a widely-used information retrieval benchmark.
  arXiv  Detail & Related papers  (2025-04-29T09:49:28Z)
• LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs [48.8849121506693]
  In traditional RAG framework, the basic retrieval units are normally short. The imbalanced heavy' retriever and light' reader design can lead to sub-optimal performance. We propose a new framework LongRAG, consisting of a long retriever' and a long reader'
  arXiv  Detail & Related papers  (2024-06-21T17:23:21Z)
• LOCR: Location-Guided Transformer for Optical Character Recognition [55.195165959662795]
  We propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.
  arXiv  Detail & Related papers  (2024-03-04T15:34:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.