Decoupling Knowledge and Task Subspaces for Composable Parametric Retrieval Augmented Generation

• URL: http://arxiv.org/abs/2604.26768v1
• Date: Wed, 29 Apr 2026 15:00:35 GMT
• Title: Decoupling Knowledge and Task Subspaces for Composable Parametric Retrieval Augmented Generation
• Abstract summary: Parametric Retrieval-Augmented Generation (PRAG) encodes external documents into lightweight parameter modules that can be retrieved and merged at inference time.<n>Despite its potential, many PRAG implementations train document adapters with task-supervised objectives.<n>This entanglement may make adapter composition less reliable when multiple adapters are merged at inference time.
• Score: 30.955158253465413
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Parametric Retrieval-Augmented Generation (PRAG) encodes external documents into lightweight parameter modules that can be retrieved and merged at inference time, offering a promising alternative to in-context retrieval augmentation. Despite its potential, many PRAG implementations train document adapters with task-supervised objectives, which may cause each adapter to encode both document-specific facts and reusable task-solving behavior. This entanglement may make adapter composition less reliable: when multiple adapters are merged at inference time, their overlapping task behaviors can accumulate together with document-specific updates, potentially making the merged adapter less stable and less focused on the intended document knowledge. To examine this issue, we explore Orthogonal Subspace Decomposition (OSD), an adapter-training setup that separates reusable task behavior from document-specific knowledge adapters. Concretely, we first train a Task LoRA to capture reusable task behavior, and then train document LoRAs to encode document-specific knowledge in a orthogonal subspace. This setup provides a controlled way to examine how orthogonalizing task and document LoRA updates affects adapter composition in multi-document PRAG. Experiments across multiple knowledge-intensive tasks and model scales suggest that this orthogonalization strategy can improve compositional robustness in parametric RAG, especially when multiple document adapters are merged.

Related papers

• ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning [16.21619173438734]
  Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities.<n>Recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing.<n>We propose ProtoAda, a prototype-guided adaptive tuning framework.
  arXiv  Detail & Related papers  (2026-06-01T17:59:13Z)
• Align then Train: Efficient Retrieval Adapter Learning [16.855103318831073]
  We propose a label-efficient framework that trains retrieval adapters in two stages: self-supervised alignment and supervised adaptation.<n>Experiments on the MAIR benchmark, spanning 126 retrieval tasks across 6 domains, show that ERA improves retrieval in low-label settings.
  arXiv  Detail & Related papers  (2026-04-03T19:04:42Z)
• Model Editing for New Document Integration in Generative Information Retrieval [110.90609826290968]
  Generative retrieval (GR) reformulates the Information Retrieval (IR) task as the generation of document identifiers (docIDs)<n>Existing GR models exhibit poor generalization to newly added documents, often failing to generate the correct docIDs.<n>We propose DOME, a novel method that effectively and efficiently adapts GR models to unseen documents.
  arXiv  Detail & Related papers  (2026-03-03T09:13:38Z)
• MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering [51.19392014547221]
  Retrieval-based multimodal document QA aims to identify and integrate relevant information from visually rich documents with complex multimodal structures.<n>Current approaches rely on query-agnostic document representations that overlook salient content.<n>We propose the Multimodal Adaptive Retrieval-Augmented (MARA) framework, which introduces query-adaptive generation.
  arXiv  Detail & Related papers  (2026-02-01T12:27:40Z)
• Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters [27.694134466842502]
  Parametric Retrieval-Augmented Generation (PRAG) integrates external knowledge directly into a Large Language Model (LLM)<n>Current PRAG approaches adopt a textbfone-to-one document encoding scheme, using a dedicated LoRA adapter for each individual document.<n>We propose a novel paradigm for encoding passages in PRAG that utilizes a latent routing encoding process.
  arXiv  Detail & Related papers  (2025-11-21T08:44:21Z)
• Pilot: Building the Federated Multimodal Instruction Tuning Framework [79.56362403673354]
  Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM.<n>In stage 1, we extract task-specific features and client-specific features from visual information.<n>In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction.
  arXiv  Detail & Related papers  (2025-01-23T07:49:24Z)
• Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projection [22.748835458594744]
  We propose a new framework for adapter reuse that moves beyond retrieval.<n>We represent each task by a latent prototype vector and aim to approximate the target task prototype as a sparse linear combination of retrieved reference prototypes.<n>The resulting combination weights are then used to blend the corresponding LoRA adapters, yielding a composite adapter tailored to the target task.
  arXiv  Detail & Related papers  (2024-10-13T16:28:38Z)
• MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair [5.006064616335817]
  Large Language Models (LLMs) have shown high capabilities in several software development-related tasks.<n> adapters offer a more efficient way to customize LLMs for particular needs.<n>Model (and adapter) merging have emerged as a technique to develop one model capable of multiple tasks.
  arXiv  Detail & Related papers  (2024-08-18T18:45:48Z)
• The Power of Summary-Source Alignments [62.76959473193149]
  Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection. alignment of corresponding sentences between a reference summary and its source documents has been leveraged to generate training data. This paper proposes extending the summary-source alignment framework by applying it at the more fine-grained proposition span level.
  arXiv  Detail & Related papers  (2024-06-02T19:35:19Z)
• Towards Modular LLMs by Building and Reusing a Library of LoRAs [64.43376695346538]
  We study how to best build a library of adapters given multi-task data. We introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters.
  arXiv  Detail & Related papers  (2024-05-18T03:02:23Z)
• Plug-and-Play Document Modules for Pre-trained Models [92.9897146991974]
  We propose to represent each document as a plug-and-play document module, i.e., a document plugin, for PTMs (PlugD) By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks. Experiments on 8 datasets of 4 typical NLP tasks show that PlugD enables models to encode documents once and for all across different scenarios.
  arXiv  Detail & Related papers  (2023-05-28T08:01:40Z)
• Multi-Head Adapter Routing for Cross-Task Generalization [56.75667096355806]
  Polytropon learns an inventory of adapters and a routing function that selects a subset of adapters for each task during both pre-training and few-shot adaptation. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation.
  arXiv  Detail & Related papers  (2022-11-07T19:35:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.