Context Parametrization with Compositional Adapters

• URL: http://arxiv.org/abs/2509.22158v1
• Date: Fri, 26 Sep 2025 10:16:28 GMT
• Title: Context Parametrization with Compositional Adapters
• Authors: Josip Jukić, Martin Tutek, Jan Šnajder,
• Abstract summary: CompAs is a meta-learning framework that translates context into adapter parameters with a compositional structure.<n>Our work establishes composable adapter generation as a practical and efficient alternative for scaling large language models.
• Score: 3.4155322317700576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, both of these approaches face key limitations: ICL is inefficient when handling many demonstrations, and SFT incurs training overhead while sacrificing flexibility. Mapping instructions or demonstrations from context directly into adapter parameters offers an appealing alternative. While prior work explored generating adapters based on a single input context, it has overlooked the need to integrate multiple chunks of information. To address this gap, we introduce CompAs, a meta-learning framework that translates context into adapter parameters with a compositional structure. Adapters generated this way can be merged algebraically, enabling instructions, demonstrations, or retrieved passages to be seamlessly combined without reprocessing long prompts. Critically, this approach yields three benefits: lower inference cost, robustness to long-context instability, and establishes a principled solution when input exceeds the model's context window. Furthermore, CompAs encodes information into adapter parameters in a reversible manner, enabling recovery of input context through a decoder, facilitating safety and security. Empirical results on diverse multiple-choice and extractive question answering tasks show that CompAs outperforms ICL and prior generator-based methods, especially when scaling to more inputs. Our work establishes composable adapter generation as a practical and efficient alternative for scaling LLM deployment.

Related papers

• On-device System of Compositional Multi-tasking in Large Language Models [29.561801948704822]
  We propose a novel approach tailored specifically for compositional multi-tasking scenarios involving summarization and translation.<n>Our technique involves adding a learnable projection layer on top of the combined summarization and translation adapters.<n>We demonstrate the practical viability of our method within an on-device environment by developing an Android app capable of executing compositional tasks seamlessly.
  arXiv  Detail & Related papers  (2025-10-11T19:49:22Z)
• ContextNav: Towards Agentic Multimodal In-Context Learning [85.05420047017513]
  ContextNav is an agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation.<n>It builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts.<n> Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets.
  arXiv  Detail & Related papers  (2025-10-06T07:49:52Z)
• AdaptiveK Sparse Autoencoders: Dynamic Sparsity Allocation for Interpretable LLM Representations [28.447024168930984]
  We propose AdaptiveK SAE (Adaptive Top K Sparse Autoencoders), a novel framework that dynamically adjusts sparsity levels based on the semantic complexity of each input.<n>We show that this complexity-driven adaptation significantly outperforms fixed-sparsity approaches on reconstruction fidelity, explained variance, cosine similarity and interpretability metrics.
  arXiv  Detail & Related papers  (2025-08-24T12:00:41Z)
• PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition [7.966499123076283]
  Vision-Language Models (VLMs) like CLIP offer promising solutions for Dynamic Facial Expression Recognition (DFER)<n>We propose PE-CLIP, a parameter-efficient fine-tuning framework that adapts CLIP for DFER while significantly reducing trainable parameters.<n>By balancing efficiency and accuracy, PE-CLIP sets a new benchmark in resource-efficient DFER.
  arXiv  Detail & Related papers  (2025-03-21T08:45:50Z)
• StreamAdapter: Efficient Test Time Adaptation from Contextual Streams [99.08657973288913]
  In-context learning (ICL) allows large language models to adapt to new tasks directly from the given demonstrations without requiring gradient updates. We propose StreamAdapter, a novel approach that directly updates model parameters from context at test time.
  arXiv  Detail & Related papers  (2024-11-14T09:03:54Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair [5.006064616335817]
  Large Language Models (LLMs) have shown high capabilities in several software development-related tasks.<n> adapters offer a more efficient way to customize LLMs for particular needs.<n>Model (and adapter) merging have emerged as a technique to develop one model capable of multiple tasks.
  arXiv  Detail & Related papers  (2024-08-18T18:45:48Z)
• CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning [101.81127587760831]
  Current fine-tuning methods build adapters widely of the context of downstream task to learn, or the context of important knowledge to maintain.<n>We propose CorDA, a Context-oriented Decomposition Adaptation method that builds learnable task-aware adapters.<n>Our method enables two options, the knowledge-preserved adaptation and the instruction-previewed adaptation.
  arXiv  Detail & Related papers  (2024-06-07T19:10:35Z)
• Towards Modular LLMs by Building and Reusing a Library of LoRAs [64.43376695346538]
  We study how to best build a library of adapters given multi-task data. We introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters.
  arXiv  Detail & Related papers  (2024-05-18T03:02:23Z)
• Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AI [10.82017289243097]
  Large Language Models (LLMs) are capable of reasoning over diverse input data modalities through pre-trained encoders. m-LLM improves the task accuracy by up to 4% compared to the best existing scheme.
  arXiv  Detail & Related papers  (2023-12-13T04:08:59Z)
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models [75.25782573728677]
  This paper presents a framework for adapter-based parameter-efficient fine-tuning (PEFT) of language models (LLMs) The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning.
  arXiv  Detail & Related papers  (2023-04-04T16:31:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.