Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learning

• URL: http://arxiv.org/abs/2607.27766v1
• Date: Thu, 30 Jul 2026 07:03:18 GMT
• Title: Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learning
• Abstract summary: Conditional Retrieval Alignment (CoRA) is a gradient-free framework that converts a frozen encoder into a task-conditioned retriever.<n>We show that CoRA's rank-constrained basis is the optimal low-rank compression of the output-conditioned fitted representation.<n>We extend the framework to multimodal retrieval by incorporating visual representations into the conditioning and retrieval spaces.
• Score: 35.54770997926101
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-device in-context learning (ICL) relies on pre-inference retrieval to select demonstrations for useful context before downstream model inference. This retrieval must exploit task-specific information while operating over local memories under limited computation, memory, and data-exposure budgets. We propose Conditional Retrieval Alignment (CoRA), a gradient-free framework that converts a frozen encoder into a task-conditioned retriever using paired candidate inputs and outputs. CoRA selects complementary encoder layers, constructs an output-derived conditioning space from candidate memory, and aligns candidate input representations to this space through closed-form ridge regression. Low-rank factorization then produces a compact retrieval basis where candidate outputs are used only during offline index construction, whereas query-time retrieval requires only the query input and precomputed index. We show that CoRA's rank-constrained basis is the optimal low-rank compression of the output-conditioned fitted representation, and derive an exact two-pass streaming construction that avoids materializing the full fitted matrix. We further extend the framework to multimodal exemplar retrieval by incorporating visual representations into the conditioning and retrieval spaces. Experiments across ten textual datasets and four multimodal benchmarks with Llama-3.2-1B, MobileLLM-Pro, OpenFlamingo-3B, and Qwen3.5-2B, as well as end-to-end Raspberry Pi~5 deployment demonstrate that CoRA supports effective task-conditioned retrieval without retriever fine-tuning, backpropagation, or target-model calls.

Related papers

• Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment [37.61076424435405]
  Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence.<n>Standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap.<n>We propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations.
  arXiv  Detail & Related papers  (2026-09-08T03:18:58Z)
• Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices [18.557908375073307]
  CORE is a two-stage sentence-level prompt compression method.<n>We implement CORE on an NVIDIA Jetson AGX Orin edge device and a Huawei Nova smartphone.<n> CORE improves accuracy by at least 30.19% compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-04-27T16:11:28Z)
• ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compression [5.223477157195793]
  "lost in the middle" phenomenon degrades ranking quality as input length grows.<n>Inference latency scales super-linearly with sequence length, rendering listwise reranking impractical for industrial deployment.<n>We present ResRank, a unified retrieval-reranking framework that fundamentally addresses both challenges.
  arXiv  Detail & Related papers  (2026-04-24T03:11:51Z)
• UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval [87.12447641908143]
  We propose UniCVR, the first unified zero-shot composed visual retrieval framework.<n>UniCVR jointly addresses all three tasks without any task-specific human-annotated data.<n>In Stage I, we train the MLLM as a compositional query embedder via contrastive learning on a curated dataset of approximately 3.5M samples.<n>In Stage II, we introduce an MLLM-guided dual-level reranking mechanism that applies adaptive budgeted subset scoring to a small number of top-ranked candidates.
  arXiv  Detail & Related papers  (2026-04-22T08:16:50Z)
• Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction [44.83333974000826]
  We introduce a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals.<n> Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits.<n>We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
  arXiv  Detail & Related papers  (2026-02-07T14:18:11Z)
• TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework [62.66056331998838]
  TeaRAG is a token-efficient agentic RAG framework capable of compressing both retrieval content and reasoning steps.<n>Our reward function evaluates the knowledge sufficiency by a knowledge matching mechanism, while penalizing excessive reasoning steps.
  arXiv  Detail & Related papers  (2025-11-07T16:08:34Z)
• When Retriever Meets Generator: A Joint Model for Code Comment Generation [3.6781644685120924]
  RAGSum is built on top offuse retrieval and generation using a single CodeT5 backbone.<n>A contrastive pre-training phase shapes code embeddings for nearest-neighbor search.<n>A lightweight self-refinement loop is deployed to polish the final output.
  arXiv  Detail & Related papers  (2025-07-16T18:12:27Z)
• Maximally-Informative Retrieval for State Space Model Generation [59.954191072042526]
  We introduce Retrieval In-Context Optimization (RICO) to minimize model uncertainty for a particular query at test-time.<n>Unlike traditional retrieval-augmented generation (RAG), which relies on externals for document retrieval, our approach leverages direct feedback from the model.<n>We show that standard top-$k$ retrieval with model gradients can approximate our optimization procedure, and provide connections to the leave-one-out loss.
  arXiv  Detail & Related papers  (2025-06-13T18:08:54Z)
• Parallel Key-Value Cache Fusion for Position Invariant RAG [55.9809686190244]
  Large Language Models (LLMs) are sensitive to the position of relevant information within contexts.<n>We introduce a framework that generates consistent outputs for decoder-only models, irrespective of the input context order.
  arXiv  Detail & Related papers  (2025-01-13T17:50:30Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.