HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering

• URL: http://arxiv.org/abs/2605.22035v1
• Date: Thu, 21 May 2026 06:12:34 GMT
• Title: HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering
• Abstract summary: Continual Visual Question Answering (VQA) requires learning from non-stationary streams of visual inputs and questions.<n>Most prior methods adapt by updating a largely shared parameter set.<n>We propose HyLoVQA, which maintains a drift-resilient memory bank of anchors.
• Score: 12.614540558911893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual Visual Question Answering (VQA) requires learning from non-stationary streams of visual inputs and questions while preserving past knowledge. Most prior methods adapt by updating a largely shared parameter set. This often leads to cross-level task interference, hindering accurate adaptation to the current task and object. To address this limitation, we propose HyLoVQA. It maintains a drift-resilient memory bank of anchors. The bank stores the content of visual objects and textual tasks, and they are updated using current input features. Conditioned on retrieved anchors, a hypernetwork generates lightweight Low-Rank Adaptation (LoRA) adapters. This ensures parameter efficiency, allowing the model to adapt to each task and object dynamically. Additionally, we formulate an alignment loss that aligns semantic discrepancies in the feature space with functional changes in the parameter space, thereby constraining LoRA adapters to remain focused on the current task and object. Extensive experiments on VQA v2 and NExT-QA under both standard and compositional settings demonstrate the superiority of HyLoVQA over prior state-of-the-art methods.

Related papers

• One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning [45.405420567098155]
  We present textbfFACET: task-conditioned textbfFetextbfAture transformation with textbfConditiontextbfEd feature consistextbfTency.<n>Our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs.
  arXiv  Detail & Related papers  (2026-08-31T17:03:45Z)
• SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass [55.28352410490407]
  SHINE is a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM)<n>We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters in a single forward pass.<n>Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling.
  arXiv  Detail & Related papers  (2026-02-06T03:40:31Z)
• HAM: Hierarchical Adapter Merging for Scalable Continual Learning [5.958899330375292]
  New knowledge can interfere with previously learned information, causing the model to forget earlier knowledge in favor of the new.<n>This paper introduces Hierarchical Adapters Merging (HAM), a novel framework that dynamically combines adapters from different tasks during training.<n>Ham significantly outperforms state-of-the-art methods, particularly as the number of tasks increases.
  arXiv  Detail & Related papers  (2025-09-16T16:18:19Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering [17.369734751262126]
  Continual Learning in Visual Question Answering (VQACL) requires models to acquire new visual-linguistic skills (plasticity) while preserving previously learned knowledge (stability)<n>Existing methods, primarily designed for unimodal settings, often fall short in addressing this dual requirement.<n>We present QUestion-only replay with Attention Distillation (QUAD), a novel approach for VQACL that leverages only past task questions for regularization.
  arXiv  Detail & Related papers  (2025-02-06T19:37:43Z)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging [80.17238673443127]
  LiNeS is a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance.<n>LiNeS demonstrates significant improvements in both single-task and multi-task settings across various benchmarks in vision and natural language processing.
  arXiv  Detail & Related papers  (2024-10-22T16:26:05Z)
• AdaCL:Adaptive Continual Learning [7.116403133334646]
  Class-Incremental Learning aims to update a deep classifier to learn new categories while maintaining or improving its accuracy on previously observed classes. Common methods to prevent forgetting previously learned classes include regularizing the neural network updates and storing exemplars in memory. We show that adapting hyperpararmeters on each new task leads to improvement in accuracy, forgetting and memory.
  arXiv  Detail & Related papers  (2023-03-23T09:00:38Z)
• Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning [60.501201259732625]
  We introduce task-adaptive saliency for EFCIL and propose a new framework, which we call Task-Adaptive Saliency Supervision (TASS) Our experiments demonstrate that our method can better preserve saliency maps across tasks and achieve state-of-the-art results on the CIFAR-100, Tiny-ImageNet, and ImageNet-Subset EFCIL benchmarks.
  arXiv  Detail & Related papers  (2022-12-16T02:43:52Z)
• Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task [12.74065821307626]
  VQA is an ambitious task aiming to answer any image-related question. It is hard to build such a system once for all since the needs of users are continuously updated. We propose a real-data-free replay-based method tailored for CL on VQA, named Scene Graph as Prompt for Replay.
  arXiv  Detail & Related papers  (2022-08-24T12:00:02Z)
• Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules [85.98177341704675]
  The problem of grounding VQA tasks has seen an increased attention in the research community recently. We propose a visual capsule module with a query-based selection mechanism of capsule features. We show that integrating the proposed capsule module in existing VQA systems significantly improves their performance on the weakly supervised grounding task.
  arXiv  Detail & Related papers  (2021-05-11T07:45:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.