Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs

• URL: http://arxiv.org/abs/2606.23063v1
• Date: Mon, 22 Jun 2026 09:13:53 GMT
• Title: Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
• Abstract summary: Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions.<n>Existing continual vision-language methods mainly preserve outputs, replay data or pseudo-data, regularize embedding geometry, or allocate task-specific parameters.<n>We propose Attention-Spectrum Regularization (ASR), a replay-free continual learning framework that preserves skill-conditioned structures of cross-modal attention.
• Score: 33.66167879532472
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions, yet continual fine-tuning often causes severe forgetting of previously acquired multimodal skills. Existing continual vision-language methods mainly preserve outputs, replay data or pseudo-data, regularize embedding geometry, or allocate task-specific parameters, but they provide limited control over how internal cross-modal attention patterns supporting old skills drift during adaptation. We propose Attention-Spectrum Regularization (ASR), a replay-free continual learning framework that preserves skill-conditioned structures of cross-modal attention. ASR treats cross-attention maps as two-dimensional signals, summarizes their scale and directional properties into compact spectral statistics, and stores only skill-wise prototype distributions instead of replaying past image-question pairs, generated pseudo-examples, or old-stage teacher snapshots. In later stages, a phase-invariant spectral regularizer constrains harmful drift of these prototypes while allowing instance-level attention to adapt to new tasks. We provide theoretical analysis showing that skill-conditioned spectral drift controls forgetting under a spectral sufficiency assumption, and that Fourier power spectra are stable to spatial translations and bounded perturbations. Experiments on continual VQA and multimodal instruction-tuning benchmarks, including VQA v2, VQACL, CLT-VQA, CoIN, and UCIT, show that ASR consistently improves final performance and reduces forgetting over strong replay-, regularization-, and adapter-based baselines. Preserving skill-level attention structure is an effective and lightweight mechanism for continual MLLMs. Code is available at https://github.com/Creative-zcx/attention-spectrum-replay

Related papers

• A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training [56.364526893066746]
  Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly.<n>Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning.<n>Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning.
  arXiv  Detail & Related papers  (2026-08-26T17:57:04Z)
• Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models [16.821897521676963]
  Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training.<n>We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation.<n>We show that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression.
  arXiv  Detail & Related papers  (2026-08-07T07:36:19Z)
• VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence [8.44621217686077]
  Industrial time series serve as the foundation for Prognostics and Health Management (PHM)<n>We propose VLT, a multimodal foundation model that jointly models time-series, frequency-spectrum visual representations, and textual knowledge.<n>VLT outperforms state-of-the-art methods, achieving superior robustness and generalization under few-shot, noisy, and incomplete-modality settings.
  arXiv  Detail & Related papers  (2026-07-16T02:53:54Z)
• Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models [46.61944018676346]
  We propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE)<n>Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain.<n>CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction.
  arXiv  Detail & Related papers  (2026-06-23T05:39:52Z)
• LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models [5.257217773590746]
  Online task-free continual learning (TFCL) requires intelligent agents to sequentially accumulate knowledge from an unbounded, non-stationary data stream.<n>Existing online TFCL paradigms rely on parameter-efficient prompt tuning or dynamic structure expansion driven by training-coupled optimization dynamics.<n>We propose LargeMonitor, a framework that leverages large pretrained foundation models to autonomously orchestrate task-free continuous adaptation.
  arXiv  Detail & Related papers  (2026-06-08T12:41:15Z)
• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• Reinforcing Structured Chain-of-Thought for Video Understanding [49.3086326382764]
  We introduce Summary-Driven Reinforcement Learning (SDRL)<n>SDRL is a single-stage RL framework that obviates the need forSupervised Fine-Tuning (SFT)<n>Our method achieves state-of-the-art performance on seven public VideoQA datasets.
  arXiv  Detail & Related papers  (2026-03-26T22:11:14Z)
• LEFT: Learnable Fusion of Tri-view Tokens for Unsupervised Time Series Anomaly Detection [53.191369031661885]
  Unsupervised time series anomaly detection aims to build a model for identifying abnormal timestamps without assuming the availability of annotations.<n>We present Learnable Fusion of Tri-view Tokens (LEFT), a unified unsupervised TSAD framework that models anomalies as inconsistencies across complementary representations.<n>Experiments on real-world benchmarks show that LEFT yields the best detection accuracy against SOTA baselines, while achieving a 5x reduction on FLOPs and 8x speed-up for training.
  arXiv  Detail & Related papers  (2026-02-09T13:33:49Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• PLanTS: Periodicity-aware Latent-state Representation Learning for Multivariate Time Series [10.332959619473652]
  We propose PLanTS, a periodicity-aware self-supervised learning framework that explicitly models irregular latent states and their transitions.<n> PLanTS consistently improves the representation quality over existing SSL methods and demonstrates superior runtime efficiency compared to DTW-based methods.
  arXiv  Detail & Related papers  (2025-09-05T20:10:09Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Is Attention Required for Transformer Inference? Explore Function-preserving Attention Replacement [13.38679135071682]
  We propose a Function-preserving Attention Replacement framework that replaces all attention blocks in pretrained transformers with learnable sequence-to-sequence modules.<n>We validate FAR on the DeiT vision transformer family and demonstrate that it matches the accuracy of the original models on ImageNet and multiple downstream tasks with reduced parameters and latency.
  arXiv  Detail & Related papers  (2025-05-24T02:23:46Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.