Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

• URL: http://arxiv.org/abs/2507.00469v1
• Date: Tue, 01 Jul 2025 06:28:57 GMT
• Title: Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
• Authors: Yue Tan, Xiaoqian Hu, Hao Xue, Celso De Melo, Flora D. Salim,
• Abstract summary: We propose Bisecle for video-language continual learning, inspired by the rapid Binding and pattern separation mechanisms in the hippocampus.<n>Bisecle mitigates forgetting and enhances cross-task generalization on several VideoQA benchmarks.
• Score: 11.710573955384511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier vision-language models (VLMs) have made remarkable improvements in video understanding tasks. However, real-world videos typically exist as continuously evolving data streams (e.g., dynamic scenes captured by wearable glasses), necessitating models to continually adapt to shifting data distributions and novel scenarios. Considering the prohibitive computational costs of fine-tuning models on new tasks, usually, a small subset of parameters is updated while the bulk of the model remains frozen. This poses new challenges to existing continual learning frameworks in the context of large multimodal foundation models, i.e., catastrophic forgetting and update conflict. While the foundation models struggle with parameter-efficient continual learning, the hippocampus in the human brain has evolved highly efficient mechanisms for memory formation and consolidation. Inspired by the rapid Binding and pattern separation mechanisms in the hippocampus, in this work, we propose Bisecle for video-language continual learning, where a multi-directional supervision module is used to capture more cross-modal relationships and a contrastive prompt learning scheme is designed to isolate task-specific knowledge to facilitate efficient memory storage. Binding and separation processes further strengthen the ability of VLMs to retain complex experiences, enabling robust and efficient continual learning in video understanding tasks. We perform a thorough evaluation of the proposed Bisecle, demonstrating its ability to mitigate forgetting and enhance cross-task generalization on several VideoQA benchmarks.

Related papers

• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning [47.195002937893115]
  CoMo aims to learn more informative continuous motion representations from diverse, internet-scale videos.<n>We introduce two new metrics for more robustly and affordably evaluating motion and guiding motion learning methods.<n>CoMo exhibits strong zero-shot generalization, enabling it to generate continuous pseudo actions for previously unseen video domains.
  arXiv  Detail & Related papers  (2025-05-22T17:58:27Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)
• Multi-View Incremental Learning with Structured Hebbian Plasticity for Enhanced Fusion Efficiency [13.512920774125776]
  Multi-view incremental framework named MVIL aims at emulating the brain's fine-grained fusion of sequentially arriving views.<n> MVIL lies two fundamental modules: structured Hebbian plasticity and synaptic partition learning.<n> Experimental results on six benchmark datasets show MVIL's effectiveness over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-17T11:10:46Z)
• Learning New Concepts, Remembering the Old: Continual Learning for Multimodal Concept Bottleneck Models [11.716449233474451]
  Concept Bottleneck Models (CBMs) enhance interpretability of AI systems by bridging visual input with human-understandable concepts.<n>Existing CBMs typically assume static datasets, which limits their adaptability to real-world, continuously evolving multimodal data streams.<n>We define a novel continual learning task for CBMs: simultaneously handling concept-incremental and class-incremental learning.
  arXiv  Detail & Related papers  (2024-11-25T10:44:14Z)
• Efficient Transfer Learning for Video-language Foundation Models [13.166348605993292]
  We propose a parameter-efficient Multi-modalpatio Ssupervised-Temporal Adapter (MSTA) to enhance alignment between textual and visual representations.<n>We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-Temporal learning.
  arXiv  Detail & Related papers  (2024-11-18T01:25:58Z)
• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters [65.15700861265432]
  We present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters. To preserve the zero-shot recognition capability of vision-language models, we introduce a Distribution Discriminative Auto-Selector.
  arXiv  Detail & Related papers  (2024-03-18T08:00:23Z)
• Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives [56.2139730920855]
  We present a systematic analysis of MM-VUFMs specifically designed for road scenes. Our objective is to provide a comprehensive overview of common practices, referring to task-specific models, unified multi-modal models, unified multi-task models, and foundation model prompting techniques. We provide insights into key challenges and future trends, such as closed-loop driving systems, interpretability, embodied driving agents, and world models.
  arXiv  Detail & Related papers  (2024-02-05T12:47:09Z)
• Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning [54.67880602409801]
  In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of visual control tasks. We introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of model-based reinforcement learning.
  arXiv  Detail & Related papers  (2023-05-29T14:29:12Z)
• Learning Temporal Dynamics from Cycles in Narrated Video [85.89096034281694]
  We propose a self-supervised solution to the problem of learning to model how the world changes as time elapses. Our model learns modality-agnostic functions to predict forward and backward in time, which must undo each other when composed. We apply the learned dynamics model without further training to various tasks, such as predicting future action and temporally ordering sets of images.
  arXiv  Detail & Related papers  (2021-01-07T02:41:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.