Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality

• URL: http://arxiv.org/abs/2403.19221v1
• Date: Thu, 28 Mar 2024 08:35:46 GMT
• Title: Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality
• Abstract summary: Video paragraph captioning (VPC) involves generating detailed narratives for long videos. Existing models are constrained by the assumption of constant availability of a single auxiliary modality. We propose a Missing-Resistant framework that harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities.
• Score: 26.55645677311152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries. However, the existing models are constrained by the assumption of constant availability of a single auxiliary modality, which is impractical given the diversity and unpredictable nature of real-world scenarios. To this end, we propose a Missing-Resistant framework MR-VPC that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities. Under this framework, we propose the Multimodal VPC (MVPC) architecture integrating video, speech, and event boundary inputs in a unified manner to process various auxiliary inputs. Moreover, to fortify the model against incomplete data, we introduce DropAM, a data augmentation strategy that randomly omits auxiliary inputs, paired with DistillAM, a regularization target that distills knowledge from teacher models trained on modality-complete data, enabling efficient learning in modality-deficient environments. Through exhaustive experimentation on YouCook2 and ActivityNet Captions, MR-VPC has proven to deliver superior performance on modality-complete and modality-missing test data. This work highlights the significance of developing resilient VPC models and paves the way for more adaptive, robust multimodal video understanding.

Related papers

• InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning [61.87294123934038]
  We present InternVideo3, a framework enhancing multimodal understanding in video tasks.<n>MCR treats understanding as a closed-loop process over a shared, evolving context.<n>We show InternVideo3 achieves strong performance on benchmarks like Video-MME, MLVU, and Ego.
  arXiv  Detail & Related papers  (2026-06-10T15:17:08Z)
• Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs [53.851880753186585]
  We propose a unified incomplete video-language model to process the incomplete multi-modal inputs.<n>Our method can serve as a plug-and-play module for previous works to improve their performance in various multi-modal tasks.
  arXiv  Detail & Related papers  (2026-05-27T03:18:10Z)
• Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection [5.097809301149341]
  In real-world deployments, some modalities may be unavailable due to sensor failures, acquisition issues, or challenging atmospheric conditions.<n>Existing multimodal segmentation models typically address missing modalities by learning a shared representation across inputs.<n>We tackle this limitation with CBC-SLP, a multimodal semantic segmentation model designed to preserve both modality-invariant and modality-specific information.
  arXiv  Detail & Related papers  (2026-04-17T09:05:22Z)
• CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning [35.562458985015944]
  We present a novel Cyclic INformative Learning framework (CyIN) to bridge the gap between complete and incomplete multimodal learning.<n>To supplement the missing information caused by incomplete multimodal input, we propose cross-modal cyclic translation.<n>CyIN succeeds in jointly optimizing complete and incomplete multimodal learning in one unified model.
  arXiv  Detail & Related papers  (2026-02-04T07:05:15Z)
• A CLIP-based Uncertainty Modal Modeling (UMM) Framework for Pedestrian Re-Identification in Autonomous Driving [6.223368492604449]
  Uncertainty Modal Modeling (UMM) framework integrates a multimodal token mapper, synthetic modality augmentation strategy, and cross-modal cue interactive learner.<n>UMM achieves strong robustness, generalization, and computational efficiency under uncertain modality conditions.
  arXiv  Detail & Related papers  (2025-08-15T04:50:27Z)
• Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models [6.610033827647869]
  In real-world scenarios, consistently acquiring complete multimodal data presents significant challenges. This often leads to the issue of missing modalities, where data for certain modalities are absent. We propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method.
  arXiv  Detail & Related papers  (2024-07-17T14:44:25Z)
• VIMI: Grounding Video Generation through Multi-modal Instruction [89.90065445082442]
  Existing text-to-video diffusion models rely solely on text-only encoders for their pretraining. We construct a large-scale multimodal prompt dataset by employing retrieval methods to pair in-context examples with the given text prompts. We finetune the model from the first stage on three video generation tasks, incorporating multi-modal instructions.
  arXiv  Detail & Related papers  (2024-07-08T18:12:49Z)
• CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion [58.15403987979496]
  CREMA is a generalizable, highly efficient, and modular modality-fusion framework for video reasoning. We propose a novel progressive multimodal fusion design supported by a lightweight fusion module and modality-sequential training strategy. We validate our method on 7 video-language reasoning tasks assisted by diverse modalities, including VideoQA and Video-Audio/3D/Touch/Thermal QA.
  arXiv  Detail & Related papers  (2024-02-08T18:27:22Z)
• Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback [38.708690624594794]
  Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF) In specific, we propose context-aware reward modeling by providing detailed video descriptions as context during the generation of preference feedback.
  arXiv  Detail & Related papers  (2024-02-06T06:27:40Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• What Makes for Robust Multi-Modal Models in the Face of Missing Modalities? [35.19295402483624]
  We model the scenarios of multi-modal models encountering missing modalities from an information-theoretic perspective. We introduce Uni-Modal Ensemble with Missing Modality Adaptation (UME-MMA) UME-MMA employs uni-modal pre-trained weights for the multi-modal model to enhance feature extraction and utilizes missing modality data augmentation techniques to better adapt to situations with missing modalities.
  arXiv  Detail & Related papers  (2023-10-10T07:47:57Z)
• One-stage Modality Distillation for Incomplete Multimodal Learning [7.791488931628906]
  This paper presents a one-stage modality distillation framework that unifies the privileged knowledge transfer and modality information fusion. The proposed framework can overcome the problem of incomplete modality input in various scenes and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-09-15T07:12:27Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering [63.87200781247364]
  Correlation Information Bottleneck (CIB) seeks a tradeoff between compression and redundancy in representations. We derive a tight theoretical upper bound for the mutual information between multimodal inputs and representations.
  arXiv  Detail & Related papers  (2022-09-14T22:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.