Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck

• URL: http://arxiv.org/abs/2605.08526v1
• Date: Fri, 08 May 2026 22:17:54 GMT
• Title: Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck
• Abstract summary: We introduce Conditional Multimodal Information Bottleneck (CMIB), a method for multimodal skill construction.<n>Unlike naive two-stream formulations, CMIB explicitly conditions the multimodal latent on the text skill, thus structurally reducing cross-modal redundancy.<n>We instantiate CMIB with a variational objective that makes its conditional decomposition tractable to optimize, yielding reusable multimodal skills.
• Score: 60.930825621351794
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While LLM-based agents excel at planning and executing long action sequences, their execution often remains inconsistent across trials, limiting reliability. Consolidating agent consistency requires distilling trial-error trajectories into reusable skills that preserve task-relevant invariants while discarding trajectory-specific noise. However, in multimodal settings, the key challenge is not only that useful invariants are distributed across vision and language information, but that different modalities support different kinds of reusable skill content: while some skills are verbalizable and interpretable, others reside in perceptual evidence beyond text. Text-only skills may lose perceptual cues, whereas storing text and perception naively introduces redundancy and noise. Existing inference-time methods, such as self-consistency, improve reliability through costly multi-sample decoding, while internalization strategies lack a way to separate verbalizable skill content from residual perceptual information. To address this, we introduce Conditional Multimodal Information Bottleneck (CMIB), a method for multimodal skill construction. CMIB begins with a joint bottleneck over multimodal skills and derives an exact sequential decomposition: (1) a text-stage bottleneck distilling interpretable skill cards, and (2) a conditional multimodal bottleneck compressing only residual information in perception that remains predictive beyond text. Unlike naive two-stream formulations, CMIB explicitly conditions the multimodal latent on the text skill, thus structurally reducing cross-modal redundancy and enabling independent control over textual and perceptual compression. We instantiate CMIB with a variational objective that makes its conditional decomposition tractable to optimize, yielding reusable multimodal skills that improve execution stability without incurring multi-sample inference overhead.

Related papers

• MODAL: Multi-Modal Object Re-ID via Model-Driven Sparse Decoupling and Text-Image Differential Filtering [51.99452481869329]
  MODAL is a novel multi-modal object re-identification framework grounded in coupled sparse coding theory and differential suppression principles.<n>A core component of MODAL is a Multi-modal Feature Sparse Decoupling module, developed in a model-driven deep unrolling manner.<n>Benefiting from the principled feature disentanglement, MODAL naturally mitigates performance degradation in incomplete-modality scenarios.<n>Experiments on four datasets demonstrate that MODAL achieves state-of-the-art performance with superior transparency.
  arXiv  Detail & Related papers  (2026-08-15T07:41:10Z)
• LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations [53.20772659095155]
  This paper tackles the more challenging setting of IML under training-time incomplete observations.<n>We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task.
  arXiv  Detail & Related papers  (2026-05-01T06:11:42Z)
• PLUME: Latent Reasoning Based Universal Multimodal Embedding [52.35354073629127]
  Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
  arXiv  Detail & Related papers  (2026-04-02T14:04:53Z)
• Enhancing Weakly Supervised Multimodal Video Anomaly Detection through Text Guidance [10.079930398169205]
  Text provides explicit semantic information that can enhance anomaly characterization and reduce false alarms.<n> extracting effective text features is challenging due to the inability of general-purpose language models to capture anomaly-specific nuances.<n> multimodal fusion often suffers from redundancy and imbalance.
  arXiv  Detail & Related papers  (2026-02-11T05:44:30Z)
• Communication-Efficient Multi-Modal Edge Inference via Uncertainty-Aware Distributed Learning [60.650628083185616]
  We propose a three-stage communication-aware distributed learning framework to improve training and inference efficiency.<n>In StageI, devices perform local multi-modal self-supervised learning to obtain shared and modality-specific encoders without device--server exchange.<n>StageII, distributed fine-tuning with centralized evidential fusion calibrates per-modality uncertainty and reliably aggregates features distorted by noise or channel fading.<n>StageIII, an uncertainty-guided feedback mechanism selectively requests additional features for uncertain samples, optimizing the communication--accuracy tradeoff in the distributed setting.
  arXiv  Detail & Related papers  (2026-01-21T12:38:02Z)
• Buffer replay enhances the robustness of multimodal learning under missing-modality [9.512378886218395]
  We introduce REplay Prompting (REP), which builds modality-wise buffers and replays them in deeper layers to mitigate information loss as network depth increases.<n>Experiments on vision-language, vision-language-audio, and temporal multimodal benchmarks demonstrate that REP consistently outperforms prior methods under both single- and multi-modality missing scenarios.<n>These results establish REP as a lightweight and effective paradigm for robust multimodal learning in challenging missing-modality environments.
  arXiv  Detail & Related papers  (2025-11-28T10:55:31Z)
• ContextNav: Towards Agentic Multimodal In-Context Learning [85.05420047017513]
  ContextNav is an agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation.<n>It builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts.<n> Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets.
  arXiv  Detail & Related papers  (2025-10-06T07:49:52Z)
• Rethinking Explainability in the Era of Multimodal AI [9.57008593971486]
  multimodal AI systems have become ubiquitous and achieved remarkable performance across high-stakes applications.<n>Most existing explainability techniques remain unimodal, generating modality-specific feature attributions, concepts, or circuit traces in isolation.<n>This paper argues that such unimodal explanations systematically misrepresent and fail to capture the cross-modal influence that drives multimodal model decisions.
  arXiv  Detail & Related papers  (2025-06-16T03:08:29Z)
• Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability [15.155556606996994]
  Narrowing Information Bottleneck Theory is a novel framework that redefines the traditional bottleneck approach.<n>Our approach enhances image interpretability by an average of 9%, text interpretability by an average of 58.83%, and accelerates processing speed by 63.95%.
  arXiv  Detail & Related papers  (2025-02-16T19:01:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.