Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

• URL: http://arxiv.org/abs/2606.00959v1
• Date: Sun, 31 May 2026 02:29:36 GMT
• Title: Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
• Abstract summary: We introduce Partial Information Decomposition (PID) as a decision-level framework that separates unique, redundant, and synergistic contributions of sensory and linguistic inputs.<n>We extend PID to tri-modal systems with Sensory PID, treating language as a control variable to decompose video--audio information gain.
• Score: 13.945344883914318
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment. We introduce Partial Information Decomposition (PID) as a decision-level framework that separates unique, redundant, and synergistic contributions of sensory and linguistic inputs, beyond representation alignment and outcome-based evaluation. Across vision--language benchmarks, PID reveals recurring modality-use profiles: reasoning and grounding-oriented tasks tend to exhibit high synergy, whereas expert and knowledge-oriented tasks show stronger language-unique reliance. These profiles generalize across model families and predict sensitivity to modality-level interventions. We further extend PID to tri-modal systems with Sensory PID, treating language as a control variable to decompose video--audio information gain. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck dominated by visual information even on audio--visual fusion tasks. Finally, PID-guided reweighting provides initial evidence for improving multimodal reasoning and grounding performance.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time [9.870369982132678]
  Multimodal large language models (MLLMs) have revolutionized the landscape of AI.<n>These models often suffer from hallucinations, generating outputs that diverge from the provided perceptual inputs.<n>We propose Learning Inference-time Modality Enhancement (LIME) to bolster multimodal grounding.
  arXiv  Detail & Related papers  (2026-05-03T07:58:02Z)
• Source-Modality Monitoring in Vision-Language Models [16.990140746029553]
  We consider source-modality monitoring as an instance of the more general binding problem.<n>We find that both syntactic and semantic signals play an important role, but that the latter tend to outweigh the former in cases when modalities are highly distinct distributionally.
  arXiv  Detail & Related papers  (2026-04-23T19:49:36Z)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• Membership Inference for Contrastive Pre-training Models with Text-only PII Queries [8.524434324715857]
  Unimodal Membership Inference Detector (UMID) is a text-only auditing framework that performs cross-modal latent inversion.<n>Experiments across diverse CLIP and CLAP architectures demonstrate that UMID significantly improves the effectiveness and efficiency over prior MIAs.
  arXiv  Detail & Related papers  (2026-03-15T04:53:39Z)
• Beyond Language Modeling: An Exploration of Multimodal Pretraining [125.34714978184638]
  We provide empirical clarity through controlled, from-scratch pretraining experiments.<n>We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision.<n>We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language.
  arXiv  Detail & Related papers  (2026-03-03T18:58:00Z)
• Unlocking Cognitive Capabilities and Analyzing the Perception-Logic Trade-off [29.48293757752123]
  We present a progressive training pipeline that integrates Perception andReasoning capabilities.<n>We identify Temporal Drift in long-context audio, where extended reasoning desynchronizes the model from acoustic timestamps.<n>This report details the architecture, the data-efficient training recipe, and a diagnostic analysis of the trade-offs between robust perception and structured reasoning.
  arXiv  Detail & Related papers  (2026-02-27T06:56:50Z)
• Foundation Model for Skeleton-Based Human Action Understanding [56.89025287217221]
  This paper presents a Unified Skeleton-based Dense Representation Learning framework.<n>USDRL consists of a Transformer-based Dense Spatio-Temporal (DSTE), Multi-Grained Feature Decorrelation (MG-FD), and Multi-Perspective Consistency Training (MPCT)
  arXiv  Detail & Related papers  (2025-08-18T02:42:16Z)
• DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis [41.29318462528406]
  We propose a Disentangled-Language-Focused (DLF) multimodal representation learning framework.<n>It incorporates a feature disentanglement module to separate modality-shared and modality-specific information.<n>A Language-Focused Attractor (LFA) is further developed to strengthen language representation by leveraging complementary modality-specific information.
  arXiv  Detail & Related papers  (2024-12-16T10:03:44Z)
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training [55.54020926284334]
  Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. Retrieval augmentation techniques have proven to be effective plugins for both LLMs and MLLMs. In this study, we propose multimodal adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training (RA-BLIP), a novel retrieval-augmented framework for various MLLMs.
  arXiv  Detail & Related papers  (2024-10-18T03:45:19Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.