Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

• URL: http://arxiv.org/abs/2607.12686v1
• Date: Tue, 14 Jul 2026 12:13:53 GMT
• Title: Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis
• Abstract summary: We propose textbfSeRIn (textbfSegregate, textbfRefine, textbfIntegrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior.<n> Modality-specific evolve representations along isolated pathways, each refined against its respective encoder context.<n>A dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams.
• Score: 41.310321156046165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose \textbf{SeRIn} (\textbf{Se}gregate, \textbf{R}efine, \textbf{In}tegrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.

Related papers

• Multimodal Distribution Matching for Vision-Language Dataset Distillation [50.411341509805936]
  Multimodal Distribution Matching is a geometry-aware framework for efficient and generalizable multimodal distillation.<n>MDM integrates complementary components at the data, model, and loss levels.<n>It yields compact synthetic sets that preserve multimodal semantics, substantially reduce distillation cost, and remain robust across architectures.
  arXiv  Detail & Related papers  (2026-05-22T10:41:58Z)
• Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations [4.67724003380452]
  Multimodal learning seeks to integrate information across diverse sensory sources.<n>Current approaches struggle to balance cross-modal generalizability with modality-specific structure.<n>We introduce CoDAAR, a novel framework that establishes semantic consensus across modality-specific codebooks.
  arXiv  Detail & Related papers  (2026-05-12T14:03:30Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Feature-level Interaction Explanations in Multimodal Transformers [1.7101146971136896]
  Multimodal Transformers produce predictions without clarifying how different modalities jointly support a decision.<n>We present Feature-level I2MoE, a structured Mixture-of-Experts layer that operates directly on token/patch sequences from frozen pretrained encoders.<n>We develop an expert-wise explanation pipeline that combines attribution with top-K% masking to assess faithfulness.
  arXiv  Detail & Related papers  (2026-03-04T18:24:31Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.