DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance

• URL: http://arxiv.org/abs/2606.08906v1
• Date: Mon, 08 Jun 2026 01:10:36 GMT
• Title: DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance
• Abstract summary: DifferSeg is a simple yet general multimodal binary segmentation framework.<n>It employs learnable differential operators to adaptively align multimodal features and enhance their complementarity.<n>It consistently surpasses 67 state-of-the-art methods across 29 public datasets.
• Score: 17.49886552219562
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In many binary segmentation tasks, most multimodal methods rely on fixed feature concatenation for cross-modal interaction and straightforward decoder designs dominated by low-frequency semantics. %ToDO: % However, they ignore two key challenges: one is the lack of an adaptive mechanism to handle modality discrepancies and complementarity, and the other is the absence of an efficient decoding strategy to balance both high- and low-frequency representations. % In this work, we propose a simple yet general multimodal binary segmentation framework, termed DifferSeg, to address both problems simultaneously. With the help of the differential perception fusion (DPF) module, DifferSeg employs learnable differential operators to adaptively align multimodal features and enhance their complementarity through residual fusion, effectively mitigating modality mismatch and fusion redundancy. % In addition, we design a frequency-guided decoder (FGD) that builds cross-frequency interactions and multi-path upsampling to maintain consistency between detailed high-frequency structures and semantic low-frequency representations, ensuring fine-grained boundary recovery and noise suppression. % Benefiting from these designs, DifferSeg can be easily generalized to diverse binary segmentation tasks, including both natural and medical modalities. Without bells and whistles, it consistently surpasses 67 state-of-the-art methods across 29 public datasets involving 18 downstream tasks, demonstrating superior generalization and segmentation accuracy.Code and pretrained models will be available at the Link.

Related papers

• BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis [2.8218290601001854]
  BTI-Net establishes bidirectional communication at every decoder level through two parallel pathways via Task Interaction Modules.<n>Uncertainty Proxy Attention gates each TIM output per instance and per level using three signals that capture cross-task alignment, scene complexity, and prediction confidence.<n>Experiments on three medical benchmarks spanning ultrasound, dermoscopy, and brain MRI demonstrate consistent improvements in segmentation IoU and classification accuracy over both encoder-sharing and decoder-interaction baselines.
  arXiv  Detail & Related papers  (2026-06-27T22:37:26Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Feature-level Interaction Explanations in Multimodal Transformers [1.7101146971136896]
  Multimodal Transformers produce predictions without clarifying how different modalities jointly support a decision.<n>We present Feature-level I2MoE, a structured Mixture-of-Experts layer that operates directly on token/patch sequences from frozen pretrained encoders.<n>We develop an expert-wise explanation pipeline that combines attribution with top-K% masking to assess faithfulness.
  arXiv  Detail & Related papers  (2026-03-04T18:24:31Z)
• Multi-Level Bidirectional Decoder Interaction for Uncertainty-Aware Breast Ultrasound Analysis [2.8218290601001854]
  Breast ultrasound interpretation requires simultaneous lesion segmentation and tissue classification.<n>We propose a multi-task framework addressing these limitations through multi-level decoder interaction and uncertainty-aware adaptive coordination.
  arXiv  Detail & Related papers  (2026-03-01T22:02:06Z)
• Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification [55.56234913868664]
  We propose Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD) for reliable learning on multimodal data.<n>The proposed method achieves superior classification performance, robustness, and generalization compared with state-of-the-art reliable multimodal learning approaches.
  arXiv  Detail & Related papers  (2026-01-12T03:14:12Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [57.577843653775]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• Enhancing Multimodal Unified Representations for Cross Modal Generalization [52.16653133604068]
  We propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID)<n>These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality.
  arXiv  Detail & Related papers  (2024-03-08T09:16:47Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.