Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder

• URL: http://arxiv.org/abs/2604.16952v1
• Date: Sat, 18 Apr 2026 10:23:00 GMT
• Title: Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder
• Abstract summary: High-resolution (HR) joint optical and synthetic aperture radar (SAR) pretraining seeks modality synergy to mutually enhance single-source representations.<n>We propose CoDe-MAE, pioneering a textitbetter synergy with less alignment philosophy.<n>CoDe-MAE successfully prevents representation degradation and establishes new state-of-the-art performance across diverse single- and bi-modal downstream tasks.
• Score: 34.73963627819185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning robust representations across extremely heterogeneous modalities remains a fundamental challenge in multi-modal vision. As a critical and profound instantiation of this challenge, high-resolution (HR) joint optical and synthetic aperture radar (SAR) pretraining seeks modality synergy to mutually enhance single-source representations; its potential is severely hindered by the Heterogeneity-Resolution Paradox: finer spatial scales drastically amplify the physical divergence between complex radar geometries and non-homologous optical textures. Consequently, migrating medium-resolution-oriented rigid alignment paradigms to HR scenarios triggers either severe feature suppression to force equivalence, or feature contamination driven by extreme epistemic uncertainty. Both extremes inevitably culminate in profound representation degradation and negative transfer. To overcome this bottleneck, we propose CoDe-MAE, pioneering a \textit{better synergy with less alignment} philosophy. First, Optical-anchored Knowledge Distillation (OKD) implicitly regularizes SAR's speckle noise by mapping it into a pure semantic manifold. Building on this, Conditioned Contrastive Learning (CCL) utilizes a gradient buffering mechanism to align shared consensus while safely preserving divergent physical signatures. Concurrently, Cross-Modal Degraded Reconstruction (CDR) deliberately strips non-homologous spectral pseudo-features, truncating the inherently ill-posed mapping to capture true structural invariants. Extensive analyses validate our theoretical claims. Pretrained on 1M samples, CoDe-MAE demonstrates remarkable data efficiency, successfully preventing representation degradation and establishing new state-of-the-art performance across diverse single- and bi-modal downstream tasks, substantially outperforming foundation models scaled on vastly larger datasets.

Related papers

• Overcoming Attention Drift: Homogeneity-Heterogeneity Guided Feature Aggregation for Low-Light Remote Sensing Image Enhancement [10.412281080988647]
  Under severe noise and illumination corruption, existing methods suffer from attention drift and severe structural blurring and color distortion.<n>We propose HALO, a dual-prior-driven enhancement framework that formulates enhancement as a guided feature aggregation problem driven by foundation model priors.<n>H2CAM achieves state-of-the-art performance across 8 challenging synthetic and real-world remote sensing benchmarks.
  arXiv  Detail & Related papers  (2026-08-06T10:11:36Z)
• Beyond Unfolding: 60x Faster One-Stage Unmixing for Closely-Spaced Infrared Small Targets [43.71958109614362]
  Closely-Spaced Infrared Small Targets (CSIST) typically exhibit energy overlap, manifesting as indistinguishable blobs in infrared images.<n>This ambiguity invalidates the one-to-one mapping assumption of traditional detection.<n>We propose a one-stage lightweight paradigm which demonstrates that deep unfolding is not necessary.
  arXiv  Detail & Related papers  (2026-07-17T14:42:51Z)
• Prototype-Anchored Generalized Manifold Regression for Unknown-Domain Object Detection [58.25418970608328]
  We study Single-Domain Generalized Object Detection (Single-DGOD), which aims to transfer a detector trained on a single source domain to multiple unseen domains.<n>Existing methods mainly rely on simulation-driven strategies, such as data augmentation or textual prompts, to enlarge the training distribution.<n>We propose Manifold Regression with Visual-Text Dual Chain-of-Thought (MR-DCoT), which formulates unknown-domain generalization as a manifold regression problem.
  arXiv  Detail & Related papers  (2026-07-08T09:27:15Z)
• MaCo-GAN: Manifold-Contrastive Adversarial Learning for Single Image Super-Resolution [5.741407725419445]
  We propose a novel manifold-contrastive GAN framework that replaces the conventional adversarial loss with a supervised contrastive objective.<n>A core component of our method is a dynamic fake sample synthesizer that transforms ground truth (GT) data into a spectrum of challenging, perceptually plausible fake images.<n>We demonstrate consistent improvements in the perception-distortion trade-off across various benchmarks.
  arXiv  Detail & Related papers  (2026-06-03T16:29:44Z)
• Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation [50.139984798361375]
  We propose a training-free framework that removes redundancy via latent discrepancy.<n> LD-Pruning substantially reduces inference while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
  arXiv  Detail & Related papers  (2026-05-29T19:34:39Z)
• Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering [57.38215918201251]
  We propose HERL, a Hyperbolic Enhanced Representation Learning framework for Incomplete Multi-View Clustering.<n> operating within the Poincaré ball, HERL constructs a structure-aware latent space to enhance representation learning.<n>We show that HERL consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-04-18T10:50:46Z)
• Energy-Regularized Spatial Masking: A Novel Approach to Enhancing Robustness and Interpretability in Vision Models [3.609252563305193]
  Deep convolutional neural networks achieve remarkable performance by exhaustively processing dense spatial feature maps.<n>This brute-force strategy introduces significant computational redundancy and encourages reliance on spurious background correlations.<n>We propose Energy-Regularized Spatial Masking (ERSM), a novel framework that reformulates feature selection as a differentiable energy minimization problem.
  arXiv  Detail & Related papers  (2026-04-08T09:48:31Z)
• DOC-GS: Dual-Domain Observation and Calibration for Reliable Sparse-View Gaussian Splatting [80.43237927269575]
  In this paper, we revisit sparse-view 3DGS reconstruction from a new perspective.<n>We identify the core challenge as the unobservability of Gaussian primitive reliability.<n>Motivated by this observation, we propose a unified Dual-domain Observation and geometrics in rendered images framework.
  arXiv  Detail & Related papers  (2026-04-08T07:01:24Z)
• Revisiting the Perception-Distortion Trade-off with Spatial-Semantic Guided Super-Resolution [25.911279124430944]
  SpaSemSR is a spatial-semantic guided diffusion framework with two complementary guidances.<n>First, spatial-grounded textual guidance integrates object-level spatial cues with semantic prompts, aligning textual and visual structures to reduce distortion.<n>Second, semantic-enhanced visual guidance with a multi-encoder design and semantic degradation constraints unifies multimodal semantic priors, improving perceptual realism under severe degradations.
  arXiv  Detail & Related papers  (2026-03-14T20:41:27Z)
• Resolving Blind Inverse Problems under Dynamic Range Compression via Structured Forward Operator Modeling [17.311804261171094]
  We introduce the textbfcascaded monotonic Bernstein (CaMB) operator to parameterize the unknown forward model.<n>CaMB enforces monotonicity as a hard architectural inductive bias, constraining optimization to physically consistent mappings.<n>We show that CaMB-Diff significantly outperforms state-of-the-art zero-shot baselines in terms of both signal fidelity and physical consistency.
  arXiv  Detail & Related papers  (2026-03-02T14:09:54Z)
• Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction [45.25461515976432]
  Plug-and-Play diffusion prior (DP) frameworks have emerged as a powerful paradigm for imaging reconstruction.<n>We present a novel approach to resolving bias-hallucination trade-off, achieving state-of-the-art gradients with significantly accelerated convergence.
  arXiv  Detail & Related papers  (2026-02-26T16:58:43Z)
• Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restoration [31.878334664450776]
  We present textbfPrefRestore, a hierarchical framework that integrates discrete semantic logic with continuous texture generation.<n>Our methodology fundamentally addresses this information disparity through two complementary strategies.<n>Pref-Restore achieves state-of-the-art performance across synthetic and real-world benchmarks.
  arXiv  Detail & Related papers  (2026-01-27T11:50:31Z)
• Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding [54.05243949024302]
  Existing robust MLLMs rely on implicit training/adaptation that focuses solely on visual encoder generalization.<n>We propose Robust-R1, a novel framework that explicitly models visual degradations through structured reasoning chains.<n>Our approach integrates: (i) supervised fine-tuning for degradation-aware reasoning foundations, (ii) reward-driven alignment for accurately perceiving degradation parameters, and (iii) dynamic reasoning depth scaling adapted to degradation intensity.
  arXiv  Detail & Related papers  (2025-12-19T12:56:17Z)
• HAD: Hierarchical Asymmetric Distillation to Bridge Spatio-Temporal Gaps in Event-Based Object Tracking [80.07224739976911]
  Event cameras offer exceptional temporal resolution and a range (modal)<n> RGB cameras excel at capturing rich texture with high resolution, whereas event cameras offer exceptional temporal resolution and a range (modal)
  arXiv  Detail & Related papers  (2025-10-22T13:15:13Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• ACMamba: Fast Unsupervised Anomaly Detection via An Asymmetrical Consensus State Space Model [51.83639270669481]
  Unsupervised anomaly detection in hyperspectral images (HSI) aims to detect unknown targets from backgrounds.<n>HSI studies are hindered by steep computational costs due to the high-dimensional property of HSI and dense sampling-based training paradigm.<n>We propose an Asymmetrical Consensus State Space Model (ACMamba) to significantly reduce computational costs without compromising accuracy.
  arXiv  Detail & Related papers  (2025-04-16T05:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.