Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization

• URL: http://arxiv.org/abs/2604.16247v1
• Date: Fri, 17 Apr 2026 17:07:35 GMT
• Title: Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization
• Abstract summary: HILBERT is a cross-attentive framework for learning document-level audio-text representations.<n>We show that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings.
• Score: 1.1242490155213278
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained speech and language encoders to extract segment-level features, which are aggregated via cross-modal attention and self-attentive pooling to form modality-specific document representations and a joint cross-attentive embedding. To align modalities while preserving modality-specific structure under severe audio-text dimensional imbalance, we introduce a reciprocal dual contrastive objective that simultaneously aligns audio-to-joint and text-to-joint representations, rather than directly contrasting audio and text alone. Two auxiliary regularizers further stabilize long-sequence fusion: a Centered Kernel Alignment (CKA) loss that preserves structural consistency between each modality and the joint embedding, and a mutual information balancing loss that prevents dominance of a single modality by equalizing information flow from audio and text into the joint space. For downstream prediction, HILBERT employs a Mixture-of-Experts (MoE) classifier over concatenated audio, text, and joint representations to accommodate heterogeneous label regimes. Extensive evaluation across multiple audio-text backbone combinations demonstrates that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings.

Related papers

• Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction [0.8749675983608171]
  cascaded multi-talker ASR (MT-ASR) performance is often capped by speaker leakage during separation.<n>We propose a complementary pruning-based paradigm that robustly identifies and removes leakage artifacts.<n>Results on LibriMix, LibriSpeechMix, and the AMI Meeting corpus show our algorithm consistently reduces cpW ER across diverse overlap conditions.
  arXiv  Detail & Related papers  (2026-08-23T03:21:45Z)
• Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation [113.24577778437295]
  Baton is a framework that introduces explicit semantic planning into joint video-audio generation.<n>Our key insight is that complementing coarse text guidance with semantically rich, modality-aware planned tokens, can simultaneously restore fine-grained semantic detail.<n> experiments on benchmarks show the effectiveness of Baton both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2026-05-24T17:55:11Z)
• Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning [3.055416693231887]
  We propose a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation.<n> Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines.
  arXiv  Detail & Related papers  (2026-04-05T19:08:51Z)
• CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment [12.793962173450494]
  We propose a novel Caption-aligned and Agreement-guided Enhancement framework (CAE-AV) for audio-visual learning.<n>Two complementary modules: Cross-modal Agreement-guided Spatio-Temporal Enrichment (CASTE) and Caption-Aligned Saliency-guided Enrichment (CASE)<n>CASTE balances spatial and temporal relations by evaluating frame-level audio-visual agreement, ensuring that key information is captured from both preceding and subsequent frames under misalignment.<n>CASE injects cross-modal semantic guidance into selectedtemporal positions, leveraging high-level semantic cues to further alleviate misalignment.
  arXiv  Detail & Related papers  (2026-02-09T06:30:25Z)
• Decoupled Audio-Visual Dataset Distillation [44.63243875072762]
  We propose DAVDD, a pretraining-based decoupled audio-visual distillation framework.<n>To address these challenges, we propose DAVDD, a pretraining-based decoupled audio-visual distillation framework.
  arXiv  Detail & Related papers  (2025-11-22T02:36:50Z)
• FLUID: Flow-Latent Unified Integration via Token Distillation for Expert Specialization in Multimodal Learning [1.912429179274357]
  We present textscFLUID-Flow-Latent Unified Integration via Token Distillation for Expert components.<n>textscFLUID contributes three core elements: (1) emphQ-transforms, learnable query tokens that distill and retain salient token-level features from modality-specific backbones; (2) a two-stage fusion scheme that enforces cross-modal consistency via contrastive alignment; and (3) a lightweight, load-balanced Mixture-of-Experts at prediction time.
  arXiv  Detail & Related papers  (2025-08-10T09:34:17Z)
• GAID: Frame-Level Gated Audio-Visual Integration with Directional Perturbation for Text-Video Retrieval [12.483734449829235]
  GAID is a framework that integrates audio and visual features under textual guidance.<n>DASP injects structure-aware perturbations into text embeddings, enhancing robustness and discrimination without incurring multi-pass inference.<n>Experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX show consistent state-of-the-art results with notable efficiency gains.
  arXiv  Detail & Related papers  (2025-08-03T10:44:24Z)
• Implicit Counterfactual Learning for Audio-Visual Segmentation [50.69377287012591]
  We propose the implicit counterfactual framework (ICF) to achieve unbiased cross-modal understanding.<n>Due to the lack of semantics, heterogeneous representations may lead to erroneous matches.<n>We introduce the multi-granularity implicit text (MIT) involving video-, segment- and frame-level as the bridge to establish the modality-shared space.
  arXiv  Detail & Related papers  (2025-07-28T11:46:35Z)
• TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models [123.17643568298116]
  We present TAViS, a novel framework that textbfcouples the knowledge of multimodal foundation models for cross-modal alignment.<n> effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision.<n>Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.
  arXiv  Detail & Related papers  (2025-06-13T03:19:47Z)
• Text-Queried Audio Source Separation via Hierarchical Modeling [53.94434504259829]
  We propose a hierarchical decomposition framework, HSM-TSS, that decouples the task into global-local semantic-guided feature separation and structure-preserving acoustic reconstruction.<n>A Q-Audio architecture is employed to align audio and text modalities, serving as pretrained global-semantic encoders.<n>Our method achieves state-of-the-art separation performance with data-efficient training while maintaining superior semantic consistency with queries in complex auditory scenes.
  arXiv  Detail & Related papers  (2025-05-27T11:00:38Z)
• Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence [83.15764564701706]
  We propose a novel framework that performs vision-language alignment by integrating Cauchy-Schwarz divergence with mutual information.<n>We find that the CS divergence seamlessly addresses the InfoNCE's alignment-uniformity conflict and serves complementary roles with InfoNCE.<n> Experiments on text-to-image generation and cross-modality retrieval tasks demonstrate the effectiveness of our method on vision-language alignment.
  arXiv  Detail & Related papers  (2025-02-24T10:29:15Z)
• Improving Joint Speech-Text Representations Without Alignment [92.60384956736536]
  We show that joint speech-text encoders naturally achieve consistent representations across modalities by disregarding sequence length. We argue that consistency losses could forgive length differences and simply assume the best alignment.
  arXiv  Detail & Related papers  (2023-08-11T13:28:48Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.