Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval

• URL: http://arxiv.org/abs/2609.15320v1
• Date: Mon, 14 Sep 2026 10:12:07 GMT
• Title: Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval
• Abstract summary: This work introduces Hypergraph-Regularized Gramian Volumes (HyVol)<n>HyVol is a training-time module that incorporates semantic relations prior to evaluating the original volume loss.<n>We train both backbones on a 150K-clip subset of VAST-27M and evaluate zero-shot performance on six benchmarks.
• Score: 5.4018351446007875
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Volume-based multimodal retrieval jointly scores a text query with a candidate's video, audio, and subtitle embeddings. While this approach captures higher-order within-candidate alignment, the score remains candidate-local, and semantically related training samples primarily serve as contrastive negatives. This work introduces Hypergraph-Regularized Gramian Volumes (HyVol), a training-time module that incorporates these semantic relations prior to evaluating the original volume loss. Document hyperedges connect the observed modalities of each candidate, whereas semantic hyperedges link candidates whose detached captions are mutual top-k neighbors. A shallow gated hyper-graph network applies residual corrections to the modality embeddings. Presence masks exclude unavailable streams from message passing, and identity padding preserves the determinant of the observed Gram submatrix without feature imputation. As refinement operates on embeddings rather than scores, the same construction applies to both Gram and HyperGram. We remove the hypergraph after training, leaving the backbone-only architecture, original scoring function, and retrieval cost unchanged. We train both backbones on a 150K-clip subset of VAST-27M and evaluate zero-shot performance on six benchmarks. Under the paired protocol, HyVol improves R@1 across all five retrieval benchmarks, with video-to-text gains reaching +8.3 on MSR-VTT and +7.6 on VATEX. Under missing-modality masking, the V2T margin remains positive in all experimental settings, although the T2V margin becomes slightly negative in four.

Related papers

• Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval [5.4018351446007875]
  Multimodal retrieval integrates video, audio, subtitles, and text.<n>Recent geometric aggregators treat all modalities symmetrically.<n>SCALAR is a query-conditioned aggregator that assigns relevance-based weights to each available modality.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment [37.61076424435405]
  Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence.<n>Standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap.<n>We propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations.
  arXiv  Detail & Related papers  (2026-09-08T03:18:58Z)
• Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval [20.130842837418726]
  We introduce HN-CLIP, which uses the text encoder's own text-text geometry to construct per-negative adaptive similarity margins.<n>The resulting objective requires only one caption-similarity matrix and a masked logit addition during training.<n>Experiments show that HN-CLIP improves over the strongest competitors by +2.5 to +4.0 R@1 while training 2.4x faster than GOAL and 5.4x faster than StructXLIP.
  arXiv  Detail & Related papers  (2026-08-19T04:16:46Z)
• ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation [63.38679125666075]
  Referring object segmentation (RVOS) requires segmenting a target specified by natural language throughout a video.<n>Recent agent approaches combine multimodal large language models with promptable segmentation models to perform RVOS without task-specific training.<n>We introduce ReflexTrack, a training-free, feedback-driven agent that closes the loop at both spatial and temporal levels.
  arXiv  Detail & Related papers  (2026-07-27T07:39:20Z)
• Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search [30.80883548177376]
  We present V-SPLADE, an inference-free sparse retriever for visual-document retrieval.<n>V-SPLADE improves average NDCG@5 across six visual-document retrieval benchmarks by +13.8pp over the same-scale dense baseline.<n>On an 18.7M-document corpus, it more than doubles R@5 over the same-scale dense baseline.
  arXiv  Detail & Related papers  (2026-05-29T07:01:45Z)
• $G^2$-Reader: Dual Evolving Graphs for Multimodal Document QA [53.491241153213565]
  $G2$-Reader is a dual-graph system for multimodal question answering.<n>$G2$-Reader with Qwen3-VL-32B-Instruct reaches 66.21% average accuracy, outperforming strong baselines and a standalone GPT-5 (53.08%)<n>On VisDoMBench across five multimodal domains, $G2$-Reader with Qwen3-VL-32B-Instruct reaches 66.21% average accuracy, outperforming strong baselines and a standalone GPT-5 (53.08%)
  arXiv  Detail & Related papers  (2026-01-29T17:52:54Z)
• Gumbel Reranking: Differentiable End-to-End Reranker Optimization [61.16471123356738]
  RAG systems rely on rerankers to identify relevant documents.<n> fine-tuning these models remains challenging due to the scarcity of annotated query-document pairs.<n>We propose Gumbel Reranking, an end-to-end training framework for rerankers aimed at minimizing the training-inference gap.
  arXiv  Detail & Related papers  (2025-02-16T13:23:39Z)
• HyperSMOTE: A Hypergraph-based Oversampling Approach for Imbalanced Node Classifications [2.172034690069702]
  We propose HyperSMOTE as a solution to alleviate the class imbalance issue in hypergraph learning. We synthesize new nodes based on samples from minority classes and their neighbors. In order to solve the problem on integrating the new node into the hypergraph, we train a decoder based on the original hypergraph incidence matrix.
  arXiv  Detail & Related papers  (2024-09-09T08:01:28Z)
• SimMatchV2: Semi-Supervised Learning with Graph Consistency [53.31681712576555]
  We introduce a new semi-supervised learning algorithm - SimMatchV2. It formulates various consistency regularizations between labeled and unlabeled data from the graph perspective. SimMatchV2 has been validated on multiple semi-supervised learning benchmarks.
  arXiv  Detail & Related papers  (2023-08-13T05:56:36Z)
• UATVR: Uncertainty-Adaptive Text-Video Retrieval [90.8952122146241]
  A common practice is to transfer text-video pairs to the same embedding space and craft cross-modal interactions with certain entities. We propose an Uncertainty-language Text-Video Retrieval approach, termed UATVR, which models each look-up as a distribution matching procedure.
  arXiv  Detail & Related papers  (2023-01-16T08:43:17Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.