STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification

• URL: http://arxiv.org/abs/2603.00695v1
• Date: Sat, 28 Feb 2026 15:07:10 GMT
• Title: STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification
• Authors: Xingguo Xu, Zhanyu Liu, Weixiang Zhou, Yuansheng Gao, Junjie Cao, Yuhao Wang, Jixiang Luo, Dell Zhang,
• Abstract summary: We propose STMI, a novel multi-modal learning framework consisting of three key components.<n>We demonstrate the effectiveness and robustness of our proposed STMI framework in multi-modal ReID scenarios.
• Score: 14.549172375231729
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal object Re-Identification (ReID) aims to exploit complementary information from different modalities to retrieve specific objects. However, existing methods often rely on hard token filtering or simple fusion strategies, which can lead to the loss of discriminative cues and increased background interference. To address these challenges, we propose STMI, a novel multi-modal learning framework consisting of three key components: (1) Segmentation-Guided Feature Modulation (SFM) module leverages SAM-generated masks to enhance foreground representations and suppress background noise through learnable attention modulation; (2) Semantic Token Reallocation (STR) module employs learnable query tokens and an adaptive reallocation mechanism to extract compact and informative representations without discarding any tokens; (3) Cross-Modal Hypergraph Interaction (CHI) module constructs a unified hypergraph across modalities to capture high-order semantic relationships. Extensive experiments on public benchmarks (i.e., RGBNT201, RGBNT100, and MSVR310) demonstrate the effectiveness and robustness of our proposed STMI framework in multi-modal ReID scenarios.

Related papers

• Distinguishing Visually Similar Actions: Prompt-Guided Semantic Prototype Modulation for Few-Shot Action Recognition [18.527513690285364]
  Few-shot action recognition aims to enable models to quickly learn new action categories from limited labeled samples.<n>This paper proposes a CLIP-SPM framework, which includes three components to address the challenges of temporal modeling and visual similarity.
  arXiv  Detail & Related papers  (2025-12-22T05:13:58Z)
• RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning [15.670921552151775]
  RingMo-Agent is designed to handle multi-modal and multi-platform data.<n>It is supported by a large-scale vision-language dataset named RS-VL3M.<n>It proves effective in both visual understanding and sophisticated analytical tasks.
  arXiv  Detail & Related papers  (2025-07-28T12:39:33Z)
• TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models [123.17643568298116]
  We present TAViS, a novel framework that textbfcouples the knowledge of multimodal foundation models for cross-modal alignment.<n> effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision.<n>Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.
  arXiv  Detail & Related papers  (2025-06-13T03:19:47Z)
• IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification [60.38841251693781]
  We propose a novel framework to generate robust multi-modal object ReIDs.<n>Our framework uses Modal Prefixes and InverseNet to integrate multi-modal information with semantic guidance from inverted text.<n>Experiments on three multi-modal object ReID benchmarks demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2025-03-13T13:00:31Z)
• Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichment [21.36633828492347]
  Cross-Domain Multi-Modal Few-Shot Object Detection (CDMM-FSOD)<n>We introduce a meta-learning-based framework designed to leverage rich textual semantics as an auxiliary modality to achieve effective domain adaptation.<n>We evaluate the proposed method on common cross-domain object detection benchmarks and demonstrate that it significantly surpasses existing few-shot object detection approaches.
  arXiv  Detail & Related papers  (2025-02-23T06:59:22Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• Modality Prompts for Arbitrary Modality Salient Object Detection [57.610000247519196]
  This paper delves into the task of arbitrary modality salient object detection (AM SOD) It aims to detect salient objects from arbitrary modalities, eg RGB images, RGB-D images, and RGB-D-T images. A novel modality-adaptive Transformer (MAT) will be proposed to investigate two fundamental challenges of AM SOD.
  arXiv  Detail & Related papers  (2024-05-06T11:02:02Z)
• Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification [64.36210786350568]
  We propose a novel learning framework named textbfEDITOR to select diverse tokens from vision Transformers for multi-modal object ReID. Our framework can generate more discriminative features for multi-modal object ReID.
  arXiv  Detail & Related papers  (2024-03-15T12:44:35Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.