FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Clues

• URL: http://arxiv.org/abs/2403.20026v1
• Date: Fri, 29 Mar 2024 07:28:50 GMT
• Title: FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Clues
• Abstract summary: Feature Swapping Multi-modal Reasoning (FSMR) model is designed to enhance multi-modal reasoning through feature swapping. FSMR incorporates a multi-modal cross-attention mechanism, facilitating the joint modeling of textual and visual information. Experiments on the PMR dataset demonstrate FSMR's superiority over state-of-the-art baseline models.
• Score: 20.587249765287183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal reasoning plays a vital role in bridging the gap between textual and visual information, enabling a deeper understanding of the context. This paper presents the Feature Swapping Multi-modal Reasoning (FSMR) model, designed to enhance multi-modal reasoning through feature swapping. FSMR leverages a pre-trained visual-language model as an encoder, accommodating both text and image inputs for effective feature representation from both modalities. It introduces a unique feature swapping module, enabling the exchange of features between identified objects in images and corresponding vocabulary words in text, thereby enhancing the model's comprehension of the interplay between images and text. To further bolster its multi-modal alignment capabilities, FSMR incorporates a multi-modal cross-attention mechanism, facilitating the joint modeling of textual and visual information. During training, we employ image-text matching and cross-entropy losses to ensure semantic consistency between visual and language elements. Extensive experiments on the PMR dataset demonstrate FSMR's superiority over state-of-the-art baseline models across various performance metrics.

Related papers

• DFM: Difference Feature Modeling with Text-Guided Gated Contrastive Loss for Remote Sensing Image Change Captioning [30.75534244211688]
  The encoder of Remote Sensing Image Change Captioning (RSICC) is to automatically generate descriptions of changes between remote sensing images captured at different time points.<n>We propose a Difference Feature Modeling (DFM) framework to replace the single autoregressive generation paradigm.<n>In particular, we introduce a Text-guided Gated Contrastive Loss (TGCL) to guide the vision to extract critical features from a text-modal perspective.
  arXiv  Detail & Related papers  (2026-06-25T11:49:21Z)
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion [10.837487434757882]
  Local Modality Substitution (LoMo) is a lightweight, architecture-agnostic data curation paradigm designed to provide supervision for cross-modal representational invariance between semantically equivalent text and image carriers.<n>LoMo significantly improves overall multimodal reasoning and yields deeper cross-modal fusion. Specifically, it delivers consistent gains across foundational models, improving over standard SFT by 2.67 points on LLaVA-Vision-1.5-8B and 2.82 points on Qwen3.5-9B.
  arXiv  Detail & Related papers  (2026-05-28T17:27:55Z)
• TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images [48.9420428682499]
  This paper proposes a text semantic-assisted cross-modal image registration framework, named TAR, for optical and SAR images.<n> TAR exploits text semantic priors from remote sensing scenes and land-cover categories to alleviate the modality gap.<n>It achieves stronger matching performance than several state-of-the-art methods and yields significant gains under large geometric deformations.
  arXiv  Detail & Related papers  (2026-05-12T12:50:36Z)
• Towards Robust Text-to-Image Person Retrieval: Multi-View Reformulation for Semantic Compensation [11.925230809051143]
  This paper proposes a semantic compensation framework (MVR) driven by Large Language Models (LLMs)<n>It enhances cross-modal representation consistency through multi-view semantic reformulation and feature compensation.<n> Experiments demonstrate that our method can improve the accuracy of the original model well without training and performs SOTA on three text-to-image person retrieval datasets.
  arXiv  Detail & Related papers  (2026-04-20T15:03:01Z)
• Query-Kontext: An Unified Multimodal Model for Image Generation and Editing [53.765351127477224]
  Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I)<n>We introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal kontext'' composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs.<n> Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases.
  arXiv  Detail & Related papers  (2025-09-30T17:59:46Z)
• MGCR-Net:Multimodal Graph-Conditioned Vision-Language Reconstruction Network for Remote Sensing Change Detection [55.702662643521265]
  We propose the multimodal graph-conditioned vision-language reconstruction network (MGCR-Net) to explore the semantic interaction capabilities of multimodal data.<n> Experimental results on four public datasets demonstrate that MGCR achieves superior performance compared to mainstream CD methods.
  arXiv  Detail & Related papers  (2025-08-03T02:50:08Z)
• DFR: A Decompose-Fuse-Reconstruct Framework for Multi-Modal Few-Shot Segmentation [21.536784761515968]
  DFR (Decompose, Fuse and Reconstruct) is a novel framework that addresses the challenge of effectively utilizing multi-modal guidance in few-shot segmentation.
  arXiv  Detail & Related papers  (2025-07-22T16:21:32Z)
• Multimodal-Aware Fusion Network for Referring Remote Sensing Image Segmentation [7.992331117310217]
  Referring remote sensing image segmentation (RRSIS) is a novel visual task in remote sensing images segmentation. We design a multimodal-aware fusion network (MAFN) to achieve fine-grained alignment and fusion between the two modalities.
  arXiv  Detail & Related papers  (2025-03-14T08:31:21Z)
• The Multi-Faceted Monosemanticity in Multimodal Representations [42.64636740703632]
  We leverage recent advancements in feature monosemanticity to extract interpretable features from deep multimodal models. Our findings reveal that this categorization aligns closely with human cognitive understandings of different modalities. These results indicate that large-scale multimodal models, equipped with task-agnostic interpretability tools, offer valuable insights into key connections and distinctions between different modalities.
  arXiv  Detail & Related papers  (2025-02-16T14:51:07Z)
• Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation [61.91492500828508]
  Few-shot 3D point cloud segmentation (FS-PCS) aims at generalizing models to segment novel categories with minimal support samples. We introduce a multimodal FS-PCS setup, utilizing textual labels and the potentially available 2D image modality. We propose a simple yet effective Test-time Adaptive Cross-modal (TACC) technique to mitigate training bias.
  arXiv  Detail & Related papers  (2024-10-29T19:28:41Z)
• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model [49.931663904599205]
  MaVEn is an innovative framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. We show that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
  arXiv  Detail & Related papers  (2024-08-22T11:57:16Z)
• Advanced Multimodal Deep Learning Architecture for Image-Text Matching [33.8315200009152]
  Image-text matching is a key multimodal task that aims to model the semantic association between images and text as a matching relationship. We introduce an advanced multimodal deep learning architecture, which combines the high-level abstract representation ability of deep neural networks for visual information with the advantages of natural language processing models for text semantic understanding. Experiments show that compared with existing image-text matching models, the optimized new model has significantly improved performance on a series of benchmark data sets.
  arXiv  Detail & Related papers  (2024-06-13T08:32:24Z)
• Few-Shot Relation Extraction with Hybrid Visual Evidence [3.154631846975021]
  We propose a multi-modal few-shot relation extraction model (MFS-HVE) MFS-HVE includes semantic feature extractors and multi-modal fusion components. Experiments conducted on two public datasets demonstrate that semantic visual information significantly improves the performance of few-shot relation prediction.
  arXiv  Detail & Related papers  (2024-03-01T18:20:11Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues [23.743431157431893]
  Conditional inference on joint textual and visual clues is a multi-modal reasoning task. We propose a Multi-modal Context Reasoning approach, named ModCR. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance.
  arXiv  Detail & Related papers  (2023-05-08T08:05:40Z)
• Multi-Modal Representation Learning with Text-Driven Soft Masks [48.19806080407593]
  We propose a visual-linguistic representation learning approach within a self-supervised learning framework. We generate diverse features for the image-text matching (ITM) task via soft-masking the regions in an image. We identify the relevant regions to each word by computing the word-conditional visual attention using multi-modal encoder.
  arXiv  Detail & Related papers  (2023-04-03T05:07:49Z)
• Named Entity and Relation Extraction with Multi-Modal Retrieval [51.660650522630526]
  Multi-modal named entity recognition (NER) and relation extraction (RE) aim to leverage relevant image information to improve the performance of NER and RE. We propose a novel Multi-modal Retrieval based framework (MoRe) MoRe contains a text retrieval module and an image-based retrieval module, which retrieve related knowledge of the input text and image in the knowledge corpus respectively.
  arXiv  Detail & Related papers  (2022-12-03T13:11:32Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)
• Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation [87.01669173673288]
  We propose an encoder fusion network (EFN), which transforms the visual encoder into a multi-modal feature learning network. A co-attention mechanism is embedded in the EFN to realize the parallel update of multi-modal features. The experiment results on four benchmark datasets demonstrate that the proposed approach achieves the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-05-05T02:27:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.