TFANet: Three-Stage Image-Text Feature Alignment Network for Robust Referring Image Segmentation

• URL: http://arxiv.org/abs/2509.13070v1
• Date: Tue, 16 Sep 2025 13:26:58 GMT
• Title: TFANet: Three-Stage Image-Text Feature Alignment Network for Robust Referring Image Segmentation
• Authors: Qianqi Lu, Yuxiang Xie, Jing Zhang, Shiwei Zou, Yan Chen, Xidao Luan,
• Abstract summary: This paper proposes TFANet, a Three-stage Image-Text Feature Alignment Network.<n>It enhances multimodal alignment through a hierarchical framework comprising three stages: Knowledge Plus Stage (KPS), Knowledge Fusion Stage (KFS), and Knowledge Intensification Stage (KIS)<n>In the KPS, we design the Multiscale Linear Cross-Attention Module (MLAM), which establishes rich and efficient alignment between image regions and different granularities of linguistic descriptions.<n>The KFS further strengthens feature alignment through the Cross-modal Feature Scanning Module (CFSM), which applies multimodal selective scanning to capture long-range dependencies
• Score: 8.48847068018671
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Image Segmentation (RIS) is a task that segments image regions based on language expressions, requiring fine-grained alignment between two modalities. However, existing methods often struggle with multimodal misalignment and language semantic loss, especially in complex scenes containing multiple visually similar objects, where uniquely described targets are frequently mislocalized or incompletely segmented. To tackle these challenges, this paper proposes TFANet, a Three-stage Image-Text Feature Alignment Network that systematically enhances multimodal alignment through a hierarchical framework comprising three stages: Knowledge Plus Stage (KPS), Knowledge Fusion Stage (KFS), and Knowledge Intensification Stage (KIS). In the first stage, we design the Multiscale Linear Cross-Attention Module (MLAM), which facilitates bidirectional semantic exchange between visual features and textual representations across multiple scales. This establishes rich and efficient alignment between image regions and different granularities of linguistic descriptions. Subsequently, the KFS further strengthens feature alignment through the Cross-modal Feature Scanning Module (CFSM), which applies multimodal selective scanning to capture long-range dependencies and construct a unified multimodal representation. This is essential for modeling long-range cross-modal dependencies and enhancing alignment accuracy in complex scenes. Finally, in the KIS, we propose the Word-level Linguistic Feature-guided Semantic Deepening Module (WFDM) to compensate for semantic degradation introduced in earlier stages.

Related papers

• Multi-Grained Text-Guided Image Fusion for Multi-Exposure and Multi-Focus Scenarios [12.461120447513487]
  Multi-grained Text-guided Image Fusion (MTIF) is a novel fusion paradigm with three key designs.<n>First, it introduces multi-grained textual descriptions that separately capture fine details, structural cues, and semantic content.<n>Second, it involves supervision signals at each to facilitate alignment between visual and textual features.<n>Third, it adopts a saliency-driven enrichment module to augment training data with dense semantic content.
  arXiv  Detail & Related papers  (2025-12-23T17:55:35Z)
• SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment [8.657941729790599]
  We introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity.<n>Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches.<n>Experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance.
  arXiv  Detail & Related papers  (2025-11-03T09:41:32Z)
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• Visual Semantic Description Generation with MLLMs for Image-Text Matching [7.246705430021142]
  We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantics.<n>Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency.
  arXiv  Detail & Related papers  (2025-07-11T13:38:01Z)
• ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification [52.405499816861635]
  Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI)<n>We propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification.
  arXiv  Detail & Related papers  (2025-02-12T13:28:46Z)
• SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches [116.1810651297801]
  SketchYourSeg establishes freehand sketches as a powerful query modality for subjective image segmentation.<n>Our evaluations demonstrate superior performance over existing approaches across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-01-27T13:07:51Z)
• Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation [12.893224628061516]
  The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression.<n>We propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges.<n>Our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets.
  arXiv  Detail & Related papers  (2025-01-01T14:24:04Z)
• Linguistic Query-Guided Mask Generation for Referring Image Segmentation [10.130530501400079]
  Referring image segmentation aims to segment the image region of interest according to the given language expression. We propose an end-to-end framework built on transformer to perform Linguistic query-Guided mask generation.
  arXiv  Detail & Related papers  (2023-01-16T13:38:22Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)
• AF$_2$: Adaptive Focus Framework for Aerial Imagery Segmentation [86.44683367028914]
  Aerial imagery segmentation has some unique challenges, the most critical one among which lies in foreground-background imbalance. We propose Adaptive Focus Framework (AF$), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations. AF$ has significantly improved the accuracy on three widely used aerial benchmarks, as fast as the mainstream method.
  arXiv  Detail & Related papers  (2022-02-18T10:14:45Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)
• CMF: Cascaded Multi-model Fusion for Referring Image Segmentation [24.942658173937563]
  We address the task of referring image segmentation (RIS), which aims at predicting a segmentation mask for the object described by a natural language expression. We propose a simple yet effective Cascaded Multi-modal Fusion (CMF) module, which stacks multiple atrous convolutional layers in parallel. Experimental results on four benchmark datasets demonstrate that our method outperforms most state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-16T08:18:39Z)
• Linguistic Structure Guided Context Modeling for Referring Image Segmentation [61.701577239317785]
  We propose a "gather-propagate-distribute" scheme to model multimodal context by cross-modal interaction. Our LSCM module builds a Dependency Parsing Tree Word Graph (DPT-WG) which guides all the words to include valid multimodal context of the sentence.
  arXiv  Detail & Related papers  (2020-10-01T16:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.