DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering

• URL: http://arxiv.org/abs/2607.23921v1
• Date: Mon, 27 Jul 2026 01:26:15 GMT
• Title: DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering
• Abstract summary: We introduce the Dynamic Dual-level Vision Transformer Fusion Network (DDVT) for answer grounding in visual question answering.<n>We propose a question-guided dynamic regional-level module (QGDR) that combines complementary image context through ROI Align and text content.<n>We also present a cross-modal multi-scale aggregation module (CMA) that enhances feature fusion between pixel-level and region-level features.
• Score: 12.19277105352502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Answer grounding in visual question answering aims to locate the region from a given natural language question associated with the visual content of an image, which has garnered significant attention due to its practical applications. In this paper, we introduce the Dynamic Dual-level Vision Transformer Fusion Network (DDVT) for answer grounding in visual question answering. Specifically, we propose a question-guided dynamic regional-level module (QGDR) that combines complementary image context through ROI Align and text content, enabling precise localization of text-related visual content. Moreover, we present a cross-modal multi-scale aggregation module (CMA) that enhances feature fusion between pixel-level and region-level features, facilitating the effective localization of visual content associated with grounded answers. Furthermore, we fuse the located visual content with text features to locate the region and provide answers to questions posed about the image. Experimental results demonstrate that our DDVT outperforms state-of-the-art methods on several widely-used benchmarks.

Related papers

• TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusion [55.34830989105704]
  Infrared and visible image fusion (IVF) aims to combine complementary information from both image modalities.<n>We introduce textual semantics at two levels: the mask semantic level and the text semantic level.<n>We propose Textual Semantic Guidance for infrared and visible image fusion, which guides the image synthesis process.
  arXiv  Detail & Related papers  (2025-06-20T03:53:07Z)
• FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity [68.15983300711355]
  FineCAPTION is a novel VLM that can recognize arbitrary masks as referential inputs and process high-resolution images for compositional image captioning at different levels. We introduce COMPOSITIONCAP, a new dataset for multi-grained region compositional image captioning, which introduces the task of compositional attribute-aware regional image captioning.
  arXiv  Detail & Related papers  (2024-11-23T02:20:32Z)
• TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding [91.30065932213758]
  Large Multimodal Models (LMMs) have sparked a surge in research aimed at harnessing their remarkable reasoning abilities. We propose TextCoT, a novel Chain-of-Thought framework for text-rich image understanding. Our method is free of extra training, offering immediate plug-and-play functionality.
  arXiv  Detail & Related papers  (2024-04-15T13:54:35Z)
• GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigation [52.65506307440127]
  We propose GeoVLN, which learns Geometry-enhanced visual representation based on slot attention for robust Visual-and-Language Navigation. We employ V&L BERT to learn a cross-modal representation that incorporate both language and vision informations.
  arXiv  Detail & Related papers  (2023-05-26T17:15:22Z)
• VITR: Augmenting Vision Transformers with Relation-Focused Learning for Cross-Modal Information Retrieval [0.3211619859724084]
  This paper introduces VITR, a novel network that enhances Vision Transformer (ViT) by extracting and reasoning about image region relations. ViTR incorporates a fusion module that combines the reasoned results with global knowledge to predict similarity scores between images and descriptions.
  arXiv  Detail & Related papers  (2023-02-13T13:34:29Z)
• Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search [61.24539128142504]
  Text-based person search (TBPS) is a challenging task that aims to search pedestrian images with the same identity from an image gallery given a query text. Most existing methods rely on explicitly generated local parts to model fine-grained correspondence between modalities. We propose an efficient joint Multi-level Alignment Network (MANet) for TBPS, which can learn aligned image/text feature representations between modalities at multiple levels.
  arXiv  Detail & Related papers  (2022-08-30T16:14:18Z)
• BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval [61.803481264081036]
  Content-Based Image Retrieval (CIR) aims to search for a target image by concurrently comprehending the composition of an example image and a complementary text. We tackle this task by a novel underlinetextbfBottom-up crunderlinetextbfOss-modal underlinetextbfSemantic compounderlinetextbfSition (textbfBOSS) with Hybrid Counterfactual Training framework.
  arXiv  Detail & Related papers  (2022-07-09T07:14:44Z)
• Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation [25.57530524167637]
  Visual dialogue needs to answer a series of coherent questions on the basis of understanding the visual environment. Visual grounding aims to explicitly locate related objects in the image guided by textual entities. multimodal incremental transformer encodes the multi-turn dialogue history combined with visual scene step by step according to the order of the dialogue and then generates a contextually and visually coherent response.
  arXiv  Detail & Related papers  (2021-09-17T11:39:29Z)
• Exploring Explicit and Implicit Visual Relationships for Image Captioning [11.82805641934772]
  In this paper, we explore explicit and implicit visual relationships to enrich region-level representations for image captioning. Explicitly, we build semantic graph over object pairs and exploit gated graph convolutional networks (Gated GCN) to selectively aggregate local neighbors' information. Implicitly, we draw global interactions among the detected objects through region-based bidirectional encoder representations from transformers.
  arXiv  Detail & Related papers  (2021-05-06T01:47:51Z)
• MAGNet: Multi-Region Attention-Assisted Grounding of Natural Language Queries at Phrase Level [6.47137925955334]
  We propose to utilize spatial attention networks for image-level visual-textual fusion. We refine region proposals with an in-network Region Proposal Network (RPN) and detect single or multiple regions for a phrase query. For such referring expression dataset ReferIt, our Multi-region Attention-assisted Grounding network (MAGNet) achieves over 12% improvement over the state-of-the-art.
  arXiv  Detail & Related papers  (2020-06-06T04:14:15Z)
• Visual Question Answering Using Semantic Information from Image Descriptions [2.6519061087638014]
  We propose a deep neural architecture that uses an attention mechanism which utilizes region based image features, the natural language question asked, and semantic knowledge extracted from the regions of an image to produce open-ended answers for questions asked in a visual question answering (VQA) task.
  arXiv  Detail & Related papers  (2020-04-23T04:35:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.