Describe Anything Model for Visual Question Answering on Text-rich Images

• URL: http://arxiv.org/abs/2507.12441v2
• Date: Sat, 02 Aug 2025 17:35:59 GMT
• Title: Describe Anything Model for Visual Question Answering on Text-rich Images
• Authors: Yen-Linh Vu, Dinh-Thang Duong, Truong-Binh Duong, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Nguyen, Jianhua Xing, Xingjian Li, Tianyang Wang, Ulas Bagci, Min Xu,
• Abstract summary: We introduce DAM-QA, a framework to harness the region-aware capabilities from DAM for the text-rich Visual Question Answering problem.<n>Our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA.<n>Results highlight the potential of DAM-like models for text-rich and broader VQA tasks when paired with efficient usage and integration strategies.
• Score: 7.618388911738171
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress has been made in region-aware vision-language modeling, particularly with the emergence of the Describe Anything Model (DAM). DAM is capable of generating detailed descriptions of any specific image areas or objects without the need for additional localized image-text alignment supervision. We hypothesize that such region-level descriptive capability is beneficial for the task of Visual Question Answering (VQA), especially in challenging scenarios involving images with dense text. In such settings, the fine-grained extraction of textual information is crucial to producing correct answers. Motivated by this, we introduce DAM-QA, a framework with a tailored evaluation protocol, developed to investigate and harness the region-aware capabilities from DAM for the text-rich VQA problem that requires reasoning over text-based information within images. DAM-QA incorporates a mechanism that aggregates answers from multiple regional views of image content, enabling more effective identification of evidence that may be tied to text-related elements. Experiments on six VQA benchmarks show that our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA. DAM-QA also achieves the best overall performance among region-aware models with fewer parameters, significantly narrowing the gap with strong generalist VLMs. These results highlight the potential of DAM-like models for text-rich and broader VQA tasks when paired with efficient usage and integration strategies. Our code is publicly available at https://github.com/Linvyl/DAM-QA.git.

Related papers

• A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents [0.619840955350879]
  Question-Answering from technical documents often involves questions whose answers are present in figures, such as flowcharts or flow diagrams.<n>We leverage graph representations of flowcharts obtained from Visual large Language Models (VLMs) and incorporate them in a text-based RAG system to show that this approach can enable image retrieval for QA in the telecom domain.
  arXiv  Detail & Related papers  (2025-07-25T07:36:13Z)
• ABC: Achieving Better Control of Multimodal Embeddings using VLMs [61.396457715710774]
  Visual embedding models excel at zero-shot tasks like visual retrieval and classification.<n>Existing CLIP-based approaches embed images and text independently, and fuse the result.<n>We introduce ABC, an open-source multimodal embedding model that uses a vision-language model backbone.
  arXiv  Detail & Related papers  (2025-03-01T03:29:02Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning [4.955697042432618]
  This paper proposes a novel agent-enhanced model collaboration framework called MoColl.<n>MoColl decomposes complex image captioning tasks into a series of interconnected question-answer subtasks.<n> Experimental results on radiology report generation validate the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2025-01-03T14:38:01Z)
• Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering [10.505845766495128]
  Multimodal large language models (MLLMs) have made significant progress in integrating visual and textual modalities.<n>We propose a novel framework based on multimodal retrieval-augmented generation (RAG)<n>RAG introduces structured scene graphs to enhance object recognition, relationship identification, and spatial understanding within images.
  arXiv  Detail & Related papers  (2024-12-30T13:16:08Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations.<n>We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions.<n>Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• See then Tell: Enhancing Key Information Extraction with Vision Grounding [54.061203106565706]
  We introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding. To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets.
  arXiv  Detail & Related papers  (2024-09-29T06:21:05Z)
• Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark [63.296342841358815]
  Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images.<n>The ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering.<n>We introduce MIRAGE, an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU.
  arXiv  Detail & Related papers  (2024-07-18T17:59:30Z)
• TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation [55.83319599681002]
  Text-VQA aims at answering questions that require understanding the textual cues in an image. We develop a new method to generate high-quality and diverse QA pairs by explicitly utilizing the existing rich text available in the scene context of each image.
  arXiv  Detail & Related papers  (2022-08-03T02:18:09Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling [12.233796960280944]
  Text-VQA (Visual Question Answering) aims at question answering through reading text information in images. LOGOS is a novel model which attempts to tackle this problem from multiple aspects.
  arXiv  Detail & Related papers  (2021-08-20T01:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.