Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing

• URL: http://arxiv.org/abs/2602.19217v1
• Date: Sun, 22 Feb 2026 14:59:00 GMT
• Title: Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing
• Authors: Siran Li, Li Mi, Javiera Castillo-Navarro, Devis Tuia,
• Abstract summary: We propose a Knowledge-aware Remote Sensing Visual Question Generation model (KRSVQG)<n>The proposed model incorporates related knowledge triplets from external knowledge sources to broaden the question content.<n>KRSVQG utilizes a vision-language pre-training and fine-tuning strategy, enabling the model's adaptation to low data regimes.
• Score: 18.383561647568502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing semantic image retrieval. However, current automatically generated questions tend to be simplistic and template-based, which hinders the deployment of question answering or visual dialogue systems for real-world applications. To enrich and diversify the questions with both image content and commonsense knowledge, we propose a Knowledge-aware Remote Sensing Visual Question Generation model (KRSVQG). The proposed model incorporates related knowledge triplets from external knowledge sources to broaden the question content, while employing image captioning as an intermediary representation to ground questions to the corresponding images. Moreover, KRSVQG utilizes a vision-language pre-training and fine-tuning strategy, enabling the model's adaptation to low data regimes. To evaluate the proposed KRSVQG model, we construct two knowledge-aware remote sensing visual question generation datasets: the NWPU-300 dataset and the TextRS-300 dataset. Evaluations, including metrics and human assessment, demonstrate that KRSVQG outperforms existing methods and leads to rich questions, grounded in both image and domain knowledge. As a key practice in vision-language research, knowledge-aware visual question generation advances the understanding of image content beyond pixels, facilitating the development of knowledge-enriched vision-language systems with vision-grounded human commonsense.

Related papers

• Knowledge-aware Visual Question Generation for Remote Sensing Images [18.383561647568502]
  We propose a knowledge-aware remote sensing visual question generation model, KRSVQG.<n>The model takes an image and a related knowledge triplet from external knowledge sources as inputs.<n>Results on two datasets demonstrate that KRSVQG outperforms existing methods.
  arXiv  Detail & Related papers  (2026-02-22T15:18:01Z)
• RS-RAG: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model [16.343935641777268]
  We propose a novel Remote Sensing Retrieval-Augmented Generation (RS-RAG) framework, which consists of two key components.<n>The RS-RAG framework retrieves relevant knowledge based on image and/or text queries, and incorporates the retrieved content into a knowledge-augmented prompt.<n>We validated the effectiveness of our approach on three representative vision-language tasks, including image captioning, image classification, and visual question answering, where RS-RAG significantly outperformed state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T12:13:43Z)
• Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook [85.43403500874889]
  Retrieval-augmented generation (RAG) has emerged as a pivotal technique in artificial intelligence (AI)<n>Recent advancements in RAG for embodied AI, with a particular focus on applications in planning, task execution, multimodal perception, interaction, and specialized domains.
  arXiv  Detail & Related papers  (2025-03-23T10:33:28Z)
• Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference [107.53380946417003]
  We propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. We develop a simple methodology to self-learn the visual hints without introducing any additional human annotations.
  arXiv  Detail & Related papers  (2024-07-06T15:07:32Z)
• Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledge [10.074327344317116]
  We propose Q&A Prompts to equip AI models with robust cross-modality reasoning ability. We first use the image-answer pairs and the corresponding questions in a training set as inputs and outputs to train a visual question generation model. We then use an image tagging model to identify various instances and send packaged image-tag pairs into the visual question generation model to generate relevant questions with the extracted image tags as answers.
  arXiv  Detail & Related papers  (2024-01-19T14:22:29Z)
• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering [75.53187719777812]
  This paper revisits visual representation in knowledge-based visual question answering (VQA) We propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions. We achieve new state-of-the-art performance, i.e., 58.0% accuracy, surpassing previous state-of-the-art method by a large margin.
  arXiv  Detail & Related papers  (2022-06-02T17:59:56Z)
• VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering [79.22069768972207]
  We propose VQA-GNN, a new VQA model that performs bidirectional fusion between unstructured and structured multimodal knowledge to obtain unified knowledge representations. Specifically, we inter-connect the scene graph and the concept graph through a super node that represents the QA context. On two challenging VQA tasks, our method outperforms strong baseline VQA methods by 3.2% on VCR and 4.6% on GQA, suggesting its strength in performing concept-level reasoning.
  arXiv  Detail & Related papers  (2022-05-23T17:55:34Z)
• K-LITE: Learning Transferable Visual Models with External Knowledge [242.3887854728843]
  K-LITE (Knowledge-augmented Language-Image Training and Evaluation) is a strategy to leverage external knowledge to build transferable visual systems. In training, it enriches entities in natural language with WordNet and Wiktionary knowledge. In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts.
  arXiv  Detail & Related papers  (2022-04-20T04:47:01Z)
• Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering [18.926582410644375]
  Knowledge-based visual question answering (VQA) is a vision-language task that requires an agent to correctly answer image-related questions. We propose a novel model named dynamic knowledge memory enhanced multi-step graph reasoning (DMMGR) Our model achieves new state-of-the-art accuracy on the KRVQR and FVQA datasets.
  arXiv  Detail & Related papers  (2022-03-06T15:19:39Z)
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge [77.77684299758494]
  We extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG) First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network (KNN), which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. To handle the diversity and complexity of KG information, we propose a Contextualized Knowledge-aware Attentive Neural Network (CKANN), which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network (GCN) and comprehensively learns context-based and knowledge-based sentence representation via
  arXiv  Detail & Related papers  (2021-04-12T05:52:20Z)
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding [140.5911760063681]
  We propose a novel dataset named Knowledge-Routed Visual Question Reasoning for VQA model evaluation. We generate the question-answer pair based on both the Visual Genome scene graph and an external knowledge base with controlled programs.
  arXiv  Detail & Related papers  (2020-12-14T00:33:44Z)
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering [27.042604046441426]
  Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. In this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. We decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA.
  arXiv  Detail & Related papers  (2020-08-31T23:25:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.