Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning

• URL: http://arxiv.org/abs/2606.15651v1
• Date: Sun, 14 Jun 2026 07:42:34 GMT
• Title: Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning
• Abstract summary: Vision-Language Models (VLMs) are AI systems that process both images and text.<n>We propose a self-questioning framework that trains a VLM to break visual questions into smaller sub-questions.<n>We show that both self-questioning and standard reinforcement learning substantially improve accuracy over the untrained model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are AI systems that process both images and text, yet they often struggle with compositional visual reasoning questions that require chaining multiple steps together, such as identifying objects, counting them, and comparing the results. Existing approaches improve this reasoning by training models on human-written step-by-step explanations, but creating these annotations is expensive and difficult to scale. We propose a self-questioning framework that trains a VLM to break visual questions into smaller sub-questions and answer each one before producing a final response, using a reinforcement learning algorithm called Group Relative Policy Optimization (GRPO). The model is never shown examples of how to decompose questions, it discovers this behavior on its own, guided by a reward signal that scores whether the output contains sub-questions and whether the final answer is correct. We apply this framework to a 3-billion-parameter model, training on both synthetic scenes of geometric shapes (CLEVR) and real-world photographs (A-OKVQA). On A-OKVQA, both self-questioning and standard reinforcement learning substantially improve accuracy over the untrained model (52.2% and 51.6% vs. 46.8%). We introduce the first self-questioning VLM by rewarding not only the final answer like standard RL but additionally for generating intermediate sub-questions, enabling it to discover compositional decomposition strategies. These results suggest that teaching AI systems to ask themselves intermediate questions is a promising strategy for complex visual reasoning, particularly when the difficulty of a question warrants explicit step-by-step decomposition.

Related papers

• Molecular Identifier Visual Prompt and Verifiable Reinforcement Learning for Chemical Reaction Diagram Parsing [52.825281124618535]
  Reaction diagram parsing (RxnDP) is critical for extracting chemical synthesis information from literature.<n>Recent Vision-Language Models (VLMs) have emerged as a promising paradigm to automate this complex visual reasoning task.<n>This work enhances VLM-based RxnDP from two complementary perspectives: prompting representation and learning paradigms.
  arXiv  Detail & Related papers  (2026-03-16T09:17:05Z)
• Decomposition-Enhanced Training for Post-Hoc Attributions In Language Models [64.49342399229529]
  We argue that post-hoc attribution can be reframed as a reasoning problem, where answers are decomposed into constituent units, each tied to specific context.<n>We introduce DecompTune, a post-training method that teaches models to produce answer decompositions as intermediate reasoning steps.<n>Across extensive experiments and ablations, DecompTune substantially improves attribution quality, outperforming prior methods and matching or exceeding state-of-the-art frontier models.
  arXiv  Detail & Related papers  (2025-10-29T17:58:59Z)
• Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models [33.398631680508814]
  We propose Answer-Consistent Reinforcement Learning that modifies the GRPO algorithm with an auxiliary consistency check.<n>We design a consistency-verification reward that grants a high reward only if both the original and the post-shuffle answers agree and are correct.<n>We evaluate ACRE on challenging Video Reasoning benchmarks and multimodal math reasoning benchmarks, achieving an average 2.2% and 1.5% improvement.
  arXiv  Detail & Related papers  (2025-10-11T08:32:52Z)
• Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning [37.194825644787294]
  We train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs.<n>Our model, named Visionary-R1, outperforms strong multimodal models on multiple visual reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-05-20T17:58:35Z)
• Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning [58.86928947970342]
  Embodied-R is a framework combining large-scale Vision-Language Models for perception and small-scale Language Models for reasoning.<n>After training on only 5k embodied video samples, Embodied-R with a 3B LM matches state-of-the-art multimodal reasoning models.<n>Embodied-R also exhibits emergent thinking patterns such as systematic analysis and contextual integration.
  arXiv  Detail & Related papers  (2025-04-17T06:16:11Z)
• QIRL: Boosting Visual Question Answering via Optimized Question-Image Relation Learning [38.24779287568123]
  Current debiasing techniques fail to capture the superior relation between images and texts.<n>No prior work has examined the degree of input relevance in debiasing studies.<n>We propose a novel framework, which employs a generation-based self-supervised learning strategy.
  arXiv  Detail & Related papers  (2025-04-04T10:38:28Z)
• Admitting Ignorance Helps the Video Question Answering Models to Answer [82.22149677979189]
  We argue that models often establish shortcuts, resulting in spurious correlations between questions and answers.<n>We propose a novel training framework in which the model is compelled to acknowledge its ignorance when presented with an intervened question.<n>In practice, we integrate a state-of-the-art model into our framework to validate its effectiveness.
  arXiv  Detail & Related papers  (2025-01-15T12:44:52Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Learning Abstract Visual Reasoning via Task Decomposition: A Case Study in Raven Progressive Matrices [0.24475591916185496]
  In Raven Progressive Matrices, the task is to choose one of the available answers given a context. In this study, we propose a deep learning architecture based on the transformer blueprint. The multidimensional predictions obtained in this way are then directly juxtaposed to choose the answer.
  arXiv  Detail & Related papers  (2023-08-12T11:02:21Z)
• Successive Prompting for Decomposing Complex Questions [50.00659445976735]
  Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting. We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:03:38Z)
• SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency [64.67155167618894]
  We present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub-questions higher than irrelevant questions for an image, reasoning-question> pair. We show that SOrT improves model consistency by upto 6.5% points over existing baselines, while also improving visual grounding.
  arXiv  Detail & Related papers  (2020-10-20T05:15:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.