SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes

• URL: http://arxiv.org/abs/2601.05600v1
• Date: Fri, 09 Jan 2026 07:40:39 GMT
• Title: SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes
• Abstract summary: Multimodal large language models often struggle with faithful reasoning in complex visual scenes.<n>Existing preference-based approaches fail to address this challenge as they allow models to exploit language priors to bypass visual grounding.<n>We propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions.
• Score: 69.03114625064282
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models often struggle with faithful reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This reasoning unfaithfulness frequently manifests as hallucinated entities, mis-grounded relations, skipped steps, and over-specified reasoning. Existing preference-based approaches, typically relying on textual perturbations or answer-conditioned rationales, fail to address this challenge as they allow models to exploit language priors to bypass visual grounding. To address this, we propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. By identifying reasoning-critical nodes and perturbing them through four targeted strategies that mimic typical grounding failures, SceneAlign constructs hard negative rationales that remain linguistically plausible but are grounded in inaccurate visual facts. These contrastive pairs are used in Direct Preference Optimization to steer models toward fine-grained, structure-faithful reasoning. Across seven visual reasoning benchmarks, SceneAlign consistently improves answer accuracy and reasoning faithfulness, highlighting the effectiveness of grounding-aware alignment for multimodal reasoning.

Related papers

• Where To Look? : Causal Tracing of Vision Encoders in VLM [0.6980863185266349]
  Vision-language models can describe an image with remarkable accuracy, yet what visual information actually drives their answers?<n>We investigate this question through causal tracing, and we observe that highly causal vision tokens often lie outside the target region.<n>We further investigate: can these models preserve visual structure when appearance cues are removed? and find that visual cues are exploited to understand visual structures.
  arXiv  Detail & Related papers  (2026-08-11T10:17:34Z)
• Improving Reasoning in Vision-Language Models via Perception Verified Self-Training [3.443084677278651]
  We propose a perception-verified self-training framework that enforces visually grounded reasoning.<n>First, our method employs a CoT template that disentangles perception from reasoning, enabling independent verification of visual understanding.<n>Using caption verification together with answer correctness, we partition the data into three subsets: easy (correct caption and conclusion), medium (correct caption but incorrect conclusion), and hard (incorrect caption)
  arXiv  Detail & Related papers  (2026-06-20T17:33:07Z)
• Thinking with Visual Grounding [55.778136782175615]
  We introduce visually grounded thinking, a reasoning process in which models interleave natural-language thoughts with explicit point or box groundings of the visual evidence used at each step.<n>To train this behavior, we construct a scalable synthesis pipeline that distills correct visual reasoning traces.<n>We propose grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards that score whether generated object references match the correct image evidence.
  arXiv  Detail & Related papers  (2026-06-15T02:28:34Z)
• RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought [50.36951914894845]
  Embodied reasoning requires models to perceive task-relevant objects and spaces in physical environments.<n>Current vision-language models rely on text-only or coordinate-augmented chain-of-thought.<n>We propose Pinned Chain-of-Thought, a structured reasoning paradigm that pins every reasoning step to visual evidence.
  arXiv  Detail & Related papers  (2026-06-14T11:30:42Z)
• Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning [82.86343313807158]
  We argue that a central challenge is how and when visual evidence is introduced into the reasoning process.<n>Motivated by this insight, we propose CSMR, a multimodal reasoning framework in which a language model controls the reasoning process.
  arXiv  Detail & Related papers  (2026-05-27T08:43:13Z)
• MIRAGE: A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks [9.397297838455238]
  We present MIRAGE, an evidence-centric framework designed to scaffold the exploration of "micro-interactions" in multi-figure artworks.<n>Results show that MIRAGE significantly improves identity consistency, reduces relational hallucinations, and increases the coverage of subtle interactions.
  arXiv  Detail & Related papers  (2026-04-26T16:25:30Z)
• MentisOculi: Revealing the Limits of Reasoning with Mental Imagery [63.285794947638614]
  We develop MentisOculi, a suite of multi-step reasoning problems amenable to visual solution.<n> evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance.<n>Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning.
  arXiv  Detail & Related papers  (2026-02-02T18:49:06Z)
• Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning [38.651924340946785]
  We formulate visual reasoning by means of video generation models.<n>We evaluate their capacity in two distinct regimes: Maze Navigation for sequential discrete planning with low visual change and Tangram Puzzle for continuous manipulation with high visual change.
  arXiv  Detail & Related papers  (2026-01-28T20:57:55Z)
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models [56.851611990473174]
  Reasoning over dynamic visual content remains a central challenge for large language models.<n>We propose a reinforcement learning approach that enhances both temporal precision and reasoning consistency.<n>The resulting model, Video R2, achieves consistently higher TAC, VAS, and accuracy across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-11-28T18:59:58Z)
• SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent [28.12183839499528]
  SceneWeaver is a framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement.<n>It can identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations.<n>It generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation.
  arXiv  Detail & Related papers  (2025-09-24T09:06:41Z)
• PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs [23.69973859198496]
  Multimodal Large Language Models (MLLMs) excel in vision-language tasks, such as image captioning and visual question answering.<n>They often suffer from over-reliance on spurious correlations, primarily due to linguistic priors that distract the model from leveraging actual visual information.<n>We introduce MMed-PostAlign, a post-multimodal alignment framework designed to enhance the visual understanding capabilities and mitigate the hallucinations of MLLMs.
  arXiv  Detail & Related papers  (2025-06-22T05:11:46Z)
• Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT [24.085953089267772]
  We show how OpenAI o3 and GPT-4o fail to grasp basic physical laws, spatial interactions, and causal effects in complex scenes.<n>We introduce MVPBench, a benchmark designed to rigorously evaluate visual physical reasoning through the lens of visual chain-of-thought (CoT)<n> Experimental results reveal a concerning trend: even cutting-edge MLLMs exhibit poor visual reasoning accuracy and weak image-text alignment in physical domains.
  arXiv  Detail & Related papers  (2025-05-30T03:48:59Z)
• Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models [58.64449765678416]
  We introduce landscape of thoughts (LoT) to inspect the reasoning trajectories with certain reasoning methods on any multi-choice dataset.<n>LoT distinguishes between strong and weak models, correct and incorrect answers, as well as different reasoning tasks.<n>We showcase this advantage by adapting LoT to a lightweight verifier that evaluates the correctness of trajectories.
  arXiv  Detail & Related papers  (2025-03-28T06:09:51Z)
• Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios [69.00444996464662]
  We propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables vision-language models to reason using visual crops corresponding to relevant entities.<n>Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting.
  arXiv  Detail & Related papers  (2025-01-08T18:31:16Z)
• GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narratives [69.36723767339001]
  We propose a novel framework named textitGPT4SGG to obtain more accurate and comprehensive scene graph signals. We show textitGPT4SGG significantly improves the performance of SGG models trained on image-caption data.
  arXiv  Detail & Related papers  (2023-12-07T14:11:00Z)
• VCD: Visual Causality Discovery for Cross-Modal Question Reasoning [11.161509939879428]
  We propose a visual question reasoning framework named Cross-Modal Question Reasoning (CMQR) To explicitly discover visual causal structure, the Visual Causality Discovery (VCD) architecture is proposed to find question-critical scene temporally. To align the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build an Interactive Visual-Linguistic Transformer (IVLT)
  arXiv  Detail & Related papers  (2023-04-17T08:56:16Z)
• Object-Centric Diagnosis of Visual Reasoning [118.36750454795428]
  This paper presents a systematical object-centric diagnosis of visual reasoning on grounding and robustness. We develop a diagnostic model, namely Graph Reasoning Machine. Our model replaces purely symbolic visual representation with probabilistic scene graph and then applies teacher-forcing training for the visual reasoning module.
  arXiv  Detail & Related papers  (2020-12-21T18:59:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.