Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

• URL: http://arxiv.org/abs/2603.23867v1
• Date: Wed, 25 Mar 2026 02:48:42 GMT
• Title: Can VLMs Reason Robustly? A Neuro-Symbolic Investigation
• Abstract summary: Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks.<n>It remains unclear whether they can reason robustly under distribution shifts.<n>We propose a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning.
• Score: 24.000218548502875
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks, yet it remains unclear whether they can reason robustly under distribution shifts. In this paper, we study covariate shifts in which the perceptual input distribution changes while the underlying prediction rules do not. To investigate this question, we consider visual deductive reasoning tasks, where a model is required to answer a query given an image and logical rules defined over the object concepts in the image. Empirically, we find that VLMs fine-tuned through gradient-based end-to-end training can achieve high in-distribution accuracy but fail to generalize under such shifts, suggesting that fine-tuning does not reliably induce the underlying reasoning function. This motivates a neuro-symbolic perspective that decouples perception from reasoning. However, we further observe that recent neuro-symbolic approaches that rely on black-box components for reasoning can still exhibit inconsistent robustness across tasks. To address this issue, we propose VLC, a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning. In particular, task rules are compiled into a symbolic program, specifically a circuit, which executes the rules exactly over the object concepts recognized by the VLM. Experiments on three visual deductive reasoning tasks with distinct rule sets show that VLC consistently achieves strong performance under covariate shifts, highlighting its ability to support robust reasoning.

Related papers

• Advantage-Guided Gate: Reshaping Open-Ended Reasoning for Vision-Based Spatial Intelligence [70.8951332710039]
  Multimodal large language models (MLLMs) have demonstrated significant potential in complex spatial scene understanding and reasoning tasks.<n>However, their open-ended reasoning process is prone to decision errors and error accumulation, leading to instability in answer quality.<n>We propose an advantage-guided gating framework that dynamically intervenes in and corrects deviations during the reasoning process.
  arXiv  Detail & Related papers  (2026-08-08T07:46:04Z)
• Bad Seeing or Bad Thinking? Rewarding Perception for Vision-Language Reasoning [45.319525299206866]
  We argue that the root cause of this trade-off is an ambiguity in modality credit assignment.<n>We introduce a reinforcement learning framework that improves perception-reasoning synergy.
  arXiv  Detail & Related papers  (2026-05-13T19:23:53Z)
• Weakly Supervised Concept Learning for Object-centric Visual Reasoning [3.6959117433332724]
  Neurosymbolic systems promise to combine deep neural network's (DNN) processing of raw sensor inputs with few-shot performance of symbolic artificial intelligence.<n>This paper introduces an efficient weak supervision scheme for the perception stage to ground its output symbols for logical induction in object-centric reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-05T22:50:43Z)
• SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models [60.088066516175026]
  We introduce a benchmark designed to evaluate the spatial logical reasoning capabilities of Vision-Language Models (VLMs)<n>We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning.<n>We propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs.
  arXiv  Detail & Related papers  (2026-02-24T13:38:37Z)
• Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning [50.352417879912515]
  Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities.<n>We propose Group Causal Counterfactual Policy Optimization to explicitly train LLMs to learn generalizable reasoning patterns.<n>We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust.
  arXiv  Detail & Related papers  (2026-02-06T08:03:11Z)
• Mapping Faithful Reasoning in Language Models [1.978797083466893]
  We introduce Concept Walk, a framework for tracing how a model's internal stance evolves with respect to a concept direction during reasoning.<n>Unlike surface text, Concept Walk operates in activation space, projecting each reasoning step onto the concept direction learned from contrastive data.<n>We find that in 'easy' cases, perturbed CoTs are quickly ignored, indicating decorative reasoning, whereas in 'hard' cases, perturbations induce sustained shifts in internal activations, consistent with faithful reasoning.
  arXiv  Detail & Related papers  (2025-10-25T16:48:19Z)
• A Survey on Latent Reasoning [100.54120559169735]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities.<n>CoT reasoning that verbalizes intermediate steps limits the model's expressive bandwidth.<n>Latent reasoning tackles this bottleneck by performing multi-step inference entirely in the model's continuous hidden state.
  arXiv  Detail & Related papers  (2025-07-08T17:29:07Z)
• What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning [26.671128120554457]
  causal reasoning is fundamental to solving complex high-level reasoning tasks.<n>Existing benchmarks often include a mixture of reasoning questions.<n>We introduce VQA-Causal and VCR-Causal to isolate and rigorously evaluate causal reasoning abilities.
  arXiv  Detail & Related papers  (2025-06-01T07:17:46Z)
• Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformers [1.3812010983144802]
  Recent neuro-symbolic approaches have successfully extracted symbolic rule-sets from CNN-based models to enhance interpretability.<n>We propose a framework for symbolic rule extraction from Vision Transformers (ViTs) by introducing a sparse concept layer inspired by Sparse Autoencoders (SAEs)<n>Our method achieves a 5.14% better classification accuracy than the standard ViT while enabling symbolic reasoning.
  arXiv  Detail & Related papers  (2025-05-10T19:45:15Z)
• Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering [58.64831511644917]
  We introduce an interpretable by design model that factors model decisions into intermediate human-legible explanations. We show that our inherently interpretable system can improve 4.64% over a comparable black-box system in reasoning-focused questions.
  arXiv  Detail & Related papers  (2023-05-24T08:33:15Z)
• Expressive Explanations of DNNs by Combining Concept Analysis with ILP [0.3867363075280543]
  We use inherent features learned by the network to build a global, expressive, verbal explanation of the rationale of a feed-forward convolutional deep neural network (DNN) We show that our explanation is faithful to the original black-box model.
  arXiv  Detail & Related papers  (2021-05-16T07:00:27Z)
• Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning" [49.76230210108583]
  We propose a framework to isolate and evaluate the reasoning aspect of visual question answering (VQA) separately from its perception. We also propose a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models.
  arXiv  Detail & Related papers  (2020-06-20T08:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.