Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2608.20237v1
• Date: Thu, 20 Aug 2026 16:28:28 GMT
• Title: Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models
• Abstract summary: RuleMaze is a controllable benchmark in which MLLMs must navigate mazes while obeying natural-language rules of varying complexity.<n>Disentangled Multimodal Planning (DMP) separates perception, execution, and rule verification through interpretable reasoning primitives.
• Score: 9.477726328920694
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) combine linguistic reasoning with visual perception, yet their ability to perform visual spatial planning under explicit or previously unseen rule constraints remains underexplored. This setting requires models to jointly understand spatial layouts, interpret natural-language rules, and plan valid actions accordingly. To address this gap, we introduce RuleMaze, a controllable benchmark in which MLLMs must navigate mazes while obeying natural-language rules of varying complexity. RuleMaze isolates rule-compliant spatial planning by requiring accurate perception, rule interpretation, and constrained action planning. To enable scalable and systematic rule construction, we propose Language-Logic-Function Hybridization, which automatically generates natural-language rules and translates them into logical representations and executable validators, eliminating manual rule engineering. To improve rule following and generalization, we introduce Disentangled Multimodal Planning (DMP), which separates perception, execution, and rule verification through interpretable reasoning primitives. By disentangling these components, DMP facilitates systematic generalization to more complex and previously unseen rules, while providing transparent intermediate planning traces. Experiments demonstrate that DMP substantially improves rule compliance and planning success compared to end-to-end textual planning baselines. Overall, RuleMaze establishes a principled benchmark for studying grounded and interpretable rule-based spatial planning in MLLMs. Code is available at https://github.com/oceanflowlab/RuleMaze.

Related papers

• STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models [9.560201418658144]
  Vision-language-action (VLA) models have shown impressive generalization, but often lack interpretability.<n>We propose a hierarchical framework that uses Signal Temporal Logic (STL) as a shared representation connecting high-level language understanding with low-level robot execution.<n>We evaluate the approach on a real-world tabletop domain, demonstrating how formal specifications can improve the precision, reliability, and interpretability of language-conditioned robot planning.
  arXiv  Detail & Related papers  (2026-07-20T23:26:32Z)
• Abductive Logical Rule Induction by Bridging Inductive Logic Programming and Multimodal Large Language Models [22.49558896794021]
  We propose ILP-CoT, a method that bridges Inductive Logic Programming (ILP) and Multimodal Large Language Models (MLLMs)<n>The task involves both discovering logical facts and inducing logical rules from a small number of unstructured textual or visual inputs.<n>Our approach automatically builds ILP tasks with pruned search spaces based on the rule structure proposals from MLLMs.
  arXiv  Detail & Related papers  (2025-09-26T04:56:19Z)
• Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts [5.520335305387487]
  We propose a novel prompting strategy Multi-Lingual Prompt, namely MLPrompt.<n> MLPrompt translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it.<n>We introduce a framework integrating MLPrompt with an auto-checking mechanism for structured data generation, with a specific case study in text-to-MIP instances.
  arXiv  Detail & Related papers  (2024-09-17T10:33:27Z)
• Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts [14.76420070558434]
  Rule extrapolation describes OOD scenarios, where the prompt violates at least one rule. We focus on formal languages, which are defined by the intersection of rules. We lay the first stones of a normative theory of rule extrapolation, inspired by the Solomonoff prior in algorithmic information theory.
  arXiv  Detail & Related papers  (2024-09-09T22:36:35Z)
• Symbolic Working Memory Enhances Language Models for Complex Rule Application [87.34281749422756]
  Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning. We propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. Our framework iteratively performs symbolic rule grounding and LLM-based rule implementation.
  arXiv  Detail & Related papers  (2024-08-24T19:11:54Z)
• DECIDER: A Dual-System Rule-Controllable Decoding Framework for Language Generation [57.07295906718989]
  Constrained decoding approaches aim to control the meaning or style of text generated by pre-trained large language (Ms also PLMs) for various tasks at inference time.<n>These methods often guide plausible continuations by greedily and explicitly selecting targets.<n>Inspired by cognitive dual-process theory, we propose a novel decoding framework DECIDER.
  arXiv  Detail & Related papers  (2024-03-04T11:49:08Z)
• SoFA: Shielded On-the-fly Alignment via Priority Rule Following [90.32819418613407]
  This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog. We present PriorityDistill, a semi-automated approach for distilling priority following signals from simulations to ensure robust rule integration and adherence.
  arXiv  Detail & Related papers  (2024-02-27T09:52:27Z)
• Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs [87.34281749422756]
  Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. We propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic.
  arXiv  Detail & Related papers  (2024-02-18T03:38:51Z)
• LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning [65.86754998249224]
  We develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach.
  arXiv  Detail & Related papers  (2023-12-30T02:53:45Z)
• ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning [107.61997887260056]
  We propose a novel framework, ChatRule, unleashing the power of large language models for mining logical rules over knowledge graphs. Specifically, the framework is initiated with an LLM-based rule generator, leveraging both the semantic and structural information of KGs. To refine the generated rules, a rule ranking module estimates the rule quality by incorporating facts from existing KGs.
  arXiv  Detail & Related papers  (2023-09-04T11:38:02Z)
• Building Rule Hierarchies for Efficient Logical Rule Learning from Knowledge Graphs [20.251630903853016]
  We propose new methods for pruning unpromising rules using rule hierarchies. We show that the application of HPMs is effective in removing unpromising rules.
  arXiv  Detail & Related papers  (2020-06-29T16:33:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.