SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations

• URL: http://arxiv.org/abs/2603.18271v2
• Date: Fri, 20 Mar 2026 01:51:13 GMT
• Title: SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations
• Abstract summary: Ambiguity poses a major challenge to large language models (LLMs) used as robotic planners.<n>We present Scene Graph-Chain-of-Thought (SG-CoT), a two-stage framework where LLMs iteratively query a scene graph representation of the environment to detect and clarify ambiguities.
• Score: 0.6299766708197883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ambiguity poses a major challenge to large language models (LLMs) used as robotic planners. In this letter, we present Scene Graph-Chain-of-Thought (SG-CoT), a two-stage framework where LLMs iteratively query a scene graph representation of the environment to detect and clarify ambiguities. First, a structured scene graph representation of the environment is constructed from input observations, capturing objects, their attributes, and relationships with other objects. Second, the LLM is equipped with retrieval functions to query portions of the scene graph that are relevant to the provided instruction. This grounds the reasoning process of the LLM in the observation, increasing the reliability of robotic planners under ambiguous situations. SG-CoT also allows the LLM to identify the source of ambiguity and pose a relevant disambiguation question to the user or another robot. Extensive experimentation demonstrates that SG-CoT consistently outperforms prior methods, with a minimum of 10% improvement in question accuracy and a minimum success rate increase of 4% in single-agent and 15% in multi-agent environments, validating its effectiveness for more generalizable robot planning.

Related papers

• LLM as Clinical Graph Structure Refiner: Enhancing Representation Learning in EEG Seizure Diagnosis [19.853206603792824]
  EEG signals are vital for automated seizure detection, but their inherent noise makes robust representation learning challenging.<n>Existing graph construction methods, whether correlation-based or learning-based, often generate redundant or irrelevant edges due to the noisy nature of EEG data.<n>Motivated by the remarkable reasoning and contextual understanding capabilities of large language models (LLMs), we explore the idea of using LLMs as graph edge refiners.
  arXiv  Detail & Related papers  (2026-04-30T17:57:12Z)
• Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs [18.414159451507153]
  Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries.<n>Existing Vision-Language Models (VLMs) commonly address zero-shot REC by measuring feature similarities between textual queries and image regions.<n>We propose textbfSGREC, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries.
  arXiv  Detail & Related papers  (2026-03-26T04:05:30Z)
• KGLAMP: Knowledge Graph-guided Language model for Adaptive Multi-robot Planning and Replanning [5.713977302944038]
  We introduce KGLAMP, a knowledge-graph-guided LLM planning framework for heterogeneous multi-robot teams.<n>The framework maintains a structured knowledge graph encoding object relations, spatial reachability, and robot capabilities.<n>Experiments on the MAT-THOR benchmark show that KGLAMP improves performance by at least 25.5% over both LLM-only and PDDL-based variants.
  arXiv  Detail & Related papers  (2026-02-04T01:46:02Z)
• GLOW: Graph-Language Co-Reasoning for Agentic Workflow Performance Prediction [51.83437071408662]
  We propose GLOW, a unified framework for AW performance prediction.<n>GLOW combines the graph-structure modeling capabilities of GNNs with the reasoning power of LLMs.<n>Experiments on FLORA-Bench show that GLOW outperforms state-of-the-art baselines in prediction accuracy and ranking utility.
  arXiv  Detail & Related papers  (2025-12-11T13:30:46Z)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning [63.109585527799005]
  GroundingAgent is a visual grounding framework that operates without task-specific fine-tuning.<n>It achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks.<n>It also offers strong interpretability, transparently illustrating each reasoning step.
  arXiv  Detail & Related papers  (2025-11-24T03:11:08Z)
• Exemplar-Guided Planing: Enhanced LLM Agent for KGQA [15.06175564601295]
  Large Language Models (LLMs) as interactive agents show significant promise in Knowledge Graph Question Answering (KGQA)<n>LLMs often struggle with the semantic gap between natural language queries and structured knowledge graph representations.<n>We propose a novel framework, Exemplar-Guided Planning (EGP), which enhances the planning capabilities of LLM agents for KGQA.
  arXiv  Detail & Related papers  (2025-10-17T03:43:06Z)
• Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enriching [61.824094419641575]
  Large Language Models (LLMs) struggle with hallucinations and factual errors in knowledge-intensive scenarios like knowledge graph question answering (KGQA)<n>We attribute this to the semantic gap between structured knowledge graphs (KGs) and unstructured queries, caused by inherent differences in their focuses and structures.<n>Existing methods usually employ resource-intensive, non-scalable reasoning on vanilla KGs, but overlook this gap.<n>We propose a flexible framework, Enrich-on-Graph (EoG), which leverages LLMs' prior knowledge to enrich KGs, bridge the semantic gap between graphs and queries.
  arXiv  Detail & Related papers  (2025-09-25T06:48:52Z)
• GRIL: Knowledge Graph Retrieval-Integrated Learning with Large Language Models [59.72897499248909]
  We propose a novel graph retriever trained end-to-end with Large Language Models (LLMs)<n>Within the extracted subgraph, structural knowledge and semantic features are encoded via soft tokens and the verbalized graph, respectively, which are infused into the LLM together.<n>Our approach consistently achieves state-of-the-art performance, validating the strength of joint graph-LLM optimization for complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-20T02:38:00Z)
• Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control [22.74768543283102]
  Graph-Fused Vision-Language-Action (GF-VLA) is a framework that enables dual-arm robotic systems to perform task-level reasoning and execution.<n>GF-VLA first extracts Shannon-information-based cues to identify hands and objects with the highest task relevance.<n>Cross-hand selection policy infers optimal assignment without explicit geometric reasoning.
  arXiv  Detail & Related papers  (2025-08-07T12:48:09Z)
• LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study [12.90392791734461]
  Large Language Models (LLMs) have paved the way for their expanding applications in embodied AI, robotics, and other real-world tasks.<n>Recent works have leveraged scene graphs, a structured representation that encodes entities, attributes, and their relationships in a scene.<n>We introduce Text-Scene Graph (TSG) Bench, a benchmark designed to assess LLMs' ability to understand scene graphs.
  arXiv  Detail & Related papers  (2025-05-26T04:45:12Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System [5.37125692728042]
  We propose an iterative-Guided Scene-Graph reasoning framework based on multi-agent Large Language Models (LLMs)<n>Two modules collaborate iteratively, enabling sequential reasoning and adaptive attention to graph information.<n>Our framework surpasses existing LLM-based approaches and baseline single-agent, tool-based Reason-while-Retrieve strategy in numerical Q&A and planning tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:50:38Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.