From Reactive to Map-Based AI: Tuned Local LLMs for Semantic Zone Inference in Object-Goal Navigation

• URL: http://arxiv.org/abs/2603.08086v1
• Date: Mon, 09 Mar 2026 08:27:28 GMT
• Title: From Reactive to Map-Based AI: Tuned Local LLMs for Semantic Zone Inference in Object-Goal Navigation
• Abstract summary: Object-Goal Navigation (ObjectNav) requires an agent to find and navigate to a target object category in unknown environments.<n>We propose a transition from reactive AI to "Map-Based AI" by integrating LLM-based semantic inference with a hybrid topological-grid mapping system.<n>Our framework employs a fine-tuned Llama-2 model via Low-Rank Adaptation (LoRA) to infer semantic zone categories and target existence probabilities from verbalized object observations.
• Score: 0.7734726150561086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object-Goal Navigation (ObjectNav) requires an agent to find and navigate to a target object category in unknown environments. While recent Large Language Model (LLM)-based agents exhibit zero-shot reasoning, they often rely on a "reactive" paradigm that lacks explicit spatial memory, leading to redundant exploration and myopic behaviors. To address these limitations, we propose a transition from reactive AI to "Map-Based AI" by integrating LLM-based semantic inference with a hybrid topological-grid mapping system. Our framework employs a fine-tuned Llama-2 model via Low-Rank Adaptation (LoRA) to infer semantic zone categories and target existence probabilities from verbalized object observations. In this study, a "zone" is defined as a functional area described by the set of observed objects, providing crucial semantic co-occurrence cues for finding the target. This semantic information is integrated into a topological graph, enabling the agent to prioritize high-probability areas and perform systematic exploration via Traveling Salesman Problem (TSP) optimization. Evaluations in the AI2-THOR simulator demonstrate that our approach significantly outperforms traditional frontier exploration and reactive LLM baselines, achieving a superior Success Rate (SR) and Success weighted by Path Length (SPL).

Related papers

• STEM: Semantic Target Search and Exploration using MAVs in Cluttered Environments [17.009653719709213]
  This paper proposes a novel framework that utilizes a semantically-guided viewpoint planner to minimize target search and exploration time.<n>Specifically, we develop a planner that generates efficient semantic exploration plans by prioritizing viewpoints that likely lead to the target.
  arXiv  Detail & Related papers  (2026-05-30T15:00:58Z)
• SignNav: Leveraging Signage for Semantic Visual Navigation in Large-Scale Indoor Environments [57.79171900005793]
  Humans routinely leverage semantic hints provided by signage to navigate to destinations within Large-Scale Indoor environments.<n>This paper introduces a novel embodied navigation task, SignNav, which requires the agent to interpret semantic hint from signage and reason about the subsequent action based on current observation.<n>We propose the spatial-aware module grounds the semantic hint of signage into physical world, while the temporal-aware module captures long-range dependencies between historical states and current observation.
  arXiv  Detail & Related papers  (2026-03-17T06:36:26Z)
• RAGNav: A Retrieval-Augmented Topological Reasoning Framework for Multi-Goal Visual-Language Navigation [1.7508558850131373]
  Vision-Language Navigation (VLN) is evolving from single-point pathfinding toward the more challenging Multi-Goal VLN.<n>RAGNav is a framework that bridges the gap between semantic reasoning and physical structure.
  arXiv  Detail & Related papers  (2026-03-04T05:31:33Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• USS-Nav: Unified Spatio-Semantic Scene Graph for Lightweight UAV Zero-Shot Object Navigation [10.934912400963588]
  USS-Nav is a lightweight framework that incrementally constructs a Unified Spatio-Semantic scene graph.<n>It enables efficient Large Language Model (LLM)-augmented Zero-Shot Object Navigation in unknown environments.<n>Our framework outperforms state-of-the-art methods in terms of computational efficiency and real-time update frequency.
  arXiv  Detail & Related papers  (2026-01-31T13:10:02Z)
• LGR: LLM-Guided Ranking of Frontiers for Object Goal Navigation [1.1874952582465603]
  This study aims to enhance recent modular mapless OGN systems by leveraging the commonsense reasoning capabilities of large language models (LLMs)<n>We address the challenge of determining the visiting order in frontier-based exploration by framing it as a frontier ranking problem.<n>Our approach is grounded in recent findings that, while LLMs cannot determine the absolute value of a frontier, they excel at evaluating the relative value between multiple frontiers viewed within a single image using the view image as context.
  arXiv  Detail & Related papers  (2025-03-26T05:15:26Z)
• Diffusion as Reasoning: Enhancing Object Navigation via Diffusion Model Conditioned on LLM-based Object-Room Knowledge [9.465351278799016]
  We propose a novel approach to enhancing the ObjectNav task.<n>We train a diffusion model to learn the statistical distribution patterns of objects in semantic maps.<n>Using the map of the explored regions during navigation as the condition to generate the map of the unknown regions, we realize the long-term goal reasoning of the target object.
  arXiv  Detail & Related papers  (2024-10-29T08:10:06Z)
• Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigation [1.2473780585666772]
  Most Vision-and-Language Navigation (VLN) algorithms are prone to making inaccurate decisions due to their lack of visual common sense and limited reasoning capabilities. We propose a Hierarchical Spatial Proximity Reasoning (HSPR) method to help the agent build a knowledge base of hierarchical spatial proximity. We validate our approach with experiments on publicly available datasets including REVERIE, SOON, R2R, and R4R.
  arXiv  Detail & Related papers  (2024-03-18T07:51:22Z)
• Probable Object Location (POLo) Score Estimation for Efficient Object Goal Navigation [15.623723522165731]
  We introduce a novel framework centered around the Probable Object Location (POLo) score. We further enhance the framework's practicality by introducing POLoNet, a neural network trained to approximate the computationally intensive POLo score. Our experiments, involving the first phase of the OVMM 2023 challenge, demonstrate that an agent equipped with POLoNet significantly outperforms a range of baseline methods.
  arXiv  Detail & Related papers  (2023-11-14T08:45:32Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers [94.46825166907831]
  We present a training-free solution to tackle the object goal navigation problem in Embodied AI. Our method builds a structured scene representation based on the classic visual simultaneous localization and mapping (V-SLAM) framework. Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers.
  arXiv  Detail & Related papers  (2023-05-26T13:38:33Z)
• Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation [58.3480730643517]
  We present LGX, a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON) Our approach makes use of Large Language Models (LLMs) for this task. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline.
  arXiv  Detail & Related papers  (2023-03-06T20:19:19Z)
• Object Goal Navigation using Goal-Oriented Semantic Exploration [98.14078233526476]
  This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. We propose a modular system called, Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently.
  arXiv  Detail & Related papers  (2020-07-01T17:52:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.