Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

• URL: http://arxiv.org/abs/2604.16388v1
• Date: Fri, 27 Mar 2026 11:47:26 GMT
• Title: Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering
• Abstract summary: Rapidly-exploring random trees (RRTs) have been widely adopted for robot motion planning due to their robustness and theoretical guarantees.<n>Existing RRT-based planners require explicit goal configurations specified as numerical joint angles.<n>We propose visual-RRT, a motion planner that enables visual-goal planning by unifying gradient-based exploitation from differentiable robot rendering.
• Score: 21.043662113128438
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rapidly-exploring random trees (RRTs) have been widely adopted for robot motion planning due to their robustness and theoretical guarantees. However, existing RRT-based planners require explicit goal configurations specified as numerical joint angles, while many practical applications provide goal specifications through visual observations such as images or demonstration videos where precise goal configurations are unavailable. In this paper, we propose visual-RRT (vRRT), a motion planner that enables visual-goal planning by unifying gradient-based exploitation from differentiable robot rendering with sampling-based exploration from RRTs. We further introduce (i) a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, and (ii) inertial gradient tree expansion that inherits optimization states across tree branches for momentum-consistent gradient exploitation. Extensive experiments across various robot manipulators including Franka, UR5e, and Fetch demonstrate that vRRT achieves effective visual-goal planning in both simulated and real-world settings, bridging the gap between sampling-based planning and vision-centric robot applications. Our code is available at https://sgvr.kaist.ac.kr/Visual-RRT.

Related papers

• Sampling-Based Visibility Task Planning [5.156484100374059]
  We introduce two new sampling-based algorithms, FOV-PRM and FOV-RRT, designed to tackle visibility-based tasks.<n>FOV-PRM employs a hierarchical decomposition of the environment, leveraging the concept of visibility, to efficiently sample configurations with a clear line-of-sight to the target.<n>A specialized Inverse Kinematics solver enables FOV-RRT to "glance" in the direction of the target at opportune moments, facilitating the rapid discovery of key configurations.
  arXiv  Detail & Related papers  (2026-08-02T06:06:02Z)
• VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models [74.05545957865228]
  Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images and video frames in vision-language models (VLMs)<n>We introduce, a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations.<n>Our approach employs continuous relaxation and straight-through estimators to enable gradient-based search, solved via the Augmented Lagrangian method.
  arXiv  Detail & Related papers  (2026-04-16T16:21:05Z)
• CeRLP: A Cross-embodiment Robot Local Planning Framework for Visual Navigation [12.047848283467019]
  We propose a Cross-embodiment Robot Local Planning (CeRLP) framework for general visual navigation.<n>CeRLP abstracts visual information into a unified geometric formulation and applies to heterogeneous robots.<n> Experiments in simulation environments demonstrate that CeRLP outperforms comparative methods.
  arXiv  Detail & Related papers  (2026-03-20T03:17:18Z)
• A Unified Experimental Architecture for Informative Path Planning: from Simulation to Deployment with GuadalPlanner [69.43049144653882]
  This paper introduces a unified architecture that decouples high-level decision-making from vehicle-specific control.<n>The proposed architecture is realized through GuadalPlanner, which defines standardized interfaces between planning, sensing, and vehicle execution.
  arXiv  Detail & Related papers  (2026-02-11T10:02:31Z)
• Aerial Vision-Language Navigation with a Unified Framework for Spatial, Temporal and Embodied Reasoning [5.517595398768408]
  We present a unified aerial VLN framework that operates solely on ego monocular RGB observations and natural language instructions.<n>This task holds promise for real-world applications such as low-altitude inspection, search-and-rescue, and autonomous aerial delivery.
  arXiv  Detail & Related papers  (2025-12-09T14:25:24Z)
• Unified Linear Parametric Map Modeling and Perception-aware Trajectory Planning for Mobile Robotics [1.7495208770207367]
  We introduce a lightweight linear parametric map by first mapping data to a high-dimensional space, followed by a sparse random projection for dimensionality reduction.<n>For UAVs, our method grid and Euclidean Signed Distance Field (ESDF) maps.<n>For UGVs, the model characterizes terrain and provides closed-form gradients, enabling online planning to circumvent large holes.
  arXiv  Detail & Related papers  (2025-07-12T16:39:19Z)
• Open-World Drone Active Tracking with Goal-Centered Rewards [62.21394499788672]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations.<n>We propose DAT, the first open-world drone active air-to-ground tracking benchmark.<n>We also propose GC-VAT, which aims to improve the performance of drone tracking targets in complex scenarios.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• SCoTT: Strategic Chain-of-Thought Tasking for Wireless-Aware Robot Navigation in Digital Twins [78.53885607559958]
  We propose SCoTT, a wireless-aware path planning framework.<n>We show that SCoTT achieves path gains within 2% of DP-WA* while consistently generating shorter trajectories.<n>We also show the practical viability of our approach by deploying SCoTT as a ROS node within Gazebo simulations.
  arXiv  Detail & Related papers  (2024-11-27T10:45:49Z)
• Reinforcement Learning with Frontier-Based Exploration via Autonomous Environment [0.0]
  This research combines an existing Visual-Graph SLAM known as ExploreORB with reinforcement learning. The proposed algorithm aims to improve the efficiency and accuracy of ExploreORB by optimizing the exploration process of frontiers to build a more accurate map.
  arXiv  Detail & Related papers  (2023-07-14T12:19:46Z)
• Target-Driven Structured Transformer Planner for Vision-Language Navigation [55.81329263674141]
  We propose a Target-Driven Structured Transformer Planner (TD-STP) for long-horizon goal-guided and room layout-aware navigation. Specifically, we devise an Imaginary Scene Tokenization mechanism for explicit estimation of the long-term target. In addition, we design a Structured Transformer Planner which elegantly incorporates the explored room layout into a neural attention architecture for structured and global planning.
  arXiv  Detail & Related papers  (2022-07-19T06:46:21Z)
• Path Planning Followed by Kinodynamic Smoothing for Multirotor Aerial Vehicles (MAVs) [61.94975011711275]
  We propose a geometrically based motion planning technique textquotedblleft RRT*textquotedblright; for this purpose. In the proposed technique, we modified original RRT* introducing an adaptive search space and a steering function. We have tested the proposed technique in various simulated environments.
  arXiv  Detail & Related papers  (2020-08-29T09:55:49Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.