IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Models

• URL: http://arxiv.org/abs/2503.10110v2
• Date: Mon, 06 Oct 2025 19:17:38 GMT
• Title: IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Models
• Abstract summary: IMPACT is a novel motion planning framework that uses Vision-Language Models to infer environment semantics.<n>Our results over 3200 simulation and 200 real-world trials suggest that IMPACT enables efficient contact-rich motion planning in cluttered settings.
• Score: 3.3871564496652335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Motion planning involves determining a sequence of robot configurations to reach a desired pose, subject to movement and safety constraints. Traditional motion planning finds collision-free paths, but this is overly restrictive in clutter, where it may not be possible for a robot to accomplish a task without contact. In addition, contacts range from relatively benign (e.g. brushing a soft pillow) to more dangerous (e.g. toppling a glass vase), making it difficult to characterize which may be acceptable. In this paper, we propose IMPACT, a novel motion planning framework that uses Vision-Language Models (VLMs) to infer environment semantics, identifying which parts of the environment can best tolerate contact based on object properties and locations. Our approach generates an anisotropic cost map that encodes directional push safety. We pair this map with a contact-aware A* planner to find stable contact-rich paths. We perform experiments using 20 simulation and 10 real-world scenes and assess using task success rate, object displacements, and feedback from human evaluators. Our results over 3200 simulation and 200 real-world trials suggest that IMPACT enables efficient contact-rich motion planning in cluttered settings while outperforming alternative methods and ablations. Our project website is available at https://impact-planning.github.io/.

Related papers

• Navigating the Clutter: Waypoint-Based Bi-Level Planning for Multi-Robot Systems [54.887871365121775]
  Multi-robot control in cluttered environments is a challenging problem that involves complex physical constraints.<n>We propose a hybrid multi-robot control framework that jointly optimize task and motion planning.<n>Our approach consistently improves task success over motion-agnostic and VLA-based baselines.
  arXiv  Detail & Related papers  (2026-04-22T22:58:47Z)
• Vectorizing Projection in Manifold-Constrained Motion Planning for Real-Time Whole-Body Control [2.940492773889387]
  We present a CPU SIMD-accelerated manifold-constrained motion planner that revisits projection-based constraint satisfaction through the lens of parallelization.<n>Our approach achieves up to 100-1000x speed-ups over the state-of-the-art, making real-time constrained motion planning feasible for the first time.
  arXiv  Detail & Related papers  (2026-04-14T22:00:53Z)
• Direct Contact-Tolerant Motion Planning With Vision Language Models [64.48049143625063]
  Navigation in cluttered environments often requires robots to tolerate contact with movable or deformable objects.<n>We propose a direct contact-tolerant (DCT) planner, which integrates vision-language models (VLMs) into direct point perception and navigation.<n>We implement DCT in Isaac Sim and a real car-like robot, demonstrating that DCT achieves robust and efficient navigation in cluttered environments with movable obstacles.
  arXiv  Detail & Related papers  (2026-03-05T10:05:00Z)
• Sampling-Based Motion Planning with Scene Graphs Under Perception Constraints [12.4666901035596]
  We discuss the problem of perception-aware motion planning for high-DoF robots with multi-object monitoring constraints.<n>We propose MOPS-PRM, a roadmap-based motion planner, that integrates the perception cost of observing multiple objects or humans directly into motion planning for high-DoF robots.<n>Our method is extensively validated in both simulated and real-world experiments, achieving more than 36% improvement in the average number of detected objects.
  arXiv  Detail & Related papers  (2026-03-03T20:46:00Z)
• To Move or Not to Move: Constraint-based Planning Enables Zero-Shot Generalization for Interactive Navigation [14.745622942938532]
  In real-world scenarios, such as home environments and warehouses, clutter can block all routes.<n>We introduce the Lifelong Interactive Navigation problem, where a mobile robot can move clutter to forge its own path.<n>We propose an LLM-driven, constraint-based planning framework with active perception.
  arXiv  Detail & Related papers  (2026-02-23T17:10:00Z)
• Cross-Entropy Optimization of Physically Grounded Task and Motion Plans [15.999074191893657]
  Previous TAMP algorithms have focused mainly on computational performance, completeness, or optimality.<n>We use a GPU-parallelized physics simulator to compute realizations of plans with motion controllers.<n>We demonstrate our approach for a set of tasks where the robot is able to exploit the environment's geometry to move an object.
  arXiv  Detail & Related papers  (2025-12-12T13:59:23Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• Adaptive Interactive Navigation of Quadruped Robots using Large Language Models [14.14967096139099]
  We present a primitive tree for task planning with large language models (LLMs)<n>We adopt reinforcement learning to pre-train a comprehensive skill library containing versatile locomotion and interaction behaviors for motion planning.<n> integrated with the tree structure, the replanning mechanism allows for convenient node addition and pruning.
  arXiv  Detail & Related papers  (2025-03-29T02:17:52Z)
• MotionCom: Automatic and Motion-Aware Image Composition with LLM and Video Diffusion Prior [51.672193627686]
  MotionCom is a training-free motion-aware diffusion based image composition. It enables seamless integration of target objects into new scenes with dynamically coherent results.
  arXiv  Detail & Related papers  (2024-09-16T08:44:17Z)
• Neural MP: A Generalist Neural Motion Planner [75.82675575009077]
  We seek to do the same by applying data-driven learning at scale to the problem of motion planning. Our approach builds a large number of complex scenes in simulation, collects expert data from a motion planner, then distills it into a reactive generalist policy. We perform a thorough evaluation of our method on 64 motion planning tasks across four diverse environments.
  arXiv  Detail & Related papers  (2024-09-09T17:59:45Z)
• A Meta-Engine Framework for Interleaved Task and Motion Planning using Topological Refinements [51.54559117314768]
  Task And Motion Planning (TAMP) is the problem of finding a solution to an automated planning problem. We propose a general and open-source framework for modeling and benchmarking TAMP problems. We introduce an innovative meta-technique to solve TAMP problems involving moving agents and multiple task-state-dependent obstacles.
  arXiv  Detail & Related papers  (2024-08-11T14:57:57Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• Neural Potential Field for Obstacle-Aware Local Motion Planning [46.42871544295734]
  We propose a neural network model that returns a differentiable collision cost based on robot pose, obstacle map, and robot footprint. Our architecture includes neural image encoders, which transform obstacle maps and robot footprints into embeddings. Experiment on Husky UGV mobile robot showed that our approach allows real-time and safe local planning.
  arXiv  Detail & Related papers  (2023-10-25T05:00:21Z)
• Language-Conditioned Path Planning [68.13248140217222]
  Language-Conditioned Collision Functions (LACO) learns a collision function using only a single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for object annotations, point cloud data, or ground-truth object meshes. In both simulation and the real world, we demonstrate that LACO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe to collide, rather than prohibiting any collision.
  arXiv  Detail & Related papers  (2023-08-31T17:56:13Z)
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models [38.503337052122234]
  Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation. We aim to synthesize robot trajectories for a variety of manipulation tasks given an open-set of instructions and an open-set of objects. We demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions.
  arXiv  Detail & Related papers  (2023-07-12T07:40:48Z)
• QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors [69.75711933065378]
  We show that headset and controller pose can generate realistic full-body poses even in highly constrained environments. We discuss three features, the environment representation, the contact reward and scene randomization, crucial to the performance of the method.
  arXiv  Detail & Related papers  (2023-06-09T04:40:38Z)
• Synthesizing Diverse Human Motions in 3D Indoor Scenes [16.948649870341782]
  We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner. Existing approaches rely on training sequences that contain captured human motions and the 3D scenes they interact with. We propose a reinforcement learning-based approach that enables virtual humans to navigate in 3D scenes and interact with objects realistically and autonomously.
  arXiv  Detail & Related papers  (2023-05-21T09:22:24Z)
• Robot Active Neural Sensing and Planning in Unknown Cluttered Environments [0.0]
  Active sensing and planning in unknown, cluttered environments is an open challenge for robots intending to provide home service, search and rescue, narrow-passage inspection, and medical assistance. We present the active neural sensing approach that generates the kinematically feasible viewpoint sequences for the robot manipulator with an in-hand camera to gather the minimum number of observations needed to reconstruct the underlying environment. Our framework actively collects the visual RGBD observations, aggregates them into scene representation, and performs object shape inference to avoid unnecessary robot interactions with the environment.
  arXiv  Detail & Related papers  (2022-08-23T16:56:54Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• Sim-to-Real Transfer for Vision-and-Language Navigation [70.86250473583354]
  We study the problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions. Recent work on the task of Vision-and-Language Navigation (VLN) has achieved significant progress in simulation. To assess the implications of this work for robotics, we transfer a VLN agent trained in simulation to a physical robot.
  arXiv  Detail & Related papers  (2020-11-07T16:49:04Z)
• Learning Obstacle Representations for Neural Motion Planning [70.80176920087136]
  We address sensor-based motion planning from a learning perspective. Motivated by recent advances in visual recognition, we argue the importance of learning appropriate representations for motion planning. We propose a new obstacle representation based on the PointNet architecture and train it jointly with policies for obstacle avoidance.
  arXiv  Detail & Related papers  (2020-08-25T17:12:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.