CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation

• URL: http://arxiv.org/abs/2605.02600v2
• Date: Tue, 12 May 2026 11:25:37 GMT
• Title: CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation
• Abstract summary: Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate remarkable capabilities in high-level reasoning and semantic understanding.<n>We propose CoRAL, a modular framework that enables zero-shot planning by decoupling high-level reasoning from low-level control.
• Score: 4.600772894287281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate remarkable capabilities in high-level reasoning and semantic understanding, applying them directly to contact-rich manipulation remains a challenge due to their lack of explicit physical grounding and inability to perform adaptive control. To bridge this gap, we propose CoRAL (Contact-Rich Adaptive LLM-based control), a modular framework that enables zero-shot planning by decoupling high-level reasoning from low-level control. Unlike black-box policies, CoRAL uses LLMs not as direct controllers, but as cost designers that synthesize context-aware objective functions for a sampling-based motion planner (MPPI). To address the ambiguity of physical parameters in visual data, we introduce a neuro-symbolic adaptation loop: a VLM provides semantic priors for environmental dynamics, such as mass and friction estimates, which are then explicitly refined in real time via online system identification, while the LLM iteratively modulates the cost-function structure to correct strategic errors based on interaction feedback. Furthermore, a retrieval-based memory unit allows the system to reuse successful strategies across recurrent tasks. This hierarchical architecture ensures real-time control stability by decoupling high-level semantic reasoning from reactive execution, effectively bridging the gap between slow LLM inference and dynamic contact requirements. We validate CoRAL on both simulation and real-world hardware across challenging and novel tasks, such as flipping objects against walls by leveraging extrinsic contacts. Experiments demonstrate that CoRAL outperforms state-of-the-art VLA and foundation-model-based planner baselines by boosting success rates over 50% on average in unseen contact-rich scenarios, effectively handling sim-to-real gaps through its adaptive physical understanding.

Related papers

• LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning [6.69561560301489]
  We propose a hierarchical framework for coordinating multiple interacting units in complex engineering systems.<n>We introduce Continuation-Aware GRPO to capture the consequences of coordination decisions over subsequent control intervals.<n>We validate the framework on multi-ramp traffic control and virtual power plant (VPP) energy management, using simplified system models for training and more realistic simulators for evaluation.
  arXiv  Detail & Related papers  (2026-08-15T05:00:02Z)
• Safe and Adaptive Cloud Healing: Verifying LLM-Generated Recovery Plans with a Neural-Symbolic World Model [9.794020127306473]
  Planning-Aware Semantic self-healing engine, PASE, is a novel fault self-healing framework.<n>Our framework advances autonomous system management by unifying LLM-based reasoning with model-assisted verification and meta-learned guidance.<n> Experiments on a real-world cloud fault injection dataset demonstrate that PASE significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-02T01:45:30Z)
• LaST-R1: Reinforcing Action via Adaptive Physical Latent Reasoning for VLA Models [112.82269746694004]
  We present textbfLaST-R1, a unified VLA framework that integrates latent Chain-of-Thought (CoT) reasoning over physical dynamics prior to action execution.<n>LAPO improves the representation of physical world modeling and enhances robustness in interactive environments.<n>LaST-R1 achieves a near-perfect 99.8% average success rate on the LIBERO benchmark with only one-shot supervised warm-up.
  arXiv  Detail & Related papers  (2026-04-30T17:59:52Z)
• AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models [60.04879435087352]
  Vision-language-action (VLA) policies generate actions within a single unified space.<n>We propose AnchorRefine, a hierarchical framework that factorizes VLA action modeling into trajectory anchor and residual refinement.<n> Experiments on LIBERO, CALVIN, and real-robot tasks demonstrate that AnchorRefine consistently improves both regression-based and diffusion-based VLA backbones.
  arXiv  Detail & Related papers  (2026-04-20T04:25:24Z)
• RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering [62.63376387138257]
  We propose a plug-and-play intervention framework that adaptively steers large language models (LLMs) reasoning in activation space.<n>RISER constructs a library of reusable reasoning vectors and employs a lightweight Router to dynamically compose them for each input.<n>The Router is optimized via reinforcement learning under task-level rewards, activating latent cognitive primitives in an emergent and compositional manner.
  arXiv  Detail & Related papers  (2026-01-14T08:04:33Z)
• COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learning [55.83415345423854]
  We propose COVLM-RL, a novel end-to-end driving framework that integrates Critical Object-oriented (CO) reasoning with reinforcement learning (RL)<n>Experiments conducted in the CARLA simulator demonstrate that COVLM-RL significantly improves the success rate by 30% in trained driving environments.
  arXiv  Detail & Related papers  (2025-12-10T06:18:16Z)
• Grounded Test-Time Adaptation for LLM Agents [75.62784644919803]
  Large language model (LLM)-based agents struggle to generalize to novel and complex environments.<n>We propose two strategies for adapting LLM agents by leveraging environment-specific information available during deployment.
  arXiv  Detail & Related papers  (2025-11-06T22:24:35Z)
• LLMs-guided adaptive compensator: Bringing Adaptivity to Automatic Control Systems with Large Language Models [22.989496527440636]
  Large Language Models (LLMs) are increasingly applied in robotics.<n>We propose an LLM-guided adaptive compensator framework that avoids designing controllers from scratch.<n>This study opens a new direction for applying LLMs in the field of automatic control.
  arXiv  Detail & Related papers  (2025-07-28T04:12:43Z)
• RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarms [17.931091064654904]
  We develop a Role-Adaptive LLM-Driven Yoked navigation algorithm RALLY.<n>RALLY uses structured natural language for efficient semantic communication and collaborative reasoning.<n> Experiments show that RALLY outperforms conventional approaches in terms of task coverage, convergence speed, and generalization.
  arXiv  Detail & Related papers  (2025-07-02T05:44:17Z)
• LLMs for sensory-motor control: Combining in-context and iterative learning [0.0]
  We propose a method that enables large language models to control embodied agents by directly mapping continuous observation vectors to continuous action vectors.<n>The method is validated on classic control tasks from the Gymnasium library and the inverted pendulum task from the MuJoCo library.<n>It successfully identifies optimal or near-optimal solutions by integrating symbolic knowledge derived through reasoning with sub-symbolic sensory-motor data gathered as the agent interacts with its environment.
  arXiv  Detail & Related papers  (2025-06-05T10:38:28Z)
• Benchmarking LLMs' Swarm intelligence [51.648605206159125]
  Large Language Models (LLMs) show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems (MAS) remains largely unexplored.<n>We introduce SwarmBench, a novel benchmark designed to systematically evaluate tasks of LLMs acting as decentralized agents.<n>We propose metrics for coordination effectiveness and analyze emergent group dynamics.
  arXiv  Detail & Related papers  (2025-05-07T12:32:01Z)
• Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning [69.5875073447454]
  This paper advances motion agents empowered by large language models (LLMs) toward autonomous navigation in dynamic and cluttered environments.<n>Our training-free framework supports multi-agent coordination, closed-loop replanning, and dynamic obstacle avoidance without retraining or fine-tuning.
  arXiv  Detail & Related papers  (2025-03-10T13:39:09Z)
• Foundation Models to the Rescue: Deadlock Resolution in Connected Multi-Robot Systems [11.012092202226855]
  Connected multi-agent robotic systems (MRS) are prone to deadlocks in an obstacle environment. This paper explores the possibility of using text-based models, i.e., large language models (LLMs), and text-and-image-based models (VLMs), as high-level planners for deadlock resolution. We propose a hierarchical control framework where a foundation model-based high-level planner helps to resolve deadlocks by assigning a leader to the MRS along with a set of waypoints for the MRS leader.
  arXiv  Detail & Related papers  (2024-04-09T16:03:26Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.