Bootstrapping Human-Like Planning via LLMs

• URL: http://arxiv.org/abs/2506.22604v1
• Date: Fri, 27 Jun 2025 20:00:51 GMT
• Title: Bootstrapping Human-Like Planning via LLMs
• Abstract summary: Two common end-user programming paradigms include drag-and-drop interfaces and natural language programming.<n>We construct a large language model (LLM)-based pipeline that accepts natural language as input and produces human-like action sequences as output.<n>Although our results reveal that larger models tend to outperform smaller ones in the production of human-like action sequences, smaller models nonetheless achieve satisfactory performance.
• Score: 0.873811641236639
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robot end users increasingly require accessible means of specifying tasks for robots to perform. Two common end-user programming paradigms include drag-and-drop interfaces and natural language programming. Although natural language interfaces harness an intuitive form of human communication, drag-and-drop interfaces enable users to meticulously and precisely dictate the key actions of the robot's task. In this paper, we investigate the degree to which both approaches can be combined. Specifically, we construct a large language model (LLM)-based pipeline that accepts natural language as input and produces human-like action sequences as output, specified at a level of granularity that a human would produce. We then compare these generated action sequences to another dataset of hand-specified action sequences. Although our results reveal that larger models tend to outperform smaller ones in the production of human-like action sequences, smaller models nonetheless achieve satisfactory performance.

Related papers

• LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music [1.7188280334580195]
  This paper introduces a novel framework that synthesizes complex robotic actions from a rich tapestry of multimodal human inputs.<n>Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection.<n>The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music.
  arXiv  Detail & Related papers  (2026-06-30T05:35:26Z)
• Distill: Uncovering the True Intent behind Human-Robot Communication [3.8732212038055827]
  We propose the Distill approach for human-robot communication interfaces.<n>Given a task specification, Distill removes unnecessary steps; generalizes the meaning behind individual steps; and relaxes ordering constraints between steps.<n>We implemented Distill on a web interface and, through a crowdsourcing study, demonstrated its ability to elicit and refine user intent from initial task specifications.
  arXiv  Detail & Related papers  (2026-05-14T02:05:49Z)
• Large Video Planner Enables Generalizable Robot Control [117.49024534548319]
  General-purpose robots require decision-making models that generalize across diverse tasks and environments.<n>Recent works build robot foundation models by extending multimodal large language models (LMs) with action outputs, creating vision--action (VLA) systems.<n>We explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models.
  arXiv  Detail & Related papers  (2025-12-17T18:35:54Z)
• Interpretable Robot Control via Structured Behavior Trees and Large Language Models [0.14990005092937678]
  This paper presents a novel framework that bridges natural language understanding and robotic execution.<n>The proposed approach is practical in real-world scenarios, with an average cognition-to-execution accuracy of approximately 94%.
  arXiv  Detail & Related papers  (2025-08-13T08:53:13Z)
• Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models [49.4824734958566]
  Chain-of-Modality (CoM) enables Vision Language Models to reason about multimodal human demonstration data.<n>CoM refines a task plan and generates detailed control parameters, enabling robots to perform manipulation tasks based on a single multimodal human video prompt.
  arXiv  Detail & Related papers  (2025-04-17T21:31:23Z)
• LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application [4.519544934630495]
  Large Language Models (LLM) and Vision Language Models (VLM) enable robots to ground natural language prompts into control actions. We propose Language-driven Intention Tracking (LIT) to model the human user's long-term behavior and to predict the next human intention to guide the robot for proactive collaboration.
  arXiv  Detail & Related papers  (2024-06-19T19:18:40Z)
• Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans [58.27029676638521]
  We show how passive human videos can serve as a rich source of data for learning such generalist robots. We learn a human plan predictor that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects.
  arXiv  Detail & Related papers  (2023-12-01T18:54:12Z)
• Proactive Human-Robot Interaction using Visuo-Lingual Transformers [0.0]
  Humans possess the innate ability to extract latent visuo-lingual cues to infer context through human interaction. We propose a learning-based method that uses visual cues from the scene, lingual commands from a user and knowledge of prior object-object interaction to identify and proactively predict the underlying goal the user intends to achieve.
  arXiv  Detail & Related papers  (2023-10-04T00:50:21Z)
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models [23.945922720555146]
  We propose a system to achieve incremental learning of complex behavior from natural interaction. We integrate the system in the robot cognitive architecture of the humanoid robot ARMAR-6.
  arXiv  Detail & Related papers  (2023-09-08T13:29:05Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)
• Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers [33.7939079214046]
  We provide a flexible language-based interface for human-robot collaboration. We take advantage of recent advancements in the field of large language models to encode the user command. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands.
  arXiv  Detail & Related papers  (2022-03-25T01:36:56Z)
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation [80.29069988090912]
  We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. We propose to leverage offline robot datasets with crowd-sourced natural language labels. We find that our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%.
  arXiv  Detail & Related papers  (2021-09-02T17:42:13Z)
• Caption Generation of Robot Behaviors based on Unsupervised Learning of Action Segments [10.356412004005767]
  Bridging robot action sequences and their natural language captions is an important task to increase explainability of human assisting robots. In this paper, we propose a system for generating natural language captions that describe behaviors of human assisting robots.
  arXiv  Detail & Related papers  (2020-03-23T03:44:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.