Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming

• URL: http://arxiv.org/abs/2407.20712v1
• Date: Tue, 30 Jul 2024 10:13:00 GMT
• Title: Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming
• Authors: Yate Ge, Yi Dai, Run Shan, Kechun Li, Yuanda Hu, Xiaohua Sun,
• Abstract summary: We introduce Cocobo, a natural language programming system with interactive diagrams powered by large language models (LLMss) Cocobo employs LLMs to understand users' authoring intentions, generate and explain robot programs, and facilitate the conversion between executable code and flowchart representations. Our user study shows that Cocobo has a low learning curve, enabling even users with zero coding experience to customize robot programs successfully.
• Score: 3.041618201510648
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-user development allows everyday users to tailor service robots or applications to their needs. One user-friendly approach is natural language programming. However, it encounters challenges such as an expansive user expression space and limited support for debugging and editing, which restrict its application in end-user programming. The emergence of large language models (LLMs) offers promising avenues for the translation and interpretation between human language instructions and the code executed by robots, but their application in end-user programming systems requires further study. We introduce Cocobo, a natural language programming system with interactive diagrams powered by LLMs. Cocobo employs LLMs to understand users' authoring intentions, generate and explain robot programs, and facilitate the conversion between executable code and flowchart representations. Our user study shows that Cocobo has a low learning curve, enabling even users with zero coding experience to customize robot programs successfully.

Related papers

• Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming [18.256529559741075]
  Large Language Models (LLMs) are designed to do in-context learning for conversational code generation. We evaluate the capabilities of state-of-the-art LLMs for synthesising this kind of code, given in-context examples.
  arXiv  Detail & Related papers  (2024-09-17T10:04:50Z)
• Dynamic Code Orchestration: Harnessing the Power of Large Language Models for Adaptive Script Execution [0.5735035463793009]
  Research examines dynamic code execution of written language directives within the context of a running application. The research clearly shows how written language directives, backed by a large language model, offer radically new programming and operating system paradigms.
  arXiv  Detail & Related papers  (2024-08-07T17:11:31Z)
• AIOS Compiler: LLM as Interpreter for Natural Language Programming and Flow Programming of AI Agents [38.580779075892636]
  We develop a novel system for Code Representation and Execution (CoRE) The proposed system unifies natural language programming, pseudo-code programming, and flow programming under the same representation for constructing language agents. During the execution, we incorporate external memory to minimize redundancy.
  arXiv  Detail & Related papers  (2024-05-11T04:29:03Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  We propose CodeGRAG, a Graphical Retrieval Augmented Code Generation framework to enhance the performance of LLMs. CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to fill the gap between programming languages and natural language. Various experiments and ablations are done on four datasets including both the C++ and python languages to validate the hard meta-graph prompt, the soft prompting technique, and the effectiveness of the objectives for pretrained GNN expert.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• PwR: Exploring the Role of Representations in Conversational Programming [17.838776812138626]
  We introduce Programming with Representations (PwR), an approach that uses representations to convey the system's understanding back to the user in natural language. We find that representations significantly improve understandability, and instilled a sense of agency among our participants.
  arXiv  Detail & Related papers  (2023-09-18T05:38:23Z)
• ChatDev: Communicative Agents for Software Development [84.90400377131962]
  ChatDev is a chat-powered software development framework in which specialized agents are guided in what to communicate. These agents actively contribute to the design, coding, and testing phases through unified language-based communication.
  arXiv  Detail & Related papers  (2023-07-16T02:11:34Z)
• PADL: Language-Directed Physics-Based Character Control [66.517142635815]
  We present PADL, which allows users to issue natural language commands for specifying high-level tasks and low-level skills that a character should perform. We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.
  arXiv  Detail & Related papers  (2023-01-31T18:59:22Z)
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy [70.45420918526926]
  We present LILAC, a framework for incorporating and adapting to natural language corrections online during execution. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot. We show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users.
  arXiv  Detail & Related papers  (2023-01-06T15:03:27Z)
• Correcting Robot Plans with Natural Language Feedback [88.92824527743105]
  We explore natural language as an expressive and flexible tool for robot correction. We show that these transformations enable users to correct goals, update robot motions, and recover from planning errors. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.
  arXiv  Detail & Related papers  (2022-04-11T15:22:43Z)
• Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers [33.7939079214046]
  We provide a flexible language-based interface for human-robot collaboration. We take advantage of recent advancements in the field of large language models to encode the user command. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands.
  arXiv  Detail & Related papers  (2022-03-25T01:36:56Z)
• LILA: Language-Informed Latent Actions [72.033770901278]
  We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA learns to use language to modulate a low-dimensional controller, providing users with a language-informed control space. We show that LILA models are not only more sample efficient and performant than imitation learning and end-effector control baselines, but that they are also qualitatively preferred by users.
  arXiv  Detail & Related papers  (2021-11-05T00:56:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.