Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming

• URL: http://arxiv.org/abs/2312.13905v2
• Date: Sun, 21 Apr 2024 09:24:39 GMT
• Title: Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming
• Authors: Benjamin Alt, Urs Keßner, Aleksandar Taranovic, Darko Katic, Andreas Hermann, Rainer Jäkel, Gerhard Neumann,
• Abstract summary: We propose a natural language-based assistant for programming of advanced, industrial robotic applications. We investigate strategies for domain-specific fine-tuning of foundation models with limited data and compute.
• Score: 47.60440351442233
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Industrial robots are applied in a widening range of industries, but robot programming mostly remains a task limited to programming experts. We propose a natural language-based assistant for programming of advanced, industrial robotic applications and investigate strategies for domain-specific fine-tuning of foundation models with limited data and compute.

Related papers

• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming [18.256529559741075]
  Large Language Models (LLMs) are designed to do in-context learning for conversational code generation. We evaluate the capabilities of state-of-the-art LLMs for synthesising this kind of code, given in-context examples.
  arXiv  Detail & Related papers  (2024-09-17T10:04:50Z)
• KiloBot: A Programming Language for Deploying Perception-Guided Industrial Manipulators at Scale [6.804432396982314]
  We would like industrial robots to handle unstructured environments with cameras and perception pipelines. Online behavior planning is required for these perception-guided industrial applications. Our DSL is mainly used by machine operators without coding experience in traditional programming languages.
  arXiv  Detail & Related papers  (2024-09-05T11:42:08Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis [82.59451639072073]
  General-purpose robots operate seamlessly in any environment, with any object, and utilize various skills to complete diverse tasks. As a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models, we devote this survey to exploring how foundation models can be applied to general-purpose robotics.
  arXiv  Detail & Related papers  (2023-12-14T10:02:55Z)
• Logic programming for deliberative robotic task planning [2.610470075814367]
  We present a survey on recent advances in the application of logic programming to the problem of task planning. We analyze different planners and their suitability for specific robotic applications, based on expressivity in domain representation, computational efficiency and software implementation.
  arXiv  Detail & Related papers  (2023-01-18T14:11:55Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• A No-Code Low-Code Paradigm for Authoring Business Automations Using Natural Language [9.03354980024123]
  We propose a paradigm for the construction of business automations using natural language. The approach applies a large language model to translate business rules and automations described in natural language.
  arXiv  Detail & Related papers  (2022-07-15T19:17:55Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.