LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation

• URL: http://arxiv.org/abs/2501.08558v1
• Date: Wed, 15 Jan 2025 03:49:08 GMT
• Title: LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation
• Authors: Yiran Tao, Jehan Yang, Dan Ding, Zackory Erickson,
• Abstract summary: We introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time.
• Score: 4.22823627787465
• License:
• Abstract: Teleoperating high degrees-of-freedom (DoF) robotic manipulators via low-DoF controllers like joysticks often requires frequent switching between control modes, where each mode maps controller movements to specific robot actions. Manually performing this frequent switching can make teleoperation cumbersome and inefficient. On the other hand, existing automatic mode-switching solutions, such as heuristic-based or learning-based methods, are often task-specific and lack generalizability. In this paper, we introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach that leverages Large Language Models (LLMs) to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time. The project website with supplementary materials is at https://lams-assistance.github.io/.

Related papers

• LLM-AutoDiff: Auto-Differentiate Any LLM Workflow [58.56731133392544]
  We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE) LLMs-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine to generate feedback-akin to textual gradients. It consistently outperforms existing textual gradient baselines in both accuracy and training cost.
  arXiv  Detail & Related papers  (2025-01-28T03:18:48Z)
• MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation [52.739500459903724]
  Large Language Models (LLMs) have demonstrated remarkable planning abilities across various domains, including robotics manipulation and navigation. We propose a novel multi-agent LLM framework that distributes high-level planning and low-level control code generation across specialized LLM agents. We evaluate our approach on nine RLBench tasks, including long-horizon tasks, and demonstrate its ability to solve robotics manipulation in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-11-26T17:53:44Z)
• Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning [61.294110816231886]
  We introduce a sparse, reusable, and flexible policy, Sparse Diffusion Policy (SDP) SDP selectively activates experts and skills, enabling efficient and task-specific learning without retraining the entire model. Demos and codes can be found in https://forrest-110.io/sparse_diffusion_policy/.
  arXiv  Detail & Related papers  (2024-07-01T17:59:56Z)
• MOSEAC: Streamlined Variable Time Step Reinforcement Learning [14.838483990647697]
  We introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method. MOSEAC features an adaptive reward scheme based on observed trends in task rewards during training. We validate the MOSEAC method through simulations in a Newtonian kinematics environment.
  arXiv  Detail & Related papers  (2024-06-03T16:51:57Z)
• In Time and Space: Towards Usable Adaptive Control for Assistive Robotic Arms [5.988522447941765]
  Robotic arms require the user to control several Degrees-of-Freedom (DoFs) to perform tasks. Modern Adaptive DoF Mapping Controls (ADMCs) have shown to decrease the necessary number of mode switches but were up to now not able to significantly reduce the perceived workload. We address this by providing feed-forward multimodal feedback using updated recommendations of ADMC.
  arXiv  Detail & Related papers  (2023-07-06T11:51:43Z)
• Disturbance Injection under Partial Automation: Robust Imitation Learning for Long-horizon Tasks [11.554935619056819]
  Partial Automation (PA) with intelligent support systems has been introduced in industrial machinery and advanced automobiles. This paper proposes Disturbance Injection under Partial Automation (DIPA) as a novel imitation learning framework. We experimentally validated the effectiveness of our method for long-horizon tasks in two simulations and a real robot environment.
  arXiv  Detail & Related papers  (2023-03-22T08:22:12Z)
• M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design [95.41238363769892]
  Multi-task learning (MTL) encapsulates multiple learned tasks in a single model and often lets those tasks learn better jointly. Current MTL regimes have to activate nearly the entire model even to just execute a single task. We present a model-accelerator co-design framework to enable efficient on-device MTL.
  arXiv  Detail & Related papers  (2022-10-26T15:40:24Z)
• A Meta-Reinforcement Learning Approach to Process Control [3.9146761527401424]
  Meta-learning aims to quickly adapt models, such as neural networks, to perform new tasks. We construct a controller and meta-train the controller using a latent context variable through a separate embedding neural network. In both cases, our meta-learning algorithm adapts very quickly to new tasks, outperforming a regular DRL controller trained from scratch.
  arXiv  Detail & Related papers  (2021-03-25T18:20:56Z)
• Controllable Pareto Multi-Task Learning [55.945680594691076]
  A multi-task learning system aims at solving multiple related tasks at the same time. With a fixed model capacity, the tasks would be conflicted with each other, and the system usually has to make a trade-off among learning all of them together. This work proposes a novel controllable multi-task learning framework, to enable the system to make real-time trade-off control among different tasks with a single model.
  arXiv  Detail & Related papers  (2020-10-13T11:53:55Z)
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation [99.2543521972137]
  ReLMoGen is a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. ReLMoGen shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.
  arXiv  Detail & Related papers  (2020-08-18T08:05:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.