Bi-LAT: Bilateral Control-Based Imitation Learning via Natural Language and Action Chunking with Transformers

• URL: http://arxiv.org/abs/2504.01301v2
• Date: Mon, 28 Jul 2025 01:12:45 GMT
• Title: Bi-LAT: Bilateral Control-Based Imitation Learning via Natural Language and Action Chunking with Transformers
• Abstract summary: Bi-LAT is a novel imitation learning framework that unifies bilateral control with natural language processing.<n>Bi-LAT learns to distinguish nuanced force requirements in real-world tasks.
• Score: 16.724380665811044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Bi-LAT, a novel imitation learning framework that unifies bilateral control with natural language processing to achieve precise force modulation in robotic manipulation. Bi-LAT leverages joint position, velocity, and torque data from leader-follower teleoperation while also integrating visual and linguistic cues to dynamically adjust applied force. By encoding human instructions such as "softly grasp the cup" or "strongly twist the sponge" through a multimodal Transformer-based model, Bi-LAT learns to distinguish nuanced force requirements in real-world tasks. We demonstrate Bi-LAT's performance in (1) unimanual cup-stacking scenario where the robot accurately modulates grasp force based on language commands, and (2) bimanual sponge-twisting task that requires coordinated force control. Experimental results show that Bi-LAT effectively reproduces the instructed force levels, particularly when incorporating SigLIP among tested language encoders. Our findings demonstrate the potential of integrating natural language cues into imitation learning, paving the way for more intuitive and adaptive human-robot interaction. For additional material, please visit: https://mertcookimg.github.io/bi-lat/

Related papers

• Brain-Language-Action (BLA) Models: Language-Conditioned EEG for Robotics Control [51.56484100374058]
  We introduce Brain-Language-Action models, a framework in which language conditions the interpretation of neural representations for robotic action generation.<n>In a BLA, a small set of reliably distinguishable brain states can be dynamically associated with different actions through a language-defined control mapping.<n>We develop a proof-of-concept BLA for drone control using motor-imagery EEG from the BCI Competition IV 2a dataset.
  arXiv  Detail & Related papers  (2026-08-29T00:37:52Z)
• CLAW: Composable Language-Annotated Whole-body Motion Generation [55.99805728566105]
  We present CLAW, a pipeline for scalable generation of language-annotated whole-body motion data.<n>CLAW composes motion primitives from a kinematic planner, parameterized by movement, heading, speed, pelvis height, and duration.<n>A low-level controller tracks these references in MuJoCo simulation, yielding physically grounded trajectories.
  arXiv  Detail & Related papers  (2026-04-13T10:02:04Z)
• KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition [89.7916653471162]
  We introduce a novel kinematics-rich vision-language-action (VLA) task, in which language commands densely encode diverse kinematic attributes.<n>In this setting, task goals remain invariant while execution trajectories must adapt to instruction-level kinematic specifications.<n>We show that KineVLA consistently outperforms strong VLA baselines on kinematics-sensitive benchmarks.
  arXiv  Detail & Related papers  (2026-03-18T09:28:49Z)
• MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers [3.890941830250993]
  We propose a lightweight multi-task imitation learning framework for bimanual manipulation.<n>MoE-ACT integrates sparse Mixture-of-Experts (MoE) modules into the Transformer encoder of ACT.<n>We show that MoE-ACT substantially improves multi-task performance.
  arXiv  Detail & Related papers  (2026-03-16T13:33:59Z)
• LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation [11.419077130835829]
  We introduce LACY (Language-Action Cycle), a unified framework that learns bidirectional mappings within a single vision-language model.<n>We show that LACY improves task success rates by 56.46% on average and yields more robust language-action grounding for robotic manipulation.
  arXiv  Detail & Related papers  (2025-11-04T04:02:51Z)
• Bi-VLA: Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation [0.3867363075280544]
  We propose Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation (Bi-VLA)<n>Bi-VLA extends bilateral control-based imitation learning to handle more than one task within a single model.<n>We show Bi-VLA successfully interprets vision-language combinations and improves task success rates.
  arXiv  Detail & Related papers  (2025-09-23T10:02:16Z)
• Compositional Coordination for Multi-Robot Teams with Large Language Models [22.35748083111824]
  LAN2CB transforms natural language (NL) mission descriptions into Python code for multi-robot systems.<n>Experiments in simulation and real-world environments demonstrate that LAN2CB enables robust and flexible multi-robot coordination from natural language.
  arXiv  Detail & Related papers  (2025-07-21T21:09:15Z)
• STEER: Flexible Robotic Manipulation via Dense Language Grounding [16.97343810491996]
  STEER is a robot learning framework that bridges high-level, commonsense reasoning with precise, flexible low-level control. Our approach translates complex situational awareness into actionable low-level behavior through training language-grounded policies with dense annotation.
  arXiv  Detail & Related papers  (2024-11-05T18:48:12Z)
• Measuring and Controlling Instruction (In)Stability in Language Model Dialogs [72.38330196290119]
  System-prompting is a tool for customizing language-model chatbots, enabling them to follow a specific instruction. We propose a benchmark to test the assumption, evaluating instruction stability via self-chats. We reveal a significant instruction drift within eight rounds of conversations. We propose a lightweight method called split-softmax, which compares favorably against two strong baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:10:29Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer [77.28871523946418]
  BatGPT is a large-scale language model designed and trained jointly by Wuhan University and Shanghai Jiao Tong University. It is capable of generating highly natural and fluent text in response to various types of input, including text prompts, images, and audio.
  arXiv  Detail & Related papers  (2023-07-01T15:10:01Z)
• Surfer: Progressive Reasoning with World Models for Robotic Manipulation [51.26109827779267]
  We introduce a novel and simple robot manipulation framework, called Surfer. Surfer treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene. It is based on the world model, treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene.
  arXiv  Detail & Related papers  (2023-06-20T07:06:04Z)
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy [70.45420918526926]
  We present LILAC, a framework for incorporating and adapting to natural language corrections online during execution. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot. We show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users.
  arXiv  Detail & Related papers  (2023-01-06T15:03:27Z)
• LaTTe: Language Trajectory TransformEr [33.7939079214046]
  This work proposes a flexible language-based framework to modify generic 3D robotic trajectories. We employ an auto-regressive transformer to map natural language inputs and contextual images into changes in 3D trajectories. We show through simulations and real-life experiments that the model can successfully follow human intent.
  arXiv  Detail & Related papers  (2022-08-04T22:43:21Z)
• Learning Flexible Translation between Robot Actions and Language Descriptions [16.538887534958555]
  We propose a paired gated autoencoders (PGAE) for flexible translation between robot actions and language descriptions. We train our model in an end-to-end fashion by pairing each action with appropriate descriptions that contain a signal informing about the translation direction. With the option to use a pretrained language model as the language encoder, our model has the potential to recognise unseen natural language input.
  arXiv  Detail & Related papers  (2022-07-15T12:37:05Z)
• Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers [33.7939079214046]
  We provide a flexible language-based interface for human-robot collaboration. We take advantage of recent advancements in the field of large language models to encode the user command. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands.
  arXiv  Detail & Related papers  (2022-03-25T01:36:56Z)
• Language Model-Based Paired Variational Autoencoders for Robotic Language Learning [18.851256771007748]
  Similar to human infants, artificial agents can learn language while interacting with their environment. We present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model.
  arXiv  Detail & Related papers  (2022-01-17T10:05:26Z)
• LILA: Language-Informed Latent Actions [72.033770901278]
  We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA learns to use language to modulate a low-dimensional controller, providing users with a language-informed control space. We show that LILA models are not only more sample efficient and performant than imitation learning and end-effector control baselines, but that they are also qualitatively preferred by users.
  arXiv  Detail & Related papers  (2021-11-05T00:56:00Z)
• Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space [109.79957125584252]
  Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model, Optimus.
  arXiv  Detail & Related papers  (2020-04-05T06:20:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.