Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination

• URL: http://arxiv.org/abs/2602.20517v1
• Date: Tue, 24 Feb 2026 03:37:42 GMT
• Title: Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination
• Authors: Rakshit Trivedi, Kartik Sharma, David C Parkes,
• Abstract summary: Imitation learning has emerged as one of the prominent approaches to build such agents.<n>We propose MIMIC, a framework that uses language as an internal representation of behavioral intent.<n>MIMIC enables fine-grained steering of behavior at inference time by conditioning the agent on behavior-specific speech.
• Score: 21.33828337975933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective human-AI coordination requires artificial agents capable of exhibiting and responding to human-like behaviors while adapting to changing contexts. Imitation learning has emerged as one of the prominent approaches to build such agents by training them to mimic human-demonstrated behaviors. However, current methods struggle to capture the inherent diversity and non-Markovian nature of human behavior and lack the ability to steer behavior at inference time. Drawing inspiration from the theory of human cognitive processes, where inner speech guides action selection before execution, we propose MIMIC (Modeling Inner Motivations for Imitation and Control), a framework that uses language as an internal representation of behavioral intent. MIMIC employs the novel use of vision-language models as linguistic scaffolding to train a conditional variational autoencoder capable of generating inner speech from observations. A diffusion-based behavior cloning policy then selects actions conditioned on current observations and the generated inner speech. MIMIC enables fine-grained steering of behavior at inference time by conditioning the agent on behavior-specific speech. Experiments across robotic manipulation tasks and human-AI collaboration games demonstrate that MIMIC significantly enhances both behavior diversity and fidelity to human demonstrations while enabling nuanced behavioral steering without training on additional demonstrations. We open source our code and provide pre-trained MIMIC agents and qualitative demos at: https://mimic-research.github.io.

Related papers

• HUMANLLM: Benchmarking and Reinforcing LLM Anthropomorphism via Human Cognitive Patterns [59.17423586203706]
  We present HUMANLLM, a framework treating psychological patterns as interacting causal forces.<n>We construct 244 patterns from 12,000 academic papers and synthesize 11,359 scenarios where 2-5 patterns reinforce, conflict, or modulate each other.<n>Our dual-level checklists evaluate both individual pattern fidelity and emergent multi-pattern dynamics, achieving strong human alignment.
  arXiv  Detail & Related papers  (2026-01-15T08:56:53Z)
• Continuous Vision-Language-Action Co-Learning with Semantic-Physical Alignment for Behavioral Cloning [22.14625208769185]
  We present Continuous vision-language-action Co-Learning with Semantic-Physical Alignment (CCoL), a novel BC framework that ensures temporally consistent execution and fine-grained semantic grounding.<n>CCoL achieves an average 8.0% relative improvement across three simulation suites, with up to 19.2% relative gain in human-demonstrated bimanual insertion tasks.
  arXiv  Detail & Related papers  (2025-11-18T12:01:06Z)
• Behavioral Exploration: Learning to Explore via In-Context Adaptation [53.92981562916783]
  We train a long-context generative model to predict expert actions conditioned on a context of past observations and a measure of how exploratory'' the expert's behaviors are relative to this context.<n>This enables the model to not only mimic the behavior of an expert, but also, by feeding its past history of interactions into its context, to select different expert behaviors than what have been previously selected.<n>We demonstrate the effectiveness of our method in both simulated locomotion and manipulation settings, as well as on real-world robotic manipulation tasks.
  arXiv  Detail & Related papers  (2025-07-11T21:36:19Z)
• Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind [51.45478233267092]
  We present a cognitively inspired neurosymbolic model, Spoken Instruction Following through Theory of Mind (SIFToM)<n>SIFToM uses a Vision-Language Model with model-based mental inference to enable robots to pragmatically follow human instructions under diverse speech conditions.<n>Results show that SIFToM can significantly improve the performance of a lightweight base VLM (Gemini 2.5 Flash), outperforming state-of-the-art VLMs (Gemini 2.5 Pro) and approaching human-level accuracy on challenging spoken instruction following tasks.
  arXiv  Detail & Related papers  (2024-09-17T02:36:10Z)
• PersLLM: A Personified Training Approach for Large Language Models [66.16513246245401]
  We propose PersLLM, a framework for better data construction and model tuning.<n>For insufficient data usage, we incorporate strategies such as Chain-of-Thought prompting and anti-induction.<n>For rigid behavior patterns, we design the tuning process and introduce automated DPO to enhance the specificity and dynamism of the models' personalities.
  arXiv  Detail & Related papers  (2024-07-17T08:13:22Z)
• Dyadic Interaction Modeling for Social Behavior Generation [6.626277726145613]
  We present an effective framework for creating 3D facial motions in dyadic interactions. The heart of our framework is Dyadic Interaction Modeling (DIM), a pre-training approach. Experiments demonstrate the superiority of our framework in generating listener motions.
  arXiv  Detail & Related papers  (2024-03-14T03:21:33Z)
• Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences [53.353022588751585]
  We present Promptable Behaviors, a novel framework that facilitates efficient personalization of robotic agents to diverse human preferences. We introduce three distinct methods to infer human preferences by leveraging different types of interactions. We evaluate the proposed method in personalized object-goal navigation and flee navigation tasks in ProcTHOR and RoboTHOR.
  arXiv  Detail & Related papers  (2023-12-14T21:00:56Z)
• ThinkBot: Embodied Instruction Following with Thought Chain Reasoning [66.09880459084901]
  Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. We propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions. Our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.
  arXiv  Detail & Related papers  (2023-12-12T08:30:09Z)
• Persistent-Transient Duality: A Multi-mechanism Approach for Modeling Human-Object Interaction [58.67761673662716]
  Humans are highly adaptable, swiftly switching between different modes to handle different tasks, situations and contexts. In Human-object interaction (HOI) activities, these modes can be attributed to two mechanisms: (1) the large-scale consistent plan for the whole activity and (2) the small-scale children interactive actions that start and end along the timeline. This work proposes to model two concurrent mechanisms that jointly control human motion.
  arXiv  Detail & Related papers  (2023-07-24T12:21:33Z)
• AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for Adapted Behavior Synthesis [6.021787236982659]
  Socially Interactive Agents (SIAs) are physical or virtual embodied agents that display similar behavior as human multimodal behavior. We propose AMII, a novel approach to synthesize adaptive facial gestures for SIAs while interacting with Users and acting as a speaker or as a listener.
  arXiv  Detail & Related papers  (2023-05-18T21:22:07Z)
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks [39.40937105264774]
  We introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent. The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions.
  arXiv  Detail & Related papers  (2020-10-22T21:49:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.