Building Knowledge from Interactions: An LLM-Based Architecture for Adaptive Tutoring and Social Reasoning

• URL: http://arxiv.org/abs/2504.01588v1
• Date: Wed, 02 Apr 2025 10:45:41 GMT
• Title: Building Knowledge from Interactions: An LLM-Based Architecture for Adaptive Tutoring and Social Reasoning
• Authors: Luca Garello, Giulia Belgiovine, Gabriele Russo, Francesco Rea, Alessandra Sciutti,
• Abstract summary: Large Language Models show promise in human-like communication, but their standalone use is hindered by memory constraints and contextual incoherence.<n>This work presents a multimodal, cognitively inspired framework that enhances LLM-based autonomous decision-making in social and task-oriented Human-Robot Interaction.<n>To further enhance autonomy and personalization, we introduce a memory system for selecting, storing and retrieving experiences.
• Score: 42.09560737219404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Integrating robotics into everyday scenarios like tutoring or physical training requires robots capable of adaptive, socially engaging, and goal-oriented interactions. While Large Language Models show promise in human-like communication, their standalone use is hindered by memory constraints and contextual incoherence. This work presents a multimodal, cognitively inspired framework that enhances LLM-based autonomous decision-making in social and task-oriented Human-Robot Interaction. Specifically, we develop an LLM-based agent for a robot trainer, balancing social conversation with task guidance and goal-driven motivation. To further enhance autonomy and personalization, we introduce a memory system for selecting, storing and retrieving experiences, facilitating generalized reasoning based on knowledge built across different interactions. A preliminary HRI user study and offline experiments with a synthetic dataset validate our approach, demonstrating the system's ability to manage complex interactions, autonomously drive training tasks, and build and retrieve contextual memories, advancing socially intelligent robotics.

Related papers

• Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy [31.818923556912495]
  We introduce a new self-supervised neuro-symbolic (NeSy) computational framework, imperative learning (IL) for robot autonomy. We formulate IL as a special bilevel optimization (BLO) which enables reciprocal learning over the three modules. We show that IL can significantly enhance robot autonomy capabilities and we anticipate that it will catalyze further research across diverse domains.
  arXiv  Detail & Related papers  (2024-06-23T12:02:17Z)
• Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory [3.3906920519220054]
  We describe our approach to developing an intelligent and robust social robotic system for the Nadine platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models. This approach is novel compared to the current state-of-the-art LLM-based agents which do not implement human-like long-term memory or sophisticated emotional appraisal.
  arXiv  Detail & Related papers  (2024-05-30T15:55:41Z)
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models [23.945922720555146]
  We propose a system to achieve incremental learning of complex behavior from natural interaction. We integrate the system in the robot cognitive architecture of the humanoid robot ARMAR-6.
  arXiv  Detail & Related papers  (2023-09-08T13:29:05Z)
• Incremental procedural and sensorimotor learning in cognitive humanoid robots [52.77024349608834]
  This work presents a cognitive agent that can learn procedures incrementally. We show the cognitive functions required in each substage and how adding new functions helps address tasks previously unsolved by the agent. Results show that this approach is capable of solving complex tasks incrementally.
  arXiv  Detail & Related papers  (2023-04-30T22:51:31Z)
• LLM as A Robotic Brain: Unifying Egocentric Memory and Control [77.0899374628474]
  Embodied AI focuses on the study and development of intelligent systems that possess a physical or virtual embodiment (i.e. robots) Memory and control are the two essential parts of an embodied system and usually require separate frameworks to model each of them. We propose a novel framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control.
  arXiv  Detail & Related papers  (2023-04-19T00:08:48Z)
• Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance [71.36749876465618]
  We describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples. experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world.
  arXiv  Detail & Related papers  (2022-12-19T22:50:40Z)
• Autonomous Open-Ended Learning of Tasks with Non-Stationary Interdependencies [64.0476282000118]
  Intrinsic motivations have proven to generate a task-agnostic signal to properly allocate the training time amongst goals. While the majority of works in the field of intrinsically motivated open-ended learning focus on scenarios where goals are independent from each other, only few of them studied the autonomous acquisition of interdependent tasks. In particular, we first deepen the analysis of a previous system, showing the importance of incorporating information about the relationships between tasks at a higher level of the architecture. Then we introduce H-GRAIL, a new system that extends the previous one by adding a new learning layer to store the autonomously acquired sequences
  arXiv  Detail & Related papers  (2022-05-16T10:43:01Z)
• A MultiModal Social Robot Toward Personalized Emotion Interaction [1.2183405753834562]
  This study demonstrates a multimodal human-robot interaction (HRI) framework with reinforcement learning to enhance the robotic interaction policy. The goal is to apply this framework in social scenarios that can let the robots generate a more natural and engaging HRI framework.
  arXiv  Detail & Related papers  (2021-10-08T00:35:44Z)
• Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition [54.749127627191655]
  The ability to recognize human partners is an important social skill to build personalized and long-term human-robot interactions. Deep learning networks have achieved state-of-the-art results and demonstrated to be suitable tools to address such a task. One solution is to make robots learn from their first-hand sensory data with self-supervision.
  arXiv  Detail & Related papers  (2021-03-16T13:50:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.