HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teams

• URL: http://arxiv.org/abs/2409.18047v3
• Date: Wed, 09 Jul 2025 20:58:34 GMT
• Title: HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teams
• Authors: Sanjay Oruganti, Sergei Nirenburg, Marjorie McShane, Jesse English, Michael K. Roberts, Christian Arndt, Sahithi Kamireddy, Carlos Gonzalez, Mingyo Seo, Luis Sentis,
• Abstract summary: This paper describes a cognitive-robotic architecture that integrates the OntoAgent cognitive framework with general-purpose robot control systems applied to human-robot teaming (HRT)<n>HarMONIC incorporates metacognition, meaningful natural language communication, and explainability capabilities required for developing mutual trust in HRT.
• Score: 2.6627293764668902
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper describes HARMONIC, a cognitive-robotic architecture that integrates the OntoAgent cognitive framework with general-purpose robot control systems applied to human-robot teaming (HRT). HARMONIC incorporates metacognition, meaningful natural language communication, and explainability capabilities required for developing mutual trust in HRT. Through simulation experiments involving a joint search task performed by a heterogeneous team of two HARMONIC-based robots and a human operator, we demonstrate heterogeneous robots that coordinate their actions, adapt to complex scenarios, and engage in natural human-robot communication. Evaluation results show that HARMONIC-based robots can reason about plans, goals, and team member attitudes while providing clear explanations for their decisions, which are essential requirements for realistic human-robot teaming.

Related papers

• Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interaction [4.276453870301421]
  We propose a Transformer-based multi-task learning framework to improve the decision-making process of social robots.<n>We construct a novel multi-party HRI dataset that captures real-world complexities, such as gaze misalignment.<n>Our findings contribute to the development of socially intelligent social robots capable of engaging in natural and context-aware multi-party interactions.
  arXiv  Detail & Related papers  (2025-07-15T03:42:14Z)
• HACTS: a Human-As-Copilot Teleoperation System for Robot Learning [47.9126187195398]
  We introduce HACTS (Human-As-Copilot Teleoperation System), a novel system that establishes bilateral, real-time joint synchronization between a robot arm and teleoperation hardware. This simple yet effective feedback mechanism, akin to a steering wheel in autonomous vehicles, enables the human copilot to intervene seamlessly while collecting action-correction data for future learning.
  arXiv  Detail & Related papers  (2025-03-31T13:28:13Z)
• Human-Artificial Interaction in the Age of Agentic AI: A System-Theoretical Approach [0.0]
  This paper presents a novel perspective on human-computer interaction (HCI), framing it as a dynamic interplay between human and computational agents. A key distinction is made between multi-agent systems (MAS) and Centaurian systems, which represent two different paradigms of human-AI collaboration. Our research has practical applications in autonomous robotics, human-in-the-loop decision making, and AI-driven cognitive architectures.
  arXiv  Detail & Related papers  (2025-02-19T07:55:34Z)
• Redefining Robot Generalization Through Interactive Intelligence [0.0]
  We argue that robot foundation models must evolve to an interactive multi-agent perspective in order to handle the complexities of real-time human-robot co-adaptation. By moving beyond single-agent designs, our position emphasizes how foundation models in robotics can achieve a more robust, personalized, and anticipatory level of performance.
  arXiv  Detail & Related papers  (2025-02-09T17:13:27Z)
• HARMONIC: A Framework for Explanatory Cognitive Robots [0.0]
  We present HARMONIC, a framework for implementing cognitive robots. The framework supports interoperability between a strategic (cognitive) layer for high-level decision-making and a tactical (robot) layer for low-level control and execution.
  arXiv  Detail & Related papers  (2024-09-26T16:42:13Z)
• Towards Human-Centered Construction Robotics: A Reinforcement Learning-Driven Companion Robot for Contextually Assisting Carpentry Workers [11.843554918145983]
  This paper introduces a human-centered approach with a "work companion rover" designed to assist construction workers within their existing practices. We conduct an in-depth study on deploying a robotic system in carpentry formwork, showcasing a prototype that emphasizes mobility, safety, and comfortable worker-robot collaboration.
  arXiv  Detail & Related papers  (2024-03-27T23:55:02Z)
• Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots [119.55240471433302]
  Habitat 3.0 is a simulation platform for studying collaborative human-robot tasks in home environments. It addresses challenges in modeling complex deformable bodies and diversity in appearance and motion. Human-in-the-loop infrastructure enables real human interaction with simulated robots via mouse/keyboard or a VR interface.
  arXiv  Detail & Related papers  (2023-10-19T17:29:17Z)
• HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration [57.045140028275036]
  We show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning. We propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.
  arXiv  Detail & Related papers  (2022-12-08T15:56:13Z)
• Generalizable Human-Robot Collaborative Assembly Using Imitation Learning and Force Control [17.270360447188196]
  We present a system for human-robot collaborative assembly using learning from demonstration and pose estimation. The proposed system is demonstrated using a physical 6 DoF manipulator in a collaborative human-robot assembly scenario.
  arXiv  Detail & Related papers  (2022-12-02T20:35:55Z)
• CoGrasp: 6-DoF Grasp Generation for Human-Robot Collaboration [0.0]
  We propose a novel, deep neural network-based method called CoGrasp that generates human-aware robot grasps. In real robot experiments, our method achieves about 88% success rate in producing stable grasps. Our approach enables a safe, natural, and socially-aware human-robot objects' co-grasping experience.
  arXiv  Detail & Related papers  (2022-10-06T19:23:25Z)
• Data-driven emotional body language generation for social robotics [58.88028813371423]
  In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration. We implement a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions. The evaluation study found that the anthropomorphism and animacy of the generated expressions are not perceived differently from the hand-designed ones.
  arXiv  Detail & Related papers  (2022-05-02T09:21:39Z)
• Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together [68.44697646919515]
  This paper presents several human-robot systems that utilize spatial computing to enable novel robot use cases. The combination of spatial computing and egocentric sensing on mixed reality devices enables them to capture and understand human actions and translate these to actions with spatial meaning.
  arXiv  Detail & Related papers  (2022-02-03T10:04:26Z)
• Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration [83.4081612443128]
  We show that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground-truth. We show that this calibration procedure not only results in better user perception, but also promotes more efficient human-robot collaborations.
  arXiv  Detail & Related papers  (2021-03-06T09:14:30Z)
• Controlling the Sense of Agency in Dyadic Robot Interaction: An Active Inference Approach [6.421670116083633]
  We examine dyadic imitative interactions of robots using a variational recurrent neural network model. We examined how regulating the complexity term to minimize free energy during training determines the dynamic characteristics of networks.
  arXiv  Detail & Related papers  (2021-03-03T02:38:09Z)
• Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks [83.37025218216888]
  We propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations. The robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications. Results show that the generated explanations of our approach significantly improves the collaboration performance and user perception of the robot.
  arXiv  Detail & Related papers  (2020-07-24T23:35:03Z)
• SAPIEN: A SimulAted Part-based Interactive ENvironment [77.4739790629284]
  SAPIEN is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks.
  arXiv  Detail & Related papers  (2020-03-19T00:11:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.