Cognition to Control - Multi-Agent Learning for Human-Humanoid Collaborative Transport

• URL: http://arxiv.org/abs/2603.03768v1
• Date: Wed, 04 Mar 2026 06:24:55 GMT
• Title: Cognition to Control - Multi-Agent Learning for Human-Humanoid Collaborative Transport
• Abstract summary: This paper introduces cognition-to-control (C2C), a three-layer hierarchy that makes the deliberation-to-control pathway explicit.<n> Experiments on collaborative manipulation tasks show higher success and robustness than single-agent and end-to-end baselines.
• Score: 36.304887250687536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective human-robot collaboration (HRC) requires translating high-level intent into contact-stable whole-body motion while continuously adapting to a human partner. Many vision-language-action (VLA) systems learn end-to-end mappings from observations and instructions to actions, but they often emphasize reactive (System 1-like) behavior and leave under-specified how sustained System 2-style deliberation can be integrated with reliable, low-latency continuous control. This gap is acute in multi-agent HRC, where long-horizon coordination decisions and physical execution must co-evolve under contact, feasibility, and safety constraints. We address this limitation with cognition-to-control (C2C), a three-layer hierarchy that makes the deliberation-to-control pathway explicit: (i) a VLM-based grounding layer that maintains persistent scene referents and infers embodiment-aware affordances/constraints; (ii) a deliberative skill/coordination layer-the System 2 core-that optimizes long-horizon skill choices and sequences under human-robot coupling via decentralized MARL cast as a Markov potential game with a shared potential encoding task progress; and (iii) a whole-body control layer that executes the selected skills at high frequency while enforcing kinematic/dynamic feasibility and contact stability. The deliberative layer is realized as a residual policy relative to a nominal controller, internalizing partner dynamics without explicit role assignment. Experiments on collaborative manipulation tasks show higher success and robustness than single-agent and end-to-end baselines, with stable coordination and emergent leader-follower behaviors.

Related papers

• Communication-Efficient Digital-Twin Coordination for Heterogeneous LLM Embodied Agents over Computing Power Networks [48.90302419915548]
  Embodied agent teams powered by heterogeneous large language models (LLMs) are being widely deployed in smart factories, warehouses, and service robotics.<n>We propose LDT-Coord, a networked coordination framework built upon a lightweight digital twin (DT)<n>Specifically, each agent independently selects its intended action and reports both the action decision and a structured temporal constraint over shared resources to the DT server.
  arXiv  Detail & Related papers  (2026-07-10T12:10:31Z)
• HRIBench: Benchmarking Interaction-Centric Human-Robot Collaboration [62.19372189311434]
  We introduce HRIBench, a diagnostic benchmark for intent-aware human-robot collaboration based on executable scenarios.<n>HRIBench represents collaborative tasks as structured scenario scripts that explicitly model agent roles, temporal dependencies, coordination constraints, and human behavior distributions.<n>Results show that current foundation robot policies struggle substantially in collaborative settings despite strong manipulation ability.
  arXiv  Detail & Related papers  (2026-07-05T09:15:35Z)
• HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning [38.43380992093738]
  We introduce HumanoidArena, a simulation-first benchmark for egocentric hierarchical whole-body learning.<n>A high-level policy converts egocentric vision, proprioception, and instructions into a compact whole-body action, which is subsequently executed by a low-level GMT.<n>Experiments show that hierarchical control enables learned policies to solve diverse leg-critical interactions, but performance is strongly tracker-conditioned and cross-GMT transfer remains fragile.
  arXiv  Detail & Related papers  (2026-06-16T12:01:31Z)
• Adaptive Human-AI Coordination via Hierarchical Action Disentanglement [1.4255659581428333]
  Human-AI collaboration requires agents that can adapt to diverse partner behaviors and skill levels while remaining robust to unseen partners.<n>We propose Intrinsic Action Disentanglement (IAD), a deep hierarchical reinforcement learning framework that learns distinct, partner-aware low-level action sequences conditioned on high-level latent skills.<n>We evaluate IAD in the Overcooked-AI domain across multiple layouts and diverse partner settings, including unseen simulated partners, a human-proxy model trained on human-human gameplay, and real human partners.
  arXiv  Detail & Related papers  (2026-05-23T01:59:41Z)
• HCLM: A Hierarchical Framework for Cooperative Loco-Manipulation with Dual Quadrupeds [24.47863400717969]
  We introduce HCLM, a hierarchical framework for general-purpose cooperative loco-manipulation with dual quadrupedal systems.<n>Our architecture systematically decouples high-level collaborative reasoning from low-level robust motion execution.<n>Results demonstrate reliable task execution, strict configurationism, and exceptional resilience against severe physical perturbations.
  arXiv  Detail & Related papers  (2026-05-17T07:23:03Z)
• Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems [65.90791804095561]
  Multi-agent systems address this through structured collaboration among specialized agents, but tighter coordination amplifies a less explored risk.<n>Existing surveys cover individual agent capabilities, multi-agent collaboration, or agent self-evolution separately, leaving the causal dependencies among them unexamined.<n>This survey provides a unified review organized around four causally linked stages, which we term the LIFE progression: Lay the capability foundation, Integrate agents through collaboration, Find faults through attribution, and Evolve through autonomous self-improvement.
  arXiv  Detail & Related papers  (2026-05-14T14:36:13Z)
• K^2-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Control [73.50217471850658]
  K2-Agent is a hierarchical framework that models human-like cognition by knowing and co-evolving declarative (what) and procedural (how) knowledge for planning and execution.<n>On the challenging AndroidWorld benchmark, K2-Agent achieves a 76.1% success rate using only raw and open-source backbones.
  arXiv  Detail & Related papers  (2026-02-28T14:33:14Z)
• TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving [37.09409794849228]
  Topology-conditioned Stackelberg Coordination is a learning framework for decentralized interactive driving under communication-free execution.<n>We show that TSC achieves superior performance over representative MARL baselines across key metrics.
  arXiv  Detail & Related papers  (2026-02-27T10:46:41Z)
• Multi-Round Human-AI Collaboration with User-Specified Requirements [26.38833436936642]
  We adopt a human centric view governed by two principles: counterfactual harm and complementarity.<n>We formalize these concepts via user defined rules, allowing users to specify exactly what harm and complementarity mean.<n>We show that our online procedure maintains prescribed counterfactual harm and complementarity violation rates even under nonstationary interaction dynamics.
  arXiv  Detail & Related papers  (2026-02-19T18:54:34Z)
• PACEvolve: Enabling Long-Horizon Progress-Aware Consistent Evolution [64.15555230987222]
  PACEvolve is a framework designed to robustly govern the agent's context and search dynamics.<n>We demonstrate that PACEvolve provides a systematic path to consistent, long-horizon self-improvement.
  arXiv  Detail & Related papers  (2026-01-15T18:25:23Z)
• CTHA: Constrained Temporal Hierarchical Architecture for Stable Multi-Agent LLM Systems [0.0]
  Multi-time-scale agent architectures have extended the ubiquitous single-loop paradigm by introducing temporal hierarchies with distinct cognitive layers.<n>We propose Constrained Temporal Hierarchical Architecture (CTHA) to restore coordination stability, while incorporating principled arbitration mechanisms to ensure coherent decision-making.<n>CTHA is effective for complex task execution at scale, offering 47% reduction in failure cascades, 2.3x improvement in sample efficiency, and superior scalability compared to unconstrained hierarchical baselines.
  arXiv  Detail & Related papers  (2026-01-09T08:03:14Z)
• QoS-Aware Hierarchical Reinforcement Learning for Joint Link Selection and Trajectory Optimization in SAGIN-Supported UAV Mobility Management [52.15690855486153]
  A space-air-ground integrated network (SAGIN) has emerged as an essential architecture for enabling ubiquitous UAV connectivity.<n>This paper formulates UAV mobility management in SAGIN as a constrained multiobjective joint optimization problem.
  arXiv  Detail & Related papers  (2025-12-17T06:22:46Z)
• Cooperative Target Detection with AUVs: A Dual-Timescale Hierarchical MARDL Approach [59.81681228738068]
  In adversarial environments, achieving efficient collaboration while ensuring covert operations is a key challenge for underwater cooperative missions.<n>We propose a novel dual time-scale Hierarchical Multi-Agent Proximal Policy Optimization framework.<n>We show that the proposed framework achieves rapid convergence, outperforms benchmark algorithms in terms of performance, and maximizes long-term cooperative efficiency while ensuring covert operations.
  arXiv  Detail & Related papers  (2025-09-16T09:31:32Z)
• Offline Multi-agent Reinforcement Learning via Score Decomposition [51.23590397383217]
  offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts.<n>This work is the first work to explicitly address the distributional gap between offline and online MARL.
  arXiv  Detail & Related papers  (2025-05-09T11:42:31Z)
• Position Paper: Towards Open Complex Human-AI Agents Collaboration Systems for Problem Solving and Knowledge Management [0.15039745292757667]
  We propose a technology-agnostic, collaboration-ready stance for Human-AI Agents Collaboration Systems (HAACS)<n>Reading empirical patterns through a seven-dimension collaboration spine and human-agent contrasts, we identify missing pieces.<n>We show interoperability with emerging agent protocols without ad hoc glue and sketch bio-cybernetic extensions.
  arXiv  Detail & Related papers  (2025-04-24T05:57:03Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions.<n>Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans.<n>We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)
• Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Tasks [17.914928652949314]
  We introduce the Hierarchical Consensus-based Multi-Agent Reinforcement Learning (HC-MARL) framework to address this limitation. HC-MARL employs contrastive learning to foster a global consensus among agents, enabling cooperative behavior without direct communication. To cater to the dynamic requirements of various tasks, consensus is divided into multiple layers, encompassing both short-term and long-term considerations.
  arXiv  Detail & Related papers  (2024-07-11T03:55:55Z)
• Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout [16.454305212398328]
  We propose a goal-conditioned hierarchical reinforcement learning (HRL) framework named Guided Cooperation via Model-based Rollout (GCMR) GCMR aims to bridge inter-layer information synchronization and cooperation by exploiting forward dynamics. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of HIGL, namely ACLG, yields more stable and robust policy improvement.
  arXiv  Detail & Related papers  (2023-09-24T00:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.