A Vision Based Framework Integrating Attention and Action Cues for Interpretable Cognitive Workload Assessment in Human-Robot Collaborative Assembly

• URL: http://arxiv.org/abs/2609.15232v2
• Date: Tue, 15 Sep 2026 04:46:09 GMT
• Title: A Vision Based Framework Integrating Attention and Action Cues for Interpretable Cognitive Workload Assessment in Human-Robot Collaborative Assembly
• Abstract summary: We present a vision-based attention-action framework for continuous and interpretable workload-related assessment in HRC assembly.<n>We evaluated the framework in a three-level collaborative gearbox assembly experiment with ten participants.
• Score: 3.2566202265353503
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The introduction of human-robot collaboration (HRC) in industrial assembly operations is revolutionizing the manufacturing landscape. In this evolving environment, operators are required to seamlessly coordinate their manual tasks with real-time task information and robotic behaviors. These demands fluctuate during operation, yet conventional workload assessments depend on body-worn physiological sensors that complicate practical deployment. Here, we present a vision-based attention--action framework for continuous and interpretable workload-related assessment in HRC assembly. The framework combines RGB-D observations with robot states and calibrated task-related areas to construct a temporally confirmed representation of operator behavior. This representation identifies where task demand is concentrated and explains how it develops when attention and action diverge, the task context changes, or the operator hesitates. We evaluated the framework in a three-level collaborative gearbox assembly experiment with ten participants, using subjective ratings and synchronized physiological signals as independent references. Raw NASA-TLX ratings confirmed increasing perceived workload across conditions, with significant effects on overall workload and its mental and temporal dimensions. The vision-derived HRC-CWL output was significantly associated with ECG-derived features in seven of nine participants with complete correlation data. Synchronized interaction episodes further showed temporal correspondence between detected hesitation and physiological activity. Real-time deployment demonstrated that the framework can operate without requiring operators to wear additional sensors. These findings support HRC-CWL as an interpretable behavioral proxy for cognitive ergonomics analysis and adaptive robot assistance, rather than a direct psychophysiological measure of workload.

Related papers

• HRIBench: Benchmarking Interaction-Centric Human-Robot Collaboration [62.19372189311434]
  We introduce HRIBench, a diagnostic benchmark for intent-aware human-robot collaboration based on executable scenarios.<n>HRIBench represents collaborative tasks as structured scenario scripts that explicitly model agent roles, temporal dependencies, coordination constraints, and human behavior distributions.<n>Results show that current foundation robot policies struggle substantially in collaborative settings despite strong manipulation ability.
  arXiv  Detail & Related papers  (2026-07-05T09:15:35Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• PACT: Proactive Asking for Continual Task Assistance in Human-Robot Collaboration [62.13457838323877]
  We propose PACT (Proactive Asking for Continual Task Assistance), an ask-or-act framework that determines whether clarification should be sought before taking action.<n>We implement its primary learned instantiation using reinforcement learning and evaluate alternative instantiations under the same framework.<n>Experiments in multi-day embodied collaboration scenarios demonstrate that, compared with passive inference baselines, PACT consistently improves both assistance accuracy and clarification utility.
  arXiv  Detail & Related papers  (2026-05-23T02:22:02Z)
• Cooperative Informative Sensing for Monitoring Dynamic Indoor Environments via Multi-Agent Reinforcement Learning [56.64821510576244]
  We formulate cooperative active observation as a decentralized control problem in which multiple robots adjust their motion to directly optimize monitoring accuracy under partial observability.<n>We propose a learning-based framework for cooperative policies from decentralized observations using multi-agent reinforcement learning (MARL), supported by an architecture that handles variable numbers of humans and temporal dependencies.
  arXiv  Detail & Related papers  (2026-04-25T07:20:15Z)
• Interaction-Aware Whole-Body Control for Compliant Object Transport [33.233203393813376]
  This paper proposes an interaction-oriented whole-body control (IO-WBC) that functions as an artificial cerebellum.<n>IO-WBC translates upstream (skill-level) commands into stable, physically consistent whole-body behavior under contact.
  arXiv  Detail & Related papers  (2026-03-04T05:50:40Z)
• GAZELOAD A Multimodal Eye-Tracking Dataset for Mental Workload in Industrial Human-Robot Collaboration [0.0]
  The data were collected in a laboratory assembly testbed where 26 participants interacted with two collaborative robots.<n>The dataset time-synchronizes eye-tracking signals with environmental real-time and continuous measurements.<n>These data can be used to develop and benchmark algorithms for mental workload estimation.
  arXiv  Detail & Related papers  (2026-01-29T15:12:58Z)
• Trainee Action Recognition through Interaction Analysis in CCATT Mixed-Reality Training [1.5641818606249476]
  Critical Care Air Transport Team members must stabilize severely injured soldiers by managing ventilators, IV pumps, and suction devices during flight.<n>Recent advances in simulation and multimodal data analytics enable more objective and comprehensive performance evaluation.<n>This study examines how CCATT members are trained using mixed-reality simulations that replicate the high-pressure conditions of aeromedical evacuation.
  arXiv  Detail & Related papers  (2025-09-22T15:19:45Z)
• HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario [63.77482302352545]
  HHI-Assist is a dataset comprising motion capture clips of human-human interactions in assistive tasks.<n>Our work has the potential to significantly enhance robotic assistance policies.
  arXiv  Detail & Related papers  (2025-09-12T09:38:17Z)
• Analyzing Participants' Engagement during Online Meetings Using Unsupervised Remote Photoplethysmography with Behavioral Features [50.82725748981231]
  Engagement measurement finds application in healthcare, education, services. Use of physiological and behavioral features is viable, but impracticality of traditional physiological measurement arises due to the need for contact sensors. We demonstrate the feasibility of the unsupervised photoplethysmography (rmography) as an alternative for contact sensors.
  arXiv  Detail & Related papers  (2024-04-05T20:39:16Z)
• Computational ergonomics for task delegation in Human-Robot Collaboration: spatiotemporal adaptation of the robot to the human through contactless gesture recognition [0.0]
  This paper proposes two hypotheses for ergonomically effective task delegation and Human Human Collaboration (HRC) The first hypothesis states that it is possible to quantify ergonomically professional tasks using motion data from a reduced set of sensors. The second hypothesis is that by including gesture recognition and spatial adaptation, the ergonomics of an HRC scenario can be improved by avoiding needless motions.
  arXiv  Detail & Related papers  (2022-03-21T14:23:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.