Human-AI Divergence in Ego-centric Action Recognition under Spatial and Spatiotemporal Manipulations

• URL: http://arxiv.org/abs/2603.08317v1
• Date: Mon, 09 Mar 2026 12:38:56 GMT
• Title: Human-AI Divergence in Ego-centric Action Recognition under Spatial and Spatiotemporal Manipulations
• Abstract summary: Humans consistently outperform state-of-the-art AI models in action recognition.<n>We present a large-scale human-AI comparative study of ego using Minimal Identifiable Recognition Crops (MIRCs)<n>We show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs.
• Score: 12.465670388296239
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans consistently outperform state-of-the-art AI models in action recognition, particularly in challenging real-world conditions involving low resolution, occlusion, and visual clutter. Understanding the sources of this performance gap is essential for developing more robust and human-aligned models. In this paper, we present a large-scale human-AI comparative study of egocentric action recognition using Minimal Identifiable Recognition Crops (MIRCs), defined as the smallest spatial or spatiotemporal regions sufficient for reliable human recognition. We used our previously introduced, Epic ReduAct, a systematically spatially reduced and temporally scrambled dataset derived from 36 EPIC KITCHENS videos, spanning multiple spatial reduction levels and temporal conditions. Recognition performance is evaluated using over 3,000 human participants and the Side4Video model. Our analysis combines quantitative metrics, Average Reduction Rate and Recognition Gap, with qualitative analyses of spatial (high-, mid-, and low-level visual features) and spatiotemporal factors, including a categorisation of actions into Low Temporal Actions (LTA) and High Temporal Actions (HTA). Results show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs, reflecting a strong reliance on sparse, semantically critical cues such as hand-object interactions. In contrast, the model degrades more gradually and often relies on contextual and mid- to low-level features, sometimes even exhibiting increased confidence under spatial reduction. Temporally, humans remain robust to scrambling when key spatial cues are preserved, whereas the model often shows insensitivity to temporal disruption, revealing class-dependent temporal sensitivities.

Related papers

• Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds [49.219348802596876]
  Understanding humans from LiDAR point clouds is one of the most critical tasks in autonomous driving.<n>Existing methods largely overlook the potential of leveraging human-object interactions to build robust 3D human pose estimation frameworks.<n>We propose a Human-Object Interaction Learning framework for robust 3D human pose estimation from LiDAR point clouds.
  arXiv  Detail & Related papers  (2026-03-17T10:20:29Z)
• Spatial Causal Prediction in Video [56.22332198384257]
  We introduce a new task paradigm that challenges models to reason beyond observation and predict spatial causal outcomes.<n>We construct a benchmark comprising 2,500 QA pairs across 1,181 videos spanning diverse viewpoints, scenes, and causal directions.<n>Through comprehensive experiments on 23 state-of-the-art models, we reveal substantial gaps between human and model performance limited temporal extrapolation, and weak causal grounding.
  arXiv  Detail & Related papers  (2026-03-04T11:09:39Z)
• Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations [63.80827184637476]
  We introduce D-STAR, a hierarchical policy that disentangles when to act from where to act.<n>We validate our framework through extensive and rigorous simulations.
  arXiv  Detail & Related papers  (2026-01-14T14:37:06Z)
• Detecting Autism Spectrum Disorder with Deep Eye Movement Features [9.225838905985958]
  Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by deficits in social communication and behavioral patterns.<n>Eye movement data offers a non-invasive diagnostic tool for ASD detection.<n>To efficiently capture subtle and complex eye movement patterns, a discrete short-term sequential (DSTS) modeling framework is designed.
  arXiv  Detail & Related papers  (2026-01-09T14:35:24Z)
• Unleashing Temporal Capacity of Spiking Neural Networks through Spatiotemporal Separation [67.69345363409835]
  Spiking Neural Networks (SNNs) are considered naturally suited for temporal processing, with membrane potential propagation widely regarded as the core temporal modeling mechanism.<n>We design Non-Stateful (NS) models progressively removing membrane propagation to its stage-wise role. Experiments reveal a counterintuitive phenomenon: moderate removal in shallow layers improves performance, while excessive removal causes collapse.
  arXiv  Detail & Related papers  (2025-12-05T07:05:53Z)
• Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos [4.872114804382539]
  We introduce a novel evaluation metric derived from a learned latent space of real-world human actions.<n>Our method first captures the nuances, constraints, and temporal smoothness of real-world motion by fusing appearance-agnostic human skeletal geometry features with appearance-based features.<n>Given a generated video, our metric quantifies its action quality by measuring the distance between its underlying representations and this learned real-world action distribution.
  arXiv  Detail & Related papers  (2025-12-01T15:36:33Z)
• Spatio-Temporal Branching for Motion Prediction using Motion Increments [55.68088298632865]
  Human motion prediction (HMP) has emerged as a popular research topic due to its diverse applications. Traditional methods rely on hand-crafted features and machine learning techniques. We propose a noveltemporal-temporal branching network using incremental information for HMP.
  arXiv  Detail & Related papers  (2023-08-02T12:04:28Z)
• Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework [3.3721926640077795]
  Vision-based human activity recognition has emerged as one of the essential research areas in video analytics domain. This paper presents a computationally efficient yet generic spatial-temporal cascaded framework that exploits the deep discriminative spatial and temporal features for human activity recognition. The proposed framework attains an improvement in execution time up to 167 times in terms of frames per second as compared to most of the contemporary action recognition methods.
  arXiv  Detail & Related papers  (2022-08-09T20:34:42Z)
• Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction [63.62263239934777]
  We conduct an indepth study on various pose representations with a focus on their effects on the motion prediction task. We propose a novel RNN architecture termed AHMR (Attentive Hierarchical Motion Recurrent network) for motion prediction. Our approach outperforms the state-of-the-art methods in short-term prediction and achieves much enhanced long-term prediction proficiency.
  arXiv  Detail & Related papers  (2021-12-30T10:45:22Z)
• Human Activity Recognition from Wearable Sensor Data Using Self-Attention [2.9023633922848586]
  We present a self-attention based neural network model for activity recognition from body-worn sensor data. We performed experiments on four popular publicly available HAR datasets: PAMAP2, Opportunity, Skoda and USC-HAD. Our model achieve significant performance improvement over recent state-of-the-art models in both benchmark test subjects and Leave-one-out-subject evaluation.
  arXiv  Detail & Related papers  (2020-03-17T14:16:57Z)
• Human Action Recognition and Assessment via Deep Neural Network Self-Organization [0.0]
  This chapter introduces a set of hierarchical models for the learning and recognition of actions from depth maps and RGB images. A particularity of these models is the use of growing self-organizing networks that quickly adapt to non-stationary distributions.
  arXiv  Detail & Related papers  (2020-01-04T15:58:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.