A Large-Scale Multimodal Dataset and Benchmarks for Human Activity Scene Understanding and Reasoning

• URL: http://arxiv.org/abs/2512.07136v1
• Date: Mon, 08 Dec 2025 03:40:52 GMT
• Title: A Large-Scale Multimodal Dataset and Benchmarks for Human Activity Scene Understanding and Reasoning
• Abstract summary: We introduce CUHK-X, a large-scale multimodal dataset and benchmark suite for HAR, HAU, and HARn.<n>To improve caption consistency, we propose a prompt-based scene creation method.<n>Experiments report average accuracies of 76.52% (HAR), 40.76% (HAU), and 70.25% (HARn)
• Score: 18.38037245999228
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal human action recognition (HAR) leverages complementary sensors for activity classification. Beyond recognition, recent advances in large language models (LLMs) enable detailed descriptions and causal reasoning, motivating new tasks: human action understanding (HAU) and human action reasoning (HARn). However, most LLMs, especially large vision language models (LVLMs), struggle with non-RGB modalities such as depth, IMU, and mmWave due to the lack of large-scale data-caption resources. Existing HAR datasets mainly provide coarse data-label annotations, which are insufficient to capture fine-grained action dynamics needed for HAU and HARn. We consider two ground-truth pair types: (1) data label (discrete category) and (2) data caption (textual description). Naively generating captions from labels often lacks logical and spatiotemporal consistency. We introduce CUHK-X, a large-scale multimodal dataset and benchmark suite for HAR, HAU, and HARn. CUHK-X contains 58,445 samples covering 40 actions performed by 30 participants across two indoor environments. To improve caption consistency, we propose a prompt-based scene creation method that leverages LLMs to generate logically connected activity sequences, followed by human validation. CUHK-X includes three benchmarks with six evaluation tasks. Experiments report average accuracies of 76.52% (HAR), 40.76% (HAU), and 70.25% (HARn). CUHK-X aims to enable the community to apply and develop data-intensive learning methods for robust, multimodal human activity analysis. Project page and code: https://openaiotlab.github.io/CUHK-X/ and https://github.com/openaiotlab/CUHK-X.

Related papers

• MODF-SIR: A Multi-agent Omni-modal Distilled Framework for Social Intelligence Reasoning [80.41987492855971]
  We propose a multi-agent collaborative framework built upon a lightweight Multimodal Large Language Model (MLLM)<n>Within this architecture, multi-modal data pertinent to social intelligence is precisely localized.<n>With around 30% of training data from IntentTrain, we achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2026-06-10T12:44:02Z)
• Detect Anything via Next Point Prediction [51.55967987350882]
  Rex- Omni is a 3B-scale MLLM that achieves state-of-the-art object perception performance.<n>On benchmarks like COCO and LVIS, Rex- Omni attains performance comparable to or exceeding regression-based models.
  arXiv  Detail & Related papers  (2025-10-14T17:59:54Z)
• Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models [44.32482918853282]
  There are no works that can leverage multi-source MLLM tasks for stable reinforcement learning.<n>We present Mixed-R1, a unified yet straightforward framework that contains a mixed reward function design (Mixed-Reward) and a mixed post-training dataset (Mixed-45K)<n>In particular, it has four different reward functions: matching reward for binary answer or multiple-choice problems, chart reward for chart-aware datasets, IoU reward for grounding problems, and open-ended reward for long-form text responses such as caption datasets.
  arXiv  Detail & Related papers  (2025-05-30T03:11:46Z)
• Can Multimodal Large Language Models Understand Spatial Relations? [16.76001474065412]
  We introduce SpatialMQA, a human-annotated spatial relation reasoning benchmark based on COCO 2017.<n>Results indicate that the current state-of-the-art MLLM achieves only 48.14% accuracy, far below the human-level accuracy of 98.40%.
  arXiv  Detail & Related papers  (2025-05-25T07:37:34Z)
• Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understanding [1.1228672751176365]
  Human activity recognition (HAR) using inertial measurement units (IMUs) leverages increasingly large language models (LLMs)<n>Our preliminary study indicates that pretrained LLMs fail catastrophically on fine-grained HAR tasks such as air-written letter recognition, achieving only near-random guessing accuracy.<n>To extend this to 3D, we designed an encoder-based pipeline that maps 3D data into 2D equivalents, preserving thetemporal information for robust letter prediction.<n>Our end-to-end pipeline achieves 78% accuracy on word recognition with up to 5 letters in mid-air writing scenarios, establishing LLMs as viable tools for
  arXiv  Detail & Related papers  (2025-04-02T03:42:58Z)
• ChatRex: Taming Multimodal LLM for Joint Perception and Understanding [16.535876222927538]
  We introduce ChatRex, an MLLM with a decoupled perception design.<n>From the data perspective, we build a fully automated data engine to support the joint training of perception and understanding.<n>ChatRex demonstrates strong perception and understanding performance, and the combination of these two capabilities also unlocks many attractive applications.
  arXiv  Detail & Related papers  (2024-11-27T14:11:10Z)
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning [40.972648044298374]
  Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. They often lack interpretability and struggle with complex visual inputs. We introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs. We propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts.
  arXiv  Detail & Related papers  (2024-03-25T17:59:23Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs [49.88461345825586]
  This paper proposes a new framework to enhance the fine-grained image understanding abilities of MLLMs. We present a new method for constructing the instruction tuning dataset at a low cost by leveraging annotations in existing datasets. We show that our model exhibits a 5.2% accuracy improvement over Qwen-VL and surpasses the accuracy of Kosmos-2 by 24.7%.
  arXiv  Detail & Related papers  (2023-10-01T05:53:15Z)
• Multi-Granularity Hand Action Detection [58.88274905101276]
  FHA-Kitchens dataset comprises 2,377 video clips and 30,047 frames, annotated with approximately 200k bounding boxes and 880 action categories. This dataset comprises 2,377 video clips and 30,047 frames, annotated with approximately 200k bounding boxes and 880 action categories. We propose MG-HAD, an End-to-End Multi-Granularity Hand Action Detection method.
  arXiv  Detail & Related papers  (2023-06-19T11:21:59Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• Deep Bidirectional Language-Knowledge Graph Pretraining [159.9645181522436]
  DRAGON is a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities.
  arXiv  Detail & Related papers  (2022-10-17T18:02:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.