M3Act: Learning from Synthetic Human Group Activities

• URL: http://arxiv.org/abs/2306.16772v6
• Date: Fri, 3 May 2024 00:00:27 GMT
• Title: M3Act: Learning from Synthetic Human Group Activities
• Authors: Che-Jui Chang, Danrui Li, Deep Patel, Parth Goel, Honglu Zhou, Seonghyeon Moon, Samuel S. Sohn, Sejong Yoon, Vladimir Pavlovic, Mubbasir Kapadia,
• Abstract summary: M3Act is a synthetic data generator for multi-view multi-group multi-person human atomic actions and group activities. Powered by Unity Engine, M3Act features multiple semantic groups, highly diverse and photorealistic images. M3Act improves the state-of-the-art MOTRv2 on DanceTrack dataset, leading to a hop on the leaderboard from 10th to 2nd place.
• Score: 18.264989896254523
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The study of complex human interactions and group activities has become a focal point in human-centric computer vision. However, progress in related tasks is often hindered by the challenges of obtaining large-scale labeled datasets from real-world scenarios. To address the limitation, we introduce M3Act, a synthetic data generator for multi-view multi-group multi-person human atomic actions and group activities. Powered by Unity Engine, M3Act features multiple semantic groups, highly diverse and photorealistic images, and a comprehensive set of annotations, which facilitates the learning of human-centered tasks across single-person, multi-person, and multi-group conditions. We demonstrate the advantages of M3Act across three core experiments. The results suggest our synthetic dataset can significantly improve the performance of several downstream methods and replace real-world datasets to reduce cost. Notably, M3Act improves the state-of-the-art MOTRv2 on DanceTrack dataset, leading to a hop on the leaderboard from 10th to 2nd place. Moreover, M3Act opens new research for controllable 3D group activity generation. We define multiple metrics and propose a competitive baseline for the novel task. Our code and data are available at our project page: http://cjerry1243.github.io/M3Act.

Related papers

• InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation [54.09384502044162]
  We introduce InterAct, a large-scale 3D HOI benchmark featuring dataset and methodological advancements.<n>First, we consolidate and standardize 21.81 hours of HOI data from diverse sources, enriching it with detailed textual annotations.<n>Second, we propose a unified optimization framework to enhance data quality by reducing artifacts and correcting hand motions.<n>Third, we define six benchmarking tasks and develop a unified HOI generative modeling perspective, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-11T15:43:54Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [73.49799596304418]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing. It is designed to accurately detect horizontal or oriented objects from any sensor modality. This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  This dataset includes figures such as schematic diagrams, simulated images, macroscopic/microscopic photos, and experimental visualizations. We developed benchmarks for scientific figure captioning and multiple-choice questions, evaluating six proprietary and over ten open-source models. The dataset and benchmarks will be released to support further research.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• Multi-Space Alignments Towards Universal LiDAR Segmentation [50.992103482269016]
  M3Net is a one-of-a-kind framework for fulfilling multi-task, multi-dataset, multi-modality LiDAR segmentation. We first combine large-scale driving datasets acquired by different types of sensors from diverse scenes. We then conduct alignments in three spaces, namely data, feature, and label spaces, during the training.
  arXiv  Detail & Related papers  (2024-05-02T17:59:57Z)
• HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment [43.6454394625555]
  HOI-M3 is a novel large-scale dataset for modeling the interactions of Multiple huMans and Multiple objects. It provides accurate 3D tracking for both humans and objects from dense RGB and object-mounted IMU inputs.
  arXiv  Detail & Related papers  (2024-03-30T09:24:25Z)
• VEnvision3D: A Synthetic Perception Dataset for 3D Multi-Task Model Research [10.764333144509571]
  VEnvision3D is a large 3D synthetic perception dataset for multi-task learning. Sub-tasks are inherently aligned in terms of the utilized data. Our dataset and code will be open-sourced upon acceptance.
  arXiv  Detail & Related papers  (2024-02-29T11:38:44Z)
• MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World [55.878173953175356]
  We propose MultiPLY, a multisensory embodied large language model. We first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks.
  arXiv  Detail & Related papers  (2024-01-16T18:59:45Z)
• An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training [79.78201886156513]
  We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Our approach achieves comparable results to single-task state-of-the-art models and demonstrates strong generalization on downstream tasks.
  arXiv  Detail & Related papers  (2023-06-29T17:59:57Z)
• The MI-Motion Dataset and Benchmark for 3D Multi-Person Motion Prediction [13.177817435234449]
  3D multi-person motion prediction is a challenging task that involves modeling individual behaviors and interactions between people. We introduce the Multi-Person Interaction Motion (MI-Motion) dataset, which includes skeleton sequences of multiple individuals collected by motion capture systems. The dataset contains 167k frames of interacting people's skeleton poses and is categorized into 5 different activity scenes.
  arXiv  Detail & Related papers  (2023-06-23T15:38:22Z)
• FLAG3D: A 3D Fitness Activity Dataset with Language Instruction [89.60371681477791]
  We present FLAG3D, a large-scale 3D fitness activity dataset with language instruction containing 180K sequences of 60 categories. We show that FLAG3D contributes great research value for various challenges, such as cross-domain human action recognition, dynamic human mesh recovery, and language-guided human action generation.
  arXiv  Detail & Related papers  (2022-12-09T02:33:33Z)
• Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets [0.0]
  Facial action units allow an objective, standardized description of facial micro movements which can be used to describe emotions in human faces. Annotating data for action units is an expensive and time-consuming task, which leads to a scarce data situation. We present our approach to create a combined database and an algorithm capable of learning under the presence of missing labels.
  arXiv  Detail & Related papers  (2020-08-17T11:58:06Z)
• The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose [108.21037046507483]
  IKEA ASM is a three million frame, multi-view, furniture assembly video dataset that includes depth, atomic actions, object segmentation, and human pose. We benchmark prominent methods for video action recognition, object segmentation and human pose estimation tasks on this challenging dataset. The dataset enables the development of holistic methods, which integrate multi-modal and multi-view data to better perform on these tasks.
  arXiv  Detail & Related papers  (2020-07-01T11:34:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.