A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition

• URL: http://arxiv.org/abs/2403.15444v1
• Date: Sun, 17 Mar 2024 22:31:14 GMT
• Title: A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition
• Authors: Abhi Kamboj, Minh Do,
• Abstract summary: We investigate how knowledge can be transferred and utilized amongst modalities for Human Activity/Action Recognition (HAR) We motivate the importance and potential of IMU data and its applicability in cross-modality learning. We discuss future research directions and applications in cross-modal HAR.
• Score: 0.9208007322096532
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite living in a multi-sensory world, most AI models are limited to textual and visual understanding of human motion and behavior. In fact, full situational awareness of human motion could best be understood through a combination of sensors. In this survey we investigate how knowledge can be transferred and utilized amongst modalities for Human Activity/Action Recognition (HAR), i.e. cross-modality transfer learning. We motivate the importance and potential of IMU data and its applicability in cross-modality learning as well as the importance of studying the HAR problem. We categorize HAR related tasks by time and abstractness and then compare various types of multimodal HAR datasets. We also distinguish and expound on many related but inconsistently used terms in the literature, such as transfer learning, domain adaptation, representation learning, sensor fusion, and multimodal learning, and describe how cross-modal learning fits with all these concepts. We then review the literature in IMU-based cross-modal transfer for HAR. The two main approaches for cross-modal transfer are instance-based transfer, where instances of one modality are mapped to another (e.g. knowledge is transferred in the input space), or feature-based transfer, where the model relates the modalities in an intermediate latent space (e.g. knowledge is transferred in the feature space). Finally, we discuss future research directions and applications in cross-modal HAR.

Related papers

• Enhancing Cross-task Transfer of Large Language Models via Activation Steering [75.41750053623298]
  Cross-task in-context learning offers a direct solution for transferring knowledge across tasks.<n>We investigate whether cross-task transfer can be achieved via latent space steering without parameter updates or input expansion.<n>We propose a novel Cross-task Activation Steering Transfer framework that enables effective transfer by manipulating the model's internal activation states.
  arXiv  Detail & Related papers  (2025-07-17T15:47:22Z)
• Quantifying Cross-Modality Memorization in Vision-Language Models [86.82366725590508]
  We study the unique characteristics of cross-modality memorization and conduct a systematic study centered on vision-language models.<n>Our results reveal that facts learned in one modality transfer to the other, but a significant gap exists between recalling information in the source and target modalities.
  arXiv  Detail & Related papers  (2025-06-05T16:10:47Z)
• Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework [58.362064122489166]
  This paper introduces the Cross-modal Few-Shot Learning task, which aims to recognize instances across multiple modalities while relying on scarce labeled data. We propose a Generative Transfer Learning framework by simulating how humans abstract and generalize concepts. We show that the GTL achieves state-of-the-art performance across seven multi-modal datasets across RGB-Sketch, RGB-Infrared, and RGB-Depth.
  arXiv  Detail & Related papers  (2024-10-14T16:09:38Z)
• A Comprehensive Methodological Survey of Human Activity Recognition Across Divers Data Modalities [2.916558661202724]
  Human Activity Recognition (HAR) systems aim to understand human behaviour and assign a label to each action. HAR can leverage various data modalities, such as RGB images and video, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, and radar signals. This paper presents a comprehensive survey of the latest advancements in HAR from 2014 to 2024.
  arXiv  Detail & Related papers  (2024-09-15T10:04:44Z)
• Unified Framework with Consistency across Modalities for Human Activity Recognition [14.639249548669756]
  We propose a comprehensive framework for robust video-based human activity recognition. Key contribution is the introduction of a novel query machine, called COMPUTER. Our approach demonstrates superior performance when compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-09-04T02:25:10Z)
• A Survey on Multimodal Wearable Sensor-based Human Action Recognition [15.054052500762559]
  Wearable Sensor-based Human Activity Recognition (WSHAR) emerges as a promising assistive technology to support the daily lives of older individuals. Recent surveys in WSHAR have been limited, focusing either solely on deep learning approaches or on a single sensor modality. In this study, we present a comprehensive survey on how to leverage multimodal learning to WSHAR domain for newcomers and researchers.
  arXiv  Detail & Related papers  (2024-04-14T18:43:16Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Multimodal Representation Learning by Alternating Unimodal Adaptation [73.15829571740866]
  We propose MLA (Multimodal Learning with Alternating Unimodal Adaptation) to overcome challenges where some modalities appear more dominant than others during multimodal learning. MLA reframes the conventional joint multimodal learning process by transforming it into an alternating unimodal learning process. It captures cross-modal interactions through a shared head, which undergoes continuous optimization across different modalities. Experiments are conducted on five diverse datasets, encompassing scenarios with complete modalities and scenarios with missing modalities.
  arXiv  Detail & Related papers  (2023-11-17T18:57:40Z)
• Cross-Domain HAR: Few Shot Transfer Learning for Human Activity Recognition [0.2944538605197902]
  We present an approach for economic use of publicly available labeled HAR datasets for effective transfer learning. We introduce a novel transfer learning framework, Cross-Domain HAR, which follows the teacher-student self-training paradigm. We demonstrate the effectiveness of our approach for practically relevant few shot activity recognition scenarios.
  arXiv  Detail & Related papers  (2023-10-22T19:13:25Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
  multimodal machine learning that incorporates data from various sources has become an increasingly popular research area. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
  arXiv  Detail & Related papers  (2022-10-05T13:14:57Z)
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning [112.51498431119616]
  This paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. A single model, HighMMT, scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas.
  arXiv  Detail & Related papers  (2022-03-02T18:56:20Z)
• Behavior Priors for Efficient Reinforcement Learning [97.81587970962232]
  We consider how information and architectural constraints can be combined with ideas from the probabilistic modeling literature to learn behavior priors. We discuss how such latent variable formulations connect to related work on hierarchical reinforcement learning (HRL) and mutual information and curiosity based objectives. We demonstrate the effectiveness of our framework by applying it to a range of simulated continuous control domains.
  arXiv  Detail & Related papers  (2020-10-27T13:17:18Z)
• Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability [27.65302656389911]
  We analyze and demonstrate the connections between knowledge transferability and adversarial transferability. Our theoretical studies show that adversarial transferability indicates knowledge transferability and vice versa. We conduct extensive experiments for different scenarios on diverse datasets, showing a positive correlation between adversarial transferability and knowledge transferability.
  arXiv  Detail & Related papers  (2020-06-25T16:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.