Continual Multimodal Egocentric Activity Recognition via Modality-Aware Novel Detection

• URL: http://arxiv.org/abs/2603.16970v1
• Date: Tue, 17 Mar 2026 11:03:27 GMT
• Title: Continual Multimodal Egocentric Activity Recognition via Modality-Aware Novel Detection
• Abstract summary: Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding.<n>Existing methods rely on the main logits for novelty scoring, without fully exploiting the complementary evidence available from individual modalities.<n>We propose MAND, a modality-aware framework for multimodal egocentric open-world continual learning.
• Score: 2.2207331768131815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding. However, deploying such systems in open-world environments requires detecting novel activities while continuously learning from non-stationary streams. Existing methods rely on the main logits for novelty scoring, without fully exploiting the complementary evidence available from individual modalities. Because these logits are often dominated by RGB, cues from other modalities, particularly IMU, remain underutilized, and this imbalance worsens over time under catastrophic forgetting. To address this, we propose MAND, a modality-aware framework for multimodal egocentric open-world continual learning. At inference, Modality-aware Adaptive Scoring (MoAS) estimates sample-wise modality reliability from energy scores and adaptively integrates modality logits to better exploit complementary modality cues for novelty detection. During training, Modality-wise Representation Stabilization Training (MoRST) preserves modality-specific discriminability across tasks via auxiliary heads and modality-wise logit distillation. Experiments on a public multimodal egocentric benchmark show that MAND improves novel activity detection AUC by up to 10\% and known-class classification accuracy by up to 2.8\% over state-of-the-art baselines.

Related papers

• BriMA: Bridged Modality Adaptation for Multi-Modal Continual Action Quality Assessment [25.689906499244533]
  Action Quality Assessment (AQA) aims to score how well an action is performed and is widely used in sports analysis, rehabilitation assessment, and human skill evaluation.<n>We introduce Bridged Modality Adaptation (BriMA), an innovative approach to multi-modal continual AQA under modality-missing conditions.<n>BriMA consists of a memory-guided bridging imputation module that reconstructs missing modalities using both task-agnostic and task-specific representations, and a modality-aware replay mechanism that prioritizes informative samples based on modality distortion and distribution drift.
  arXiv  Detail & Related papers  (2026-02-22T13:00:52Z)
• Dual-level Modality Debiasing Learning for Unsupervised Visible-Infrared Person Re-Identification [59.59359638389348]
  We propose a Dual-level Modality Debiasing Learning framework that implements debiasing at both the model and optimization levels.<n>Experiments on benchmark datasets demonstrate that DMDL could enable modality-invariant feature learning and a more generalized model.
  arXiv  Detail & Related papers  (2025-12-03T12:43:16Z)
• MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series [7.657107258507061]
  We introduce MAESTRO, a novel framework that overcomes key limitations of existing multimodal learning approaches.<n>At its core, MAESTRO facilitates dynamic intra- and cross-modal interactions based on task relevance.<n>We evaluate MAESTRO against 10 baselines on four diverse datasets spanning three applications.
  arXiv  Detail & Related papers  (2025-09-29T03:07:06Z)
• Active Multimodal Distillation for Few-shot Action Recognition [19.872938560809988]
  This paper presents a novel framework that actively identifies reliable modalities for each sample using task-specific contextual cues.<n>Our framework integrates an Active Sample Inference (ASI) module, which utilizes active inference to predict reliable modalities.<n>Unlike reinforcement learning, active inference replaces rewards with evidence-based preferences, making more stable predictions.
  arXiv  Detail & Related papers  (2025-06-16T10:10:56Z)
• Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities [45.38477844113637]
  KARMMA is a multimodal knowledge distillation approach for egocentric Action Recognition robust to Missing ModAlities.<n>Our student achieves competitive accuracy while significantly reducing accuracy drops under missing modality conditions.
  arXiv  Detail & Related papers  (2025-04-11T14:30:42Z)
• Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos [92.38662956154256]
  Real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues.<n>We propose a novel approach to address this issue at test time without requiring retraining.<n>MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time.
  arXiv  Detail & Related papers  (2024-04-23T16:01:33Z)
• Exploring Missing Modality in Multimodal Egocentric Datasets [89.76463983679058]
  We introduce a novel concept -Missing Modality Token (MMT)-to maintain performance even when modalities are absent. Our method mitigates the performance loss, reducing it from its original $sim 30%$ drop to only $sim 10%$ when half of the test set is modal-incomplete.
  arXiv  Detail & Related papers  (2024-01-21T11:55:42Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition [1.869225486385596]
  We explore the hypothesis that leveraging multiple modalities can lead to better recognition. We extend a number of recent contrastive self-supervised approaches for the task of Human Activity Recognition. We propose a flexible, general-purpose framework for performing multimodal self-supervised learning.
  arXiv  Detail & Related papers  (2022-05-20T10:39:16Z)
• Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification [208.1227090864602]
  Visible-infrared person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. Existing VI-ReID methods tend to learn global representations, which have limited discriminability and weak robustness to noisy images. We propose a novel dynamic dual-attentive aggregation (DDAG) learning method by mining both intra-modality part-level and cross-modality graph-level contextual cues for VI-ReID.
  arXiv  Detail & Related papers  (2020-07-18T03:08:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.