Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion

• URL: http://arxiv.org/abs/2501.07408v1
• Date: Mon, 13 Jan 2025 15:24:10 GMT
• Title: Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion
• Abstract summary: We propose Open Vocabulary HAR (OV-HAR), a framework that overcomes the limitation by first converting each activity into natural language.<n>Unlike other works that rely on auto-regressive large language models (LLMs), OV-HAR achieves open vocabulary recognition without the computational overhead of such models.
• Score: 4.831029473163422
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Conventional human activity recognition (HAR) relies on classifiers trained to predict discrete activity classes, inherently limiting recognition to activities explicitly present in the training set. Such classifiers would invariably fail, putting zero likelihood, when encountering unseen activities. We propose Open Vocabulary HAR (OV-HAR), a framework that overcomes this limitation by first converting each activity into natural language and breaking it into a sequence of elementary motions. This descriptive text is then encoded into a fixed-size embedding. The model is trained to regress this embedding, which is subsequently decoded back into natural language using a pre-trained embedding inversion model. Unlike other works that rely on auto-regressive large language models (LLMs) at their core, OV-HAR achieves open vocabulary recognition without the computational overhead of such models. The generated text can be transformed into a single activity class using LLM prompt engineering. We have evaluated our approach on different modalities, including vision (pose), IMU, and pressure sensors, demonstrating robust generalization across unseen activities and modalities, offering a fundamentally different paradigm from contemporary classifiers.

Related papers

• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic [45.298314099319775]
  Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations.<n>We propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models.<n>We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model.
  arXiv  Detail & Related papers  (2026-06-17T14:16:12Z)
• ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding [3.8013366586299093]
  We show how to model open-ended activity narratives by aligning wearable sensor data with natural-language descriptions in an open-vocabulary setting.<n>Results establish open-ended narrative modeling as a practical and effective foundation for real-world wearable HAR.
  arXiv  Detail & Related papers  (2026-04-01T11:31:44Z)
• Learning Yourself: Class-Incremental Semantic Segmentation with Language-Inspired Bootstrapped Disentanglement [10.912635927529218]
  Class-Incremental Semantic (CISS) requires continuous learning of newly introduced classes while retaining knowledge of past classes.<n>This phenomenon involves Prototype-Feature Entanglement caused by semantic misalignment during the incremental process, and Background-Increment Entanglement due to dynamic data evolution.<n>We introduce a Language-inspired Bootstrapped Disentanglement framework (LBD)<n>We achieve state-of-the-art performance on both Pascal VOC and ADE20k, particularly in multi-step scenarios.
  arXiv  Detail & Related papers  (2025-08-30T15:18:58Z)
• Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting [1.1871535995163365]
  Textual Inversion (TI) allows extending the VLM vocabulary by learning new or improving existing tokens to accurately detect novel or fine-grained objects from as little as three examples.<n>The storage and gradient calculations are limited to the token embedding dimension, requiring significantly less compute than full-model fine-tuning.<n>We evaluate whether the method matches or outperforms the baseline methods that suffer from forgetting in a variety of quantitative and qualitative experiments.
  arXiv  Detail & Related papers  (2025-08-07T12:28:08Z)
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization [50.27950279695363]
  Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages.<n>A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.<n>We propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding.
  arXiv  Detail & Related papers  (2025-04-21T19:40:32Z)
• Spatio-Temporal Context Prompting for Zero-Shot Action Detection [13.22912547389941]
  We propose a method which can effectively leverage the rich knowledge of visual-language models to perform Person-Context Interaction. To address the challenge of recognizing distinct actions by multiple people at the same timestamp, we design the Interest Token Spotting mechanism. Our method achieves superior results compared to previous approaches and can be further extended to multi-action videos.
  arXiv  Detail & Related papers  (2024-08-28T17:59:05Z)
• Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations [9.444540281544715]
  We introduce a novel agent for active open-vocabulary recognition. The proposed method leverages inter-frame and inter-concept similarities to navigate agent movements and to fuse features, without relying on class-specific knowledge.
  arXiv  Detail & Related papers  (2023-11-28T19:24:07Z)
• Towards Open-Ended Visual Recognition with Large Language Model [27.56182473356992]
  We introduce the OmniScient Model (OSM), a novel Large Language Model (LLM) based mask classifier. OSM predicts class labels in a generative manner, thus removing the supply of class names during both training and testing. It also enables cross-dataset training without any human interference.
  arXiv  Detail & Related papers  (2023-11-14T18:59:01Z)
• Multi-Semantic Fusion Model for Generalized Zero-Shot Skeleton-Based Action Recognition [32.291333054680855]
  Generalized zero-shot skeleton-based action recognition (GZSSAR) is a new challenging problem in computer vision community. We propose a multi-semantic fusion (MSF) model for improving the performance of GZSSAR.
  arXiv  Detail & Related papers  (2023-09-18T09:00:25Z)
• Global Knowledge Calibration for Fast Open-Vocabulary Segmentation [124.74256749281625]
  We introduce a text diversification strategy that generates a set of synonyms for each training category. We also employ a text-guided knowledge distillation method to preserve the generalizable knowledge of CLIP. Our proposed model achieves robust generalization performance across various datasets.
  arXiv  Detail & Related papers  (2023-03-16T09:51:41Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Learning Primitive-aware Discriminative Representations for Few-shot Learning [28.17404445820028]
  Few-shot learning aims to learn a classifier that can be easily adapted to recognize novel classes with only a few labeled examples. We propose a Primitive Mining and Reasoning Network (PMRN) to learn primitive-aware representations. Our method achieves state-of-the-art results on six standard benchmarks.
  arXiv  Detail & Related papers  (2022-08-20T16:22:22Z)
• Generative Action Description Prompts for Skeleton-based Action Recognition [15.38417530693649]
  We propose a Generative Action-description Prompts (GAP) approach for skeleton-based action recognition. We employ a pre-trained large-scale language model as the knowledge engine to automatically generate text descriptions for body parts movements of actions. Our proposed GAP method achieves noticeable improvements over various baseline models without extra cost at inference.
  arXiv  Detail & Related papers  (2022-08-10T12:55:56Z)
• Pre-Trained Language Models for Interactive Decision-Making [72.77825666035203]
  We describe a framework for imitation learning in which goals and observations are represented as a sequence of embeddings. We demonstrate that this framework enables effective generalization across different environments. For test tasks involving novel goals or novel scenes, initializing policies with language models improves task completion rates by 43.6%.
  arXiv  Detail & Related papers  (2022-02-03T18:55:52Z)
• Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates [52.164757178369804]
  Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We conduct an empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pre-trained models in the active learning framework. We also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance.
  arXiv  Detail & Related papers  (2021-01-20T13:59:25Z)
• SLM: Learning a Discourse Language Representation with Sentence Unshuffling [53.42814722621715]
  We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation. We show that this feature of our model improves the performance of the original BERT by large margins.
  arXiv  Detail & Related papers  (2020-10-30T13:33:41Z)
• Pre-trained Language Model Based Active Learning for Sentence Matching [18.48335957524662]
  We propose a pre-trained language model based active learning approach for sentence matching. Our approach can achieve greater accuracy with fewer labeled training instances.
  arXiv  Detail & Related papers  (2020-10-12T08:24:36Z)
• Inferring Temporal Compositions of Actions Using Probabilistic Automata [61.09176771931052]
  We propose to express temporal compositions of actions as semantic regular expressions and derive an inference framework using probabilistic automata. Our approach is different from existing works that either predict long-range complex activities as unordered sets of atomic actions, or retrieve videos using natural language sentences.
  arXiv  Detail & Related papers  (2020-04-28T00:15:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.