RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals

• URL: http://arxiv.org/abs/2608.00289v1
• Date: Fri, 31 Jul 2026 20:58:00 GMT
• Title: RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals
• Abstract summary: We introduce RF-HOI, the first framework that only uses radio frequency (RF) signals for HOI recognition.<n>A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with.<n>We develop a simulator that synthesizes multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data.
• Score: 19.108452211899984
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality, embodied AI, and assistive robotics. However, vision-based HOI methods face challenges in privacy concerns and poor light conditions. In this work, we introduce RF-HOI, the first framework that only uses radio frequency (RF) signals for HOI recognition. A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with. RF-HOI addresses this through a novel modality fusion that combines mmWave radar and RFID, enabling simultaneous action recognition and target identification. Another challenge is limited training data across diverse setups, which impairs the generalizability of the recognition model. To overcome this, we develop a simulator that synthesizes multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data. Experiment results show that RF-HOI outperforms all baselines, approaching vision model performance, and that our diverse synthetic training data can significantly boost our system's performance on real-world scenarios. These results highlight the potential of multimodal RF sensing for robust and privacy-preserving HOI recognition as well as the effectiveness of our RF data synthesis.

Related papers

• Semantically Annotated Multimodal Dataset for RF Interpretation and Prediction [4.872063946977515]
  Current limitations in wireless modeling and radio frequency (RF)-based AI are driven by a lack of high-quality, measurement-based datasets that connect RF signals to their physical environments.<n>We propose a new class of multimodal datasets that combines RF measurements with auxiliary modalities like high-resolution cameras and lidar to bridge the gap between RF signals and their physical causes.
  arXiv  Detail & Related papers  (2026-04-01T22:18:01Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Models [11.630591232366255]
  Vision-based gesture recognition suffers from high computational cost, sensitivity to lighting conditions, and privacy leakage concerns.<n> Acoustic sensing provides an attractive alternative: by emitting inaudible high-frequency signals and capturing their reflections, channel impulse response (CIR) encodes how gestures perturb the acoustic field in a low-cost and user-transparent manner.<n>We propose the first framework that leverages large language models (LLMs) for CIR-based gesture recognition in VR/AR systems.
  arXiv  Detail & Related papers  (2025-11-10T13:19:58Z)
• Scalable RF Simulation in Generative 4D Worlds [12.977761033016503]
  WaveVerse is a prompt-based, scalable framework that simulates realistic RF signals from generated indoor scenes with human motions.<n> Experiments demonstrate the effectiveness of our approach in conditioned human motion generation and highlight how phase coherence is applied to beamforming and respiration monitoring.
  arXiv  Detail & Related papers  (2025-08-16T23:02:14Z)
• Rydberg Atomic Quantum Receivers for Classical Wireless Communications and Sensing: Their Models and Performance [78.76421728334013]
  Rydberg atomic quantum receivers (RAQRs) are an eminent solution for detecting the electric field of radio frequency (RF) signals.<n>We introduce the RAQR to the wireless community by presenting an end-to-end reception scheme.<n>We then develop a corresponding equivalent baseband signal model relying on a realistic reception flow.
  arXiv  Detail & Related papers  (2024-12-07T06:25:54Z)
• Rydberg Atomic Quantum Receivers for Classical Wireless Communication and Sensing [71.94873601156017]
  Rydberg atomic quantum receivers (RAQR) are emerging quantum precision sensing platforms designed for receiving radio frequency (RF) signals.<n>RAQRs realize RF-to-optical conversions based on light-atom interactions.<n>Initial experimental studies have demonstrated their capabilities in classical wireless communications and sensing.
  arXiv  Detail & Related papers  (2024-09-22T15:55:02Z)
• RF Challenge: The Data-Driven Radio Frequency Signal Separation Challenge [66.33067693672696]
  We address the critical problem of interference rejection in radio-frequency (RF) signals using a data-driven approach that leverages deep-learning methods.<n>A primary contribution of this paper is the introduction of the RF Challenge, which is a publicly available, diverse RF signal dataset.
  arXiv  Detail & Related papers  (2024-09-13T13:53:41Z)
• Physical-Layer Semantic-Aware Network for Zero-Shot Wireless Sensing [74.12670841657038]
  Device-free wireless sensing has recently attracted significant interest due to its potential to support a wide range of immersive human-machine interactive applications. Data heterogeneity in wireless signals and data privacy regulation of distributed sensing have been considered as the major challenges that hinder the wide applications of wireless sensing in large area networking systems. We propose a novel zero-shot wireless sensing solution that allows models constructed in one or a limited number of locations to be directly transferred to other locations without any labeled data.
  arXiv  Detail & Related papers  (2023-12-08T13:50:30Z)
• Robust Semi-supervised Federated Learning for Images Automatic Recognition in Internet of Drones [57.468730437381076]
  We present a Semi-supervised Federated Learning (SSFL) framework for privacy-preserving UAV image recognition. There are significant differences in the number, features, and distribution of local data collected by UAVs using different camera modules. We propose an aggregation rule based on the frequency of the client's participation in training, namely the FedFreq aggregation rule.
  arXiv  Detail & Related papers  (2022-01-03T16:49:33Z)
• RF-Net: a Unified Meta-learning Framework for RF-enabled One-shot Human Activity Recognition [9.135311655929366]
  Device-free (or contactless) sensing is more sensitive to environment changes than device-based (or wearable) sensing. Existing solutions to RF-HAR entail a laborious data collection process for adapting to new environments. We propose RF-Net as a meta-learning based approach to one-shot RF-HAR; it reduces the labeling efforts for environment adaptation to the minimum level.
  arXiv  Detail & Related papers  (2021-10-29T01:58:29Z)
• Robotic Grasping of Fully-Occluded Objects using RF Perception [18.339320861642722]
  RF-Grasp is a robotic system that can grasp fully-occluded objects in unstructured environments. RF-Grasp relies on an eye-in-hand camera and batteryless RFID tags attached to objects of interest.
  arXiv  Detail & Related papers  (2020-12-31T04:01:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.