RFMask: A Simple Baseline for Human Silhouette Segmentation with Radio Signals

• URL: http://arxiv.org/abs/2201.10175v1
• Date: Tue, 25 Jan 2022 08:43:01 GMT
• Title: RFMask: A Simple Baseline for Human Silhouette Segmentation with Radio Signals
• Authors: Zhi Wu, Dongheng Zhang, Chunyang Xie, Cong Yu, Jinbo Chen, Yang Hu, Yan Chen
• Abstract summary: We propose to utilize the radio signals, which can traverse obstacles and are unaffected by the lighting conditions to achieve silhouette segmentation. The proposed RFMask framework is composed of three modules. We collect a dataset containing 804,760 radio frames and 402,380 camera frames with human activities under various scenes.
• Score: 9.663978351279422
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human silhouette segmentation, which is originally defined in computer vision, has achieved promising results for understanding human activities. However, the physical limitation makes existing systems based on optical cameras suffer from severe performance degradation under low illumination, smoke, and/or opaque obstruction conditions. To overcome such limitations, in this paper, we propose to utilize the radio signals, which can traverse obstacles and are unaffected by the lighting conditions to achieve silhouette segmentation. The proposed RFMask framework is composed of three modules. It first transforms RF signals captured by millimeter wave radar on two planes into spatial domain and suppress interference with the signal processing module. Then, it locates human reflections on RF frames and extract features from surrounding signals with human detection module. Finally, the extracted features from RF frames are aggregated with an attention based mask generation module. To verify our proposed framework, we collect a dataset containing 804,760 radio frames and 402,380 camera frames with human activities under various scenes. Experimental results show that the proposed framework can achieve impressive human silhouette segmentation even under the challenging scenarios(such as low light and occlusion scenarios) where traditional optical-camera-based methods fail. To the best of our knowledge, this is the first investigation towards segmenting human silhouette based on millimeter wave signals. We hope that our work can serve as a baseline and inspire further research that perform vision tasks with radio signals. The dataset and codes will be made in public.

Related papers

• RF Challenge: The Data-Driven Radio Frequency Signal Separation Challenge [66.33067693672696]
  This paper addresses the critical problem of interference rejection in radio-frequency (RF) signals using a novel, data-driven approach. First, we present an insightful signal model that serves as a foundation for developing and analyzing interference rejection algorithms. Second, we introduce the RF Challenge, a publicly available dataset featuring diverse RF signals along with code templates. Third, we propose novel AI-based rejection algorithms, specifically architectures like UNet and WaveNet, and evaluate their performance across eight different signal mixture types.
  arXiv  Detail & Related papers  (2024-09-13T13:53:41Z)
• Enabling Visual Recognition at Radio Frequency [13.399148413043411]
  PanoRadar is a novel RF imaging system that brings RF resolution close to that of LiDAR. Results enable, for the first time, a variety of visual recognition tasks at radio frequency. Our results demonstrate PanoRadar's robust performance across 12 buildings.
  arXiv  Detail & Related papers  (2024-05-29T20:52:59Z)
• Diffusion Model is a Good Pose Estimator from 3D RF-Vision [32.72703340013302]
  Human pose estimation (HPE) from Radio Frequency vision (RF-vision) performs human sensing using RF signals. mmWave radar has emerged as a promising RF-vision sensor, providing radar point clouds by processing RF signals. This work proposes mmDiff, a novel diffusion-based pose estimator tailored for noisy radar data.
  arXiv  Detail & Related papers  (2024-03-24T15:39:52Z)
• Multi-Space Neural Radiance Fields [74.46513422075438]
  Existing Neural Radiance Fields (NeRF) methods suffer from the existence of reflective objects. We propose a multi-space neural radiance field (MS-NeRF) that represents the scene using a group of feature fields in parallel sub-spaces. Our approach significantly outperforms the existing single-space NeRF methods for rendering high-quality scenes.
  arXiv  Detail & Related papers  (2023-05-07T13:11:07Z)
• DensePose From WiFi [86.61881052177228]
  We develop a deep neural network that maps the phase and amplitude of WiFi signals to UV coordinates within 24 human regions. Our model can estimate the dense pose of multiple subjects, with comparable performance to image-based approaches.
  arXiv  Detail & Related papers  (2022-12-31T16:48:43Z)
• HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar [30.51398364813315]
  This paper introduces a novel human pose estimation benchmark, Human Pose with Millimeter Wave Radar (HuPR) This dataset is created using cross-calibrated mmWave radar sensors and a monocular RGB camera for cross-modality training of radar-based human pose estimation.
  arXiv  Detail & Related papers  (2022-10-22T22:28:40Z)
• Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition [56.91538445510214]
  We present a learning algorithm for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset.
  arXiv  Detail & Related papers  (2022-09-15T22:16:52Z)
• Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenario [87.72258480670627]
  Existing face forgery detection methods based on frequency domain find that the GAN forged images have obvious grid-like visual artifacts in the frequency spectrum compared to the real images. This paper proposes a Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) to achieve a more comprehensive spatial-temporal feature representation.
  arXiv  Detail & Related papers  (2022-07-05T09:27:53Z)
• Human Silhouette and Skeleton Video Synthesis through Wi-Fi signals [24.313281453214614]
  This paper presents a novel two-branch generative neural network that effectively maps radio data into visual features. Once trained, the proposed method synthesizes human silhouette and skeleton videos using exclusively Wi-Fi signals.
  arXiv  Detail & Related papers  (2022-03-11T11:40:34Z)
• Wavelet-Based Network For High Dynamic Range Imaging [64.66969585951207]
  Existing methods, such as optical flow based and end-to-end deep learning based solutions, are error-prone either in detail restoration or ghosting artifacts removal. In this work, we propose a novel frequency-guided end-to-end deep neural network (FNet) to conduct HDR fusion in the frequency domain, and Wavelet Transform (DWT) is used to decompose inputs into different frequency bands. The low-frequency signals are used to avoid specific ghosting artifacts, while the high-frequency signals are used for preserving details.
  arXiv  Detail & Related papers  (2021-08-03T12:26:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.