Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention

• URL: http://arxiv.org/abs/2211.04253v2
• Date: Mon, 18 Dec 2023 14:52:42 GMT
• Title: Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network with Attention
• Abstract summary: We propose a novel contactless radar-based approach for food intake monitoring. A Frequency Modulated Continuous Wave (FMCW) radar sensor is employed to recognize fine-grained eating and drinking gestures. A 3D temporal convolutional network with self-attention (3D-TCN-Att) is developed to detect and segment eating and drinking gestures in meal sessions.
• Score: 1.8932585560127178
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unhealthy dietary habits are considered as the primary cause of various chronic diseases, including obesity and diabetes. The automatic food intake monitoring system has the potential to improve the quality of life (QoL) of people with diet-related diseases through dietary assessment. In this work, we propose a novel contactless radar-based approach for food intake monitoring. Specifically, a Frequency Modulated Continuous Wave (FMCW) radar sensor is employed to recognize fine-grained eating and drinking gestures. The fine-grained eating/drinking gesture contains a series of movements from raising the hand to the mouth until putting away the hand from the mouth. A 3D temporal convolutional network with self-attention (3D-TCN-Att) is developed to detect and segment eating and drinking gestures in meal sessions by processing the Range-Doppler Cube (RD Cube). Unlike previous radar-based research, this work collects data in continuous meal sessions (more realistic scenarios). We create a public dataset comprising 70 meal sessions (4,132 eating gestures and 893 drinking gestures) from 70 participants with a total duration of 1,155 minutes. Four eating styles (fork & knife, chopsticks, spoon, hand) are included in this dataset. To validate the performance of the proposed approach, seven-fold cross-validation method is applied. The 3D-TCN-Att model achieves a segmental F1-score of 0.896 and 0.868 for eating and drinking gestures, respectively. The results of the proposed approach indicate the feasibility of using radar for fine-grained eating and drinking gesture detection and segmentation in meal sessions.

Related papers

• What's on Your Plate? Inferring Chinese Cuisine Intake from Wearable IMUs [6.064832298655888]
  Existing wearable-based methods primarily focus on a limited number of food types, such as hamburgers and pizza.<n>We propose CuisineSense, a system that classifies Chinese food types by integrating hand motion cues from a smartwatch with head dynamics from smart glasses.<n>Experiments demonstrate that CuisineSense achieves high accuracy in both eating state detection and food classification.
  arXiv  Detail & Related papers  (2025-11-07T14:54:37Z)
• Robust Multimodal Learning Framework For Intake Gesture Detection Using Contactless Radar and Wearable IMU Sensors [1.6180992915701704]
  This study explores whether combining wearable and contactless sensing modalities through multimodal learning can further improve detection performance.<n>We propose a robust multimodal temporal convolutional network with cross-modal attention (MM-TCN-CMA) to integrate IMU and radar data, enhance gesture detection, and maintain performance under missing modality conditions.<n>This is the first study to demonstrate a robust multimodal learning framework that effectively fuses IMU and radar data for food intake gesture detection.
  arXiv  Detail & Related papers  (2025-07-09T20:15:40Z)
• Dietary Intake Estimation via Continuous 3D Reconstruction of Food [5.010690651107531]
  This study proposes an approach to accurately monitor ingest behaviours by leveraging 3D food models constructed from monocular 2D video. Experiments with toy models and real food items demonstrate the approach's potential.
  arXiv  Detail & Related papers  (2025-05-01T15:35:42Z)
• Skeleton-Based Intake Gesture Detection With Spatial-Temporal Graph Convolutional Networks [1.5228527154365612]
  This study introduces a skeleton based approach using a model that combines a dilated spatial-temporal graph convolutional network (ST-GCN) with a bidirectional long-short-term memory (BiLSTM) framework to detect intake gestures. The results confirm the feasibility of utilizing skeleton data for intake gesture detection and highlight the robustness of the proposed approach in cross-dataset validation.
  arXiv  Detail & Related papers  (2025-04-14T18:35:32Z)
• MetaFood3D: 3D Food Dataset with Nutrition Values [52.16894900096017]
  This dataset consists of 743 meticulously scanned and labeled 3D food objects across 131 categories. Our MetaFood3D dataset emphasizes intra-class diversity and includes rich modalities such as textured mesh files, RGB-D videos, and segmentation masks.
  arXiv  Detail & Related papers  (2024-09-03T15:02:52Z)
• NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images [63.314702537010355]
  Self-reporting methods are often inaccurate and suffer from substantial bias. Recent work has explored using computer vision prediction systems to predict nutritional information from food images. This paper aims to enhance the efficacy of dietary intake estimation by leveraging various neural network architectures.
  arXiv  Detail & Related papers  (2024-05-13T14:56:55Z)
• How Much You Ate? Food Portion Estimation on Spoons [63.611551981684244]
  Current image-based food portion estimation algorithms assume that users take images of their meals one or two times. We introduce an innovative solution that utilizes stationary user-facing cameras to track food items on utensils. The system is reliable for estimation of nutritional content of liquid-solid heterogeneous mixtures such as soups and stews.
  arXiv  Detail & Related papers  (2024-05-12T00:16:02Z)
• NutritionVerse-Real: An Open Access Manually Collected 2D Food Scene Dataset for Dietary Intake Estimation [68.49526750115429]
  We introduce NutritionVerse-Real, an open access manually collected 2D food scene dataset for dietary intake estimation. The NutritionVerse-Real dataset was created by manually collecting images of food scenes in real life, measuring the weight of every ingredient and computing the associated dietary content of each dish.
  arXiv  Detail & Related papers  (2023-11-20T11:05:20Z)
• NutritionVerse: Empirical Study of Various Dietary Intake Estimation Approaches [59.38343165508926]
  Accurate dietary intake estimation is critical for informing policies and programs to support healthy eating. Recent work has focused on using computer vision and machine learning to automatically estimate dietary intake from food images. We introduce NutritionVerse- Synth, the first large-scale dataset of 84,984 synthetic 2D food images with associated dietary information. We also collect a real image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to evaluate realism.
  arXiv  Detail & Related papers  (2023-09-14T13:29:41Z)
• NutritionVerse-3D: A 3D Food Model Dataset for Nutritional Intake Estimation [65.47310907481042]
  One in four older adults are malnourished. Machine learning and computer vision show promise of automated nutrition tracking methods of food. NutritionVerse-3D is a large-scale high-resolution dataset of 105 3D food models.
  arXiv  Detail & Related papers  (2023-04-12T05:27:30Z)
• Vision-Based Food Analysis for Automatic Dietary Assessment [49.32348549508578]
  This review presents one unified Vision-Based Dietary Assessment (VBDA) framework, which generally consists of three stages: food image analysis, volume estimation and nutrient derivation. Deep learning makes VBDA gradually move to an end-to-end implementation, which applies food images to a single network to directly estimate the nutrition.
  arXiv  Detail & Related papers  (2021-08-06T05:46:01Z)
• MyFood: A Food Segmentation and Classification System to Aid Nutritional Monitoring [1.5469452301122173]
  The absence of food monitoring has contributed significantly to the increase in the population's weight. Some solutions have been proposed in computer vision to recognize food images, but few are specialized in nutritional monitoring. This work presents the development of an intelligent system that classifies and segments food presented in images to help the automatic monitoring of user diet and nutritional intake.
  arXiv  Detail & Related papers  (2020-12-05T17:40:05Z)
• A Data Driven End-to-end Approach for In-the-wild Monitoring of Eating Behavior Using Smartwatches [8.257740966456172]
  This paper presents a complete framework towards the automated i) modeling of in-meal eating behavior and ii) temporal localization of meals. We present an end-to-end Neural Network which detects food intake events (i.e. bites) We show how the distribution of the detected bites throughout the day can be used to estimate the start and end points of meals, using signal processing algorithms.
  arXiv  Detail & Related papers  (2020-10-12T12:35:56Z)
• Eating Habits Discovery in Egocentric Photo-streams [9.436913961194671]
  We build a food-related behavioural pattern discovery model, which discloses nutritional routines from the activities performed throughout the days. Within this framework, we present a simple, but robust and fast novel classification pipeline. We show an application for the identification of food-related scenes when the camera eats in isolation.
  arXiv  Detail & Related papers  (2020-09-16T12:46:35Z)
• An Artificial Intelligence-Based System to Assess Nutrient Intake for Hospitalised Patients [4.048427587958764]
  Regular monitoring of nutrient intake in hospitalised patients plays a critical role in reducing the risk of disease-related malnutrition. We propose a novel system based on artificial intelligence (AI) to accurately estimate nutrient intake.
  arXiv  Detail & Related papers  (2020-03-18T15:28:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.