Multimodal Fusion of Glucose Monitoring and Food Imagery for Caloric Content Prediction

• URL: http://arxiv.org/abs/2505.09018v2
• Date: Tue, 20 May 2025 15:25:23 GMT
• Title: Multimodal Fusion of Glucose Monitoring and Food Imagery for Caloric Content Prediction
• Authors: Adarsh Kumar,
• Abstract summary: We introduce a multimodal deep learning framework that jointly leverages CGM time-series data, Demographic/Microbiome, and pre-meal food images to enhance caloric estimation.<n>Our model achieves a Root Mean Squared Relative Error (RMSRE) of 0.2544, outperforming the baselines models by over 50%.
• Score: 2.189594222851135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective dietary monitoring is critical for managing Type 2 diabetes, yet accurately estimating caloric intake remains a major challenge. While continuous glucose monitors (CGMs) offer valuable physiological data, they often fall short in capturing the full nutritional profile of meals due to inter-individual and meal-specific variability. In this work, we introduce a multimodal deep learning framework that jointly leverages CGM time-series data, Demographic/Microbiome, and pre-meal food images to enhance caloric estimation. Our model utilizes attention based encoding and a convolutional feature extraction for meal imagery, multi-layer perceptrons for CGM and Microbiome data followed by a late fusion strategy for joint reasoning. We evaluate our approach on a curated dataset of over 40 participants, incorporating synchronized CGM, Demographic and Microbiome data and meal photographs with standardized caloric labels. Our model achieves a Root Mean Squared Relative Error (RMSRE) of 0.2544, outperforming the baselines models by over 50%. These findings demonstrate the potential of multimodal sensing to improve automated dietary assessment tools for chronic disease management.

Related papers

• Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion [69.84988999191343]
  We introduce FastFood, a dataset with 84,446 images across 908 fast food categories, featuring ingredient and nutritional annotations.<n>We propose a new model-agnostic Visual-Ingredient Feature Fusion (VIF$2$) method to enhance nutrition estimation.
  arXiv  Detail & Related papers  (2025-05-13T17:01:21Z)
• AttenGluco: Multimodal Transformer-Based Blood Glucose Forecasting on AI-READI Dataset [8.063401183752347]
  Diabetes is a chronic metabolic disorder characterized by persistently high blood glucose levels (BGLs)<n>Recent deep learning models show promise in improving BGL prediction.<n>We propose AttenGluco, a multimodal Transformer-based framework for long-term blood glucose prediction.
  arXiv  Detail & Related papers  (2025-02-14T05:07:38Z)
• From Glucose Patterns to Health Outcomes: A Generalizable Foundation Model for Continuous Glucose Monitor Data Analysis [47.23780364438969]
  We present GluFormer, a generative foundation model for CGM data that learns nuanced glycemic patterns and translates them into predictive representations of metabolic health.<n>GluFormer generalizes to 19 external cohorts spanning different ethnicities and ages, 5 countries, 8 CGM devices, and diverse pathophysiological states.<n>In a longitudinal study of 580 adults with CGM data and 12-year follow-up, GluFormer identifies individuals at elevated risk of developing diabetes more effectively than blood HbA1C%.
  arXiv  Detail & Related papers  (2024-08-20T13:19:06Z)
• NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images [63.314702537010355]
  Self-reporting methods are often inaccurate and suffer from substantial bias. Recent work has explored using computer vision prediction systems to predict nutritional information from food images. This paper aims to enhance the efficacy of dietary intake estimation by leveraging various neural network architectures.
  arXiv  Detail & Related papers  (2024-05-13T14:56:55Z)
• Interpretable Mechanistic Representations for Meal-level Glycemic Control in the Wild [10.240619571788786]
  We propose a hybrid variational autoencoder to learn interpretable representations of CGM and meal data. Our method grounds the latent space to the inputs of a mechanistic differential equation, producing embeddings that reflect physiological quantities. Our embeddings produce clusters that are up to 4x better than naive, expert, black-box, and pure mechanistic features.
  arXiv  Detail & Related papers  (2023-12-06T08:36:23Z)
• NutritionVerse-Real: An Open Access Manually Collected 2D Food Scene Dataset for Dietary Intake Estimation [68.49526750115429]
  We introduce NutritionVerse-Real, an open access manually collected 2D food scene dataset for dietary intake estimation. The NutritionVerse-Real dataset was created by manually collecting images of food scenes in real life, measuring the weight of every ingredient and computing the associated dietary content of each dish.
  arXiv  Detail & Related papers  (2023-11-20T11:05:20Z)
• NutritionVerse: Empirical Study of Various Dietary Intake Estimation Approaches [59.38343165508926]
  Accurate dietary intake estimation is critical for informing policies and programs to support healthy eating. Recent work has focused on using computer vision and machine learning to automatically estimate dietary intake from food images. We introduce NutritionVerse- Synth, the first large-scale dataset of 84,984 synthetic 2D food images with associated dietary information. We also collect a real image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to evaluate realism.
  arXiv  Detail & Related papers  (2023-09-14T13:29:41Z)
• Predicting the meal macronutrient composition from continuous glucose monitors [16.911400979837417]
  Dietary intake is an essential component of clinical interventions for type 2 diabetes (T2DM) Current techniques to monitor food intake are time intensive and error prone. We are developing techniques to automatically monitor food intake and the composition of those foods using continuous glucose monitors (CGMs)
  arXiv  Detail & Related papers  (2022-06-23T17:41:25Z)
• Enhancing Food Intake Tracking in Long-Term Care with Automated Food Imaging and Nutrient Intake Tracking (AFINI-T) Technology [71.37011431958805]
  Half of long-term care (LTC) residents are malnourished increasing hospitalization, mortality, morbidity, with lower quality of life. This paper presents the automated food imaging and nutrient intake tracking (AFINI-T) technology designed for LTC.
  arXiv  Detail & Related papers  (2021-12-08T22:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.