NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis

• URL: http://arxiv.org/abs/2606.08948v1
• Date: Mon, 08 Jun 2026 02:50:08 GMT
• Title: NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis
• Abstract summary: We show that existing multimodal large language models (MLLMs) are unreliable for comprehensive micronutrient estimation.<n>To address this gap without costly expert annotation, we repurposed a decade of 24-hour dietary recalls as structured prompts for text-to-image generation.<n>This pipeline produced a synthetic corpus of about 1.1 million image-description-nutrient triplets, each pairing a generated food image with a complete 65-nutrient label.
• Score: 4.334404218740802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Comprehensive estimation of dietary micronutrients from food images could improve clinical nutrition care, but training such models requires large multimodal datasets linking diverse foods to complete nutrient profiles. We first show that existing multimodal large language models (MLLMs), including leading proprietary models, are unreliable for this task. Across five model families and four independent evaluation benchmarks (ASA24, SNAPMe, FNDDS, and NutriBench), models frequently abstained or returned statistically implausible values. To address this gap without costly expert annotation, we repurposed a decade of population-scale 24-hour dietary recalls as structured prompts for text-to-image generation. This pipeline produced a synthetic corpus of about 1.1 million image-description-nutrient triplets, each pairing a generated food image with a complete 65-nutrient label. To our knowledge, this is the largest synthetic food-image corpus with comprehensive micronutrient annotation planned for public release upon publication. Fine-tuning Qwen3-VL (2B/4B/8B/30B) and GLM-4.6V-Flash on this corpus yielded NutriMLLM, the first family of vision-language models specialized for comprehensive dietary micronutrient estimation. We evaluate these models with a four-component framework that separately measures abstention, hallucination, overall usability, and per-nutrient numerical accuracy. On real food images, every NutriMLLM variant achieved near-complete coverage across all 65 nutrients, and the largest variant matched or exceeded proprietary baselines (GPT-5, Gemini 3, and Claude Sonnet 4.5) in accuracy on most nutrients. These results show that recall-driven synthetic supervision can make image-based comprehensive micronutrient estimation a tractable engineering problem and support dietary assessment, personalized nutrition guidance, and population-scale micronutrient surveillance.

Related papers

• OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion [32.43875223470578]
  Most existing food datasets primarily focus on Western cuisines and lack sufficient coverage of Chinese dishes.<n>We introduce OmniFood8K, a comprehensive multimodal dataset comprising 8,036 food samples.<n>We propose an end-to-end framework for nutritional prediction from a single RGB image.
  arXiv  Detail & Related papers  (2026-04-14T06:45:32Z)
• Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata [16.03960240895014]
  Large Multimodal Models (LMMs) are increasingly applied to meal images for nutrition analysis.<n>This work investigates how interpreting contextual metadata can enhance LMM performance in estimating key nutritional values.<n> Empirical results show that integrating metadata intelligently, when applied through straightforward prompting strategies, can significantly reduce the Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) in predicted nutritional values.
  arXiv  Detail & Related papers  (2025-07-09T17:10:33Z)
• Multimodal Fusion of Glucose Monitoring and Food Imagery for Caloric Content Prediction [2.189594222851135]
  We introduce a multimodal deep learning framework that jointly leverages CGM time-series data, Demographic/Microbiome, and pre-meal food images to enhance caloric estimation.<n>Our model achieves a Root Mean Squared Relative Error (RMSRE) of 0.2544, outperforming the baselines models by over 50%.
  arXiv  Detail & Related papers  (2025-05-13T23:12:54Z)
• Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion [69.84988999191343]
  We introduce FastFood, a dataset with 84,446 images across 908 fast food categories, featuring ingredient and nutritional annotations.<n>We propose a new model-agnostic Visual-Ingredient Feature Fusion (VIF$2$) method to enhance nutrition estimation.
  arXiv  Detail & Related papers  (2025-05-13T17:01:21Z)
• NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptions [6.223619389512576]
  NutriBench is the first publicly available natural language meal description nutrition benchmark.<n>It consists of 11,857 meal descriptions generated from real-world global dietary intake data.<n>The data is human-verified and annotated with macro-nutrient labels, including carbohydrates, proteins, fats, and calories.
  arXiv  Detail & Related papers  (2024-07-04T15:10:51Z)
• NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images [63.314702537010355]
  Self-reporting methods are often inaccurate and suffer from substantial bias. Recent work has explored using computer vision prediction systems to predict nutritional information from food images. This paper aims to enhance the efficacy of dietary intake estimation by leveraging various neural network architectures.
  arXiv  Detail & Related papers  (2024-05-13T14:56:55Z)
• NutritionVerse-Real: An Open Access Manually Collected 2D Food Scene Dataset for Dietary Intake Estimation [68.49526750115429]
  We introduce NutritionVerse-Real, an open access manually collected 2D food scene dataset for dietary intake estimation. The NutritionVerse-Real dataset was created by manually collecting images of food scenes in real life, measuring the weight of every ingredient and computing the associated dietary content of each dish.
  arXiv  Detail & Related papers  (2023-11-20T11:05:20Z)
• NutritionVerse: Empirical Study of Various Dietary Intake Estimation Approaches [59.38343165508926]
  Accurate dietary intake estimation is critical for informing policies and programs to support healthy eating. Recent work has focused on using computer vision and machine learning to automatically estimate dietary intake from food images. We introduce NutritionVerse- Synth, the first large-scale dataset of 84,984 synthetic 2D food images with associated dietary information. We also collect a real image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to evaluate realism.
  arXiv  Detail & Related papers  (2023-09-14T13:29:41Z)
• UMDFood: Vision-language models boost food composition compilation [26.5694236976957]
  We propose a novel vision-language model, UMDFood-VL, using front-of-package labeling and product images to accurately estimate food composition profiles. Up to 82.2% of selected products' estimated error between chemical analysis results and model estimation results are less than 10%. This performance sheds light on generalization towards other food and nutrition-related data compilation and catalyzation.
  arXiv  Detail & Related papers  (2023-05-18T03:18:12Z)
• NutritionVerse-3D: A 3D Food Model Dataset for Nutritional Intake Estimation [65.47310907481042]
  One in four older adults are malnourished. Machine learning and computer vision show promise of automated nutrition tracking methods of food. NutritionVerse-3D is a large-scale high-resolution dataset of 105 3D food models.
  arXiv  Detail & Related papers  (2023-04-12T05:27:30Z)
• Towards the Creation of a Nutrition and Food Group Based Image Database [58.429385707376554]
  We propose a framework to create a nutrition and food group based image database. We design a protocol for linking food group based food codes in the U.S. Department of Agriculture's (USDA) Food and Nutrient Database for Dietary Studies (FNDDS) Our proposed method is used to build a nutrition and food group based image database including 16,114 food datasets.
  arXiv  Detail & Related papers  (2022-06-05T02:41:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.