Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning

• URL: http://arxiv.org/abs/2606.04986v1
• Date: Wed, 03 Jun 2026 15:07:12 GMT
• Title: Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning
• Abstract summary: We introduce CalorieBench-80K, a large-scale benchmark with curated calorie labels and dietary advice annotations.<n>To the best of our knowledge, it is the first food image benchmark to incorporate Chain-of-Thought (CoT) annotations for calorie reasoning.<n>We also propose Food-R1, a unified food VLM trained in a multi-task learning paradigm to equip the model with broad capabilities.
• Score: 44.68518637735983
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies have explored Vision-Language Models (VLMs) for food analysis. However, most existing methods rely primarily on supervised fine-tuning (SFT), which often limits reasoning and generalization capabilities. Moreover, high-quality large-scale nutritional annotations remain scarce. To address these issues, we introduce CalorieBench-80K, a large-scale benchmark with curated calorie labels and dietary advice annotations. To the best of our knowledge, it is the first food image benchmark to incorporate Chain-of-Thought (CoT) annotations for calorie reasoning. We also propose Food-R1, a unified food VLM trained in a multi-task learning paradigm to equip the model with broad capabilities. Food-R1 undergoes CoT-based cold-start instruction tuning, followed by reinforcement fine-tuning (RFT) using Group Relative Policy Optimization (GRPO) to improve reasoning and performance. Experiments on CalorieBench-80K and representative benchmarks show that Food-R1 consistently outperforms strong baselines across food-related tasks. The code, model weights, and benchmark annotations are available at the project repository.

Related papers

• OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion [32.43875223470578]
  Most existing food datasets primarily focus on Western cuisines and lack sufficient coverage of Chinese dishes.<n>We introduce OmniFood8K, a comprehensive multimodal dataset comprising 8,036 food samples.<n>We propose an end-to-end framework for nutritional prediction from a single RGB image.
  arXiv  Detail & Related papers  (2026-04-14T06:45:32Z)
• MultiFoodhat: A potential new paradigm for intelligent food quality inspection [7.966483944010341]
  MultiFoodChat is a dialogue-driven multi-agent reasoning framework for zero-shot food recognition.<n>An Object Perception Token (OPT) captures fine-grained visual attributes, while an Interactive Reasoning Agent (IRA) dynamically interprets contextual cues to refine predictions.<n>Experiments on multiple public food datasets demonstrate that MultiFoodChat achieves superior recognition accuracy and interpretability compared with existing unsupervised and few-shot methods.
  arXiv  Detail & Related papers  (2025-10-14T03:39:03Z)
• FoodSEM: Large Language Model Specialized in Food Named-Entity Linking [4.7696209609483]
  FoodSEM is a state-of-the-art fine-tuned large model for named-entity linking (NEL) to food-related open-source language models.<n>FoodSEM achieves model state-of-the-art performance compared to related models/systems, with F1 scores even reaching 98% on some datasets.
  arXiv  Detail & Related papers  (2025-09-26T09:47:35Z)
• RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models [96.43285670458803]
  Uni-Food is a unified food dataset that comprises over 100,000 images with various food labels.<n>Uni-Food is designed to provide a more holistic approach to food data analysis.<n>We introduce a novel Linear Rectification Mixture of Diverse Experts (RoDE) approach to address the inherent challenges of food-related multitasking.
  arXiv  Detail & Related papers  (2024-07-17T16:49:34Z)
• From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios [92.58097090916166]
  We present two new benchmarks, namely DailyFood-172 and DailyFood-16, designed to curate food images from everyday meals. These two datasets are used to evaluate the transferability of approaches from the well-curated food image domain to the everyday-life food image domain.
  arXiv  Detail & Related papers  (2024-03-12T08:32:23Z)
• Food Recommendation as Language Processing (F-RLP): A Personalized and Contextual Paradigm [3.0057566758557974]
  We introduce Food Recommendation as Language Processing (F-RLP), a novel framework that offers a food-specific, tailored infrastructure. F-RLP leverages the capabilities of Large Language Models (LLMs) to maximize their potential, thereby paving the way for more accurate, personalized food recommendations.
  arXiv  Detail & Related papers  (2024-02-12T08:32:29Z)
• FoodLMM: A Versatile Food Assistant using Large Multi-modal Model [96.76271649854542]
  Large Multi-modal Models (LMMs) have made impressive progress in many vision-language tasks. This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities. We introduce a series of novel task-specific tokens and heads, enabling the model to predict food nutritional values and multiple segmentation masks.
  arXiv  Detail & Related papers  (2023-12-22T11:56:22Z)
• Single-Stage Heavy-Tailed Food Classification [7.800379384628357]
  We introduce a novel single-stage heavy-tailed food classification framework. Our method is evaluated on two heavy-tailed food benchmark datasets, Food101-LT and VFN-LT.
  arXiv  Detail & Related papers  (2023-07-01T00:45:35Z)
• Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions [65.50975507723827]
  Food image segmentation is an important task that has ubiquitous applications, such as estimating the nutritional value of a plate of food. One challenge is that food items can overlap and mix, making them difficult to distinguish. Two models are trained and compared, one based on convolutional neural networks and the other on Bidirectional representation for Image Transformers (BEiT) The BEiT model outperforms the previous state-of-the-art model by achieving a mean intersection over union of 49.4 on FoodSeg103.
  arXiv  Detail & Related papers  (2023-06-15T15:38:10Z)
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
  We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
  arXiv  Detail & Related papers  (2020-03-09T07:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.