FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis

• URL: http://arxiv.org/abs/2605.05499v1
• Date: Wed, 06 May 2026 22:46:59 GMT
• Title: FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis
• Abstract summary: Real-world food images present challenges due to high intra-class similarity.<n>Deep learning models achieve strong performance in coarse grained classification.<n>Open-ended generation in modern vision-language models can produce non-canonical labels.<n>We propose FoodCHA, a multimodal agentic framework that reformulates food recognition as a hierarchical decision-making process.
• Score: 15.616960241461934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread adoption of camera-equipped mobile devices and wearables has enabled convenient capture of meal images, making food recognition a key component for real time dietary monitoring. However, real-world food images present challenges due to high intra-class similarity and the frequent presence of multiple food items within a single image. While deep learning models achieve strong performance in coarse grained classification, they often struggle to capture fine-grained attributes such as cooking style. Moreover, open-ended generation in modern vision-language models can produce non-canonical labels, limiting their practical deployment. We propose FoodCHA, a multimodal agentic framework that reformulates food recognition as a hierarchical decision-making process. By progressively anchoring predictions, FoodCHA guides subcategory identification using high-level categories and guides cooking style recognition using subcategories, improving semantic consistency and attribute-level discrimination. To ensure practical deployability, FoodCHA utilizes the compact Moondream-2B vision language model, which provides strong reasoning capability while maintaining lower computational and memory overhead. Experiments on FoodNExTDB show that FoodCHA outperforms Food-Llama-3.2-11B by 13.8% and 38.2% in category and subcategory recognition precision, respectively, and achieves a striking 153.2% improvement in cooking style classification precision.

Related papers

• LLMs-based Augmentation for Domain Adaptation in Long-tailed Food Datasets [54.527878056610156]
  We present a framework empowered with large language models (LLMs) to address these challenges in food recognition.<n>We first leverage LLMs to parse food images to generate food titles and ingredients.<n>Then, we project the generated texts and food images from different domains to a shared embedding space to maximize the pair similarities.
  arXiv  Detail & Related papers  (2025-11-20T04:38:56Z)
• MultiFoodhat: A potential new paradigm for intelligent food quality inspection [7.966483944010341]
  MultiFoodChat is a dialogue-driven multi-agent reasoning framework for zero-shot food recognition.<n>An Object Perception Token (OPT) captures fine-grained visual attributes, while an Interactive Reasoning Agent (IRA) dynamically interprets contextual cues to refine predictions.<n>Experiments on multiple public food datasets demonstrate that MultiFoodChat achieves superior recognition accuracy and interpretability compared with existing unsupervised and few-shot methods.
  arXiv  Detail & Related papers  (2025-10-14T03:39:03Z)
• From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios [92.58097090916166]
  We present two new benchmarks, namely DailyFood-172 and DailyFood-16, designed to curate food images from everyday meals. These two datasets are used to evaluate the transferability of approaches from the well-curated food image domain to the everyday-life food image domain.
  arXiv  Detail & Related papers  (2024-03-12T08:32:23Z)
• FoodLMM: A Versatile Food Assistant using Large Multi-modal Model [96.76271649854542]
  Large Multi-modal Models (LMMs) have made impressive progress in many vision-language tasks. This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities. We introduce a series of novel task-specific tokens and heads, enabling the model to predict food nutritional values and multiple segmentation masks.
  arXiv  Detail & Related papers  (2023-12-22T11:56:22Z)
• Food Image Classification and Segmentation with Attention-based Multiple Instance Learning [51.279800092581844]
  The paper presents a weakly supervised methodology for training food image classification and semantic segmentation models. The proposed methodology is based on a multiple instance learning approach in combination with an attention-based mechanism. We conduct experiments on two meta-classes within the FoodSeg103 data set to verify the feasibility of the proposed approach.
  arXiv  Detail & Related papers  (2023-08-22T13:59:47Z)
• Single-Stage Heavy-Tailed Food Classification [7.800379384628357]
  We introduce a novel single-stage heavy-tailed food classification framework. Our method is evaluated on two heavy-tailed food benchmark datasets, Food101-LT and VFN-LT.
  arXiv  Detail & Related papers  (2023-07-01T00:45:35Z)
• Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions [65.50975507723827]
  Food image segmentation is an important task that has ubiquitous applications, such as estimating the nutritional value of a plate of food. One challenge is that food items can overlap and mix, making them difficult to distinguish. Two models are trained and compared, one based on convolutional neural networks and the other on Bidirectional representation for Image Transformers (BEiT) The BEiT model outperforms the previous state-of-the-art model by achieving a mean intersection over union of 49.4 on FoodSeg103.
  arXiv  Detail & Related papers  (2023-06-15T15:38:10Z)
• A Mobile Food Recognition System for Dietary Assessment [6.982738885923204]
  We focus on developing a mobile friendly, Middle Eastern cuisine focused food recognition application for assisted living purposes. Using Mobilenet-v2 architecture for this task is beneficial in terms of both accuracy and the memory usage. The developed mobile application has potential to serve the visually impaired in automatic food recognition via images.
  arXiv  Detail & Related papers  (2022-04-20T12:49:36Z)
• Large Scale Visual Food Recognition [43.43598316339732]
  We introduce Food2K, which is the largest food recognition dataset with 2,000 categories and over 1 million images. Food2K bypasses them in both categories and images by one order of magnitude. We propose a deep progressive region enhancement network for food recognition.
  arXiv  Detail & Related papers  (2021-03-30T06:41:42Z)
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
  We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
  arXiv  Detail & Related papers  (2020-03-09T07:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.