FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination

• URL: http://arxiv.org/abs/2406.10261v1
• Date: Tue, 11 Jun 2024 01:27:00 GMT
• Title: FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination
• Authors: Pengfei Zhou, Weiqing Min, Chaoran Fu, Ying Jin, Mingyu Huang, Xiangyang Li, Shuhuan Mei, Shuqiang Jiang,
• Abstract summary: We introduce Food-oriented Large Language Models (LLMs) to comprehend food data through perception and reasoning. Considering the complexity and typicality of Chinese cuisine, we first construct one comprehensive Chinese food corpus FoodEarth. We then propose Topic-based Selective State Space Model (TS3M) and the Hierarchical Topic Retrieval Augmented Generation (HTRAG) mechanism to enhance FoodSky.
• Score: 37.11551779015218
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Food is foundational to human life, serving not only as a source of nourishment but also as a cornerstone of cultural identity and social interaction. As the complexity of global dietary needs and preferences grows, food intelligence is needed to enable food perception and reasoning for various tasks, ranging from recipe generation and dietary recommendation to diet-disease correlation discovery and understanding. Towards this goal, for powerful capabilities across various domains and tasks in Large Language Models (LLMs), we introduce Food-oriented LLM FoodSky to comprehend food data through perception and reasoning. Considering the complexity and typicality of Chinese cuisine, we first construct one comprehensive Chinese food corpus FoodEarth from various authoritative sources, which can be leveraged by FoodSky to achieve deep understanding of food-related data. We then propose Topic-based Selective State Space Model (TS3M) and the Hierarchical Topic Retrieval Augmented Generation (HTRAG) mechanism to enhance FoodSky in capturing fine-grained food semantics and generating context-aware food-relevant text, respectively. Our extensive evaluations demonstrate that FoodSky significantly outperforms general-purpose LLMs in both chef and dietetic examinations, with an accuracy of 67.2% and 66.4% on the Chinese National Chef Exam and the National Dietetic Exam, respectively. FoodSky not only promises to enhance culinary creativity and promote healthier eating patterns, but also sets a new standard for domain-specific LLMs that address complex real-world issues in the food domain. An online demonstration of FoodSky is available at http://222.92.101.211:8200.

Related papers

• FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture [60.51749998013166]
  We introduce FoodieQA, a fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China. We evaluate vision-language Models (VLMs) and large language models (LLMs) on newly collected, unseen food images and corresponding questions. Our findings highlight that understanding food and its cultural implications remains a challenging and under-explored direction.
  arXiv  Detail & Related papers  (2024-06-16T17:59:32Z)
• NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images [63.314702537010355]
  Self-reporting methods are often inaccurate and suffer from substantial bias. Recent work has explored using computer vision prediction systems to predict nutritional information from food images. This paper aims to enhance the efficacy of dietary intake estimation by leveraging various neural network architectures.
  arXiv  Detail & Related papers  (2024-05-13T14:56:55Z)
• NutritionVerse-3D: A 3D Food Model Dataset for Nutritional Intake Estimation [65.47310907481042]
  One in four older adults are malnourished. Machine learning and computer vision show promise of automated nutrition tracking methods of food. NutritionVerse-3D is a large-scale high-resolution dataset of 105 3D food models.
  arXiv  Detail & Related papers  (2023-04-12T05:27:30Z)
• Leveraging Automatic Personalised Nutrition: Food Image Recognition Benchmark and Dataset based on Nutrition Taxonomy [14.569887684034061]
  This study presents the first nutrition database that incorporates food images and a nutrition taxonomy based on recommendations by national and international health authorities. The AI4Food-NutritionDB opens the doors to new food computing approaches in terms of food intake frequency, quality, and categorisation.
  arXiv  Detail & Related papers  (2022-11-14T15:14:50Z)
• MenuAI: Restaurant Food Recommendation System via a Transformer-based Deep Learning Model [15.248362664235845]
  A novel restaurant food recommendation system is proposed in this paper. It uses Optical Character Recognition (OCR) technology and a transformer-based deep learning model, Learning to Rank (LTR) model. Our system is able to rank the food dishes in terms of the input search key (e.g., calorie, protein level)
  arXiv  Detail & Related papers  (2022-10-15T11:45:44Z)
• Towards the Creation of a Nutrition and Food Group Based Image Database [58.429385707376554]
  We propose a framework to create a nutrition and food group based image database. We design a protocol for linking food group based food codes in the U.S. Department of Agriculture's (USDA) Food and Nutrient Database for Dietary Studies (FNDDS) Our proposed method is used to build a nutrition and food group based image database including 16,114 food datasets.
  arXiv  Detail & Related papers  (2022-06-05T02:41:44Z)
• Towards Building a Food Knowledge Graph for Internet of Food [66.57235827087092]
  We review the evolution of food knowledge organization, from food classification to food to food knowledge graphs. Food knowledge graphs play an important role in food search and Question Answering (QA), personalized dietary recommendation, food analysis and visualization. Future directions for food knowledge graphs cover several fields such as multimodal food knowledge graphs and food intelligence.
  arXiv  Detail & Related papers  (2021-07-13T06:26:53Z)
• Large Scale Visual Food Recognition [43.43598316339732]
  We introduce Food2K, which is the largest food recognition dataset with 2,000 categories and over 1 million images. Food2K bypasses them in both categories and images by one order of magnitude. We propose a deep progressive region enhancement network for food recognition.
  arXiv  Detail & Related papers  (2021-03-30T06:41:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.