Muti-Stage Hierarchical Food Classification
- URL: http://arxiv.org/abs/2309.01075v1
- Date: Sun, 3 Sep 2023 04:45:44 GMT
- Title: Muti-Stage Hierarchical Food Classification
- Authors: Xinyue Pan, Jiangpeng He, Fengqing Zhu
- Abstract summary: We propose a multi-stage hierarchical framework for food item classification by iteratively clustering and merging food items during the training process.
Our method is evaluated on VFN-nutrient dataset and achieve promising results compared with existing work in terms of both food type and food item classification.
- Score: 9.013592803864086
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Food image classification serves as a fundamental and critical step in
image-based dietary assessment, facilitating nutrient intake analysis from
captured food images. However, existing works in food classification
predominantly focuses on predicting 'food types', which do not contain direct
nutritional composition information. This limitation arises from the inherent
discrepancies in nutrition databases, which are tasked with associating each
'food item' with its respective information. Therefore, in this work we aim to
classify food items to align with nutrition database. To this end, we first
introduce VFN-nutrient dataset by annotating each food image in VFN with a food
item that includes nutritional composition information. Such annotation of food
items, being more discriminative than food types, creates a hierarchical
structure within the dataset. However, since the food item annotations are
solely based on nutritional composition information, they do not always show
visual relations with each other, which poses significant challenges when
applying deep learning-based techniques for classification. To address this
issue, we then propose a multi-stage hierarchical framework for food item
classification by iteratively clustering and merging food items during the
training process, which allows the deep model to extract image features that
are discriminative across labels. Our method is evaluated on VFN-nutrient
dataset and achieve promising results compared with existing work in terms of
both food type and food item classification.
Related papers
- NutritionVerse-Real: An Open Access Manually Collected 2D Food Scene
Dataset for Dietary Intake Estimation [68.49526750115429]
We introduce NutritionVerse-Real, an open access manually collected 2D food scene dataset for dietary intake estimation.
The NutritionVerse-Real dataset was created by manually collecting images of food scenes in real life, measuring the weight of every ingredient and computing the associated dietary content of each dish.
arXiv Detail & Related papers (2023-11-20T11:05:20Z) - NutritionVerse: Empirical Study of Various Dietary Intake Estimation Approaches [59.38343165508926]
Accurate dietary intake estimation is critical for informing policies and programs to support healthy eating.
Recent work has focused on using computer vision and machine learning to automatically estimate dietary intake from food images.
We introduce NutritionVerse- Synth, the first large-scale dataset of 84,984 synthetic 2D food images with associated dietary information.
We also collect a real image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to evaluate realism.
arXiv Detail & Related papers (2023-09-14T13:29:41Z) - Transferring Knowledge for Food Image Segmentation using Transformers
and Convolutions [65.50975507723827]
Food image segmentation is an important task that has ubiquitous applications, such as estimating the nutritional value of a plate of food.
One challenge is that food items can overlap and mix, making them difficult to distinguish.
Two models are trained and compared, one based on convolutional neural networks and the other on Bidirectional representation for Image Transformers (BEiT)
The BEiT model outperforms the previous state-of-the-art model by achieving a mean intersection over union of 49.4 on FoodSeg103.
arXiv Detail & Related papers (2023-06-15T15:38:10Z) - Towards the Creation of a Nutrition and Food Group Based Image Database [58.429385707376554]
We propose a framework to create a nutrition and food group based image database.
We design a protocol for linking food group based food codes in the U.S. Department of Agriculture's (USDA) Food and Nutrient Database for Dietary Studies (FNDDS)
Our proposed method is used to build a nutrition and food group based image database including 16,114 food datasets.
arXiv Detail & Related papers (2022-06-05T02:41:44Z) - Improving Dietary Assessment Via Integrated Hierarchy Food
Classification [7.398060062678395]
We introduce a new food classification framework to improve the quality of predictions by integrating the information from multiple domains.
Our method is validated on the modified VIPER-FoodNet (VFN) food image dataset by including associated energy and nutrient information.
arXiv Detail & Related papers (2021-09-06T20:59:58Z) - A Large-Scale Benchmark for Food Image Segmentation [62.28029856051079]
We build a new food image dataset FoodSeg103 (and its extension FoodSeg154) containing 9,490 images.
We annotate these images with 154 ingredient classes and each image has an average of 6 ingredient labels and pixel-wise masks.
We propose a multi-modality pre-training approach called ReLeM that explicitly equips a segmentation model with rich and semantic food knowledge.
arXiv Detail & Related papers (2021-05-12T03:00:07Z) - Saliency-Aware Class-Agnostic Food Image Segmentation [10.664526852464812]
We propose a class-agnostic food image segmentation method.
Using information from both the before and after eating images, we can segment food images by finding the salient missing objects.
Our method is validated on food images collected from a dietary study.
arXiv Detail & Related papers (2021-02-13T08:05:19Z) - An End-to-End Food Image Analysis System [8.622335099019214]
We propose an image-based food analysis framework that integrates food localization, classification and portion size estimation.
Our proposed framework is end-to-end, i.e., the input can be an arbitrary food image containing multiple food items.
Our framework is evaluated on a real life food image dataset collected from a nutrition feeding study.
arXiv Detail & Related papers (2021-02-01T05:36:20Z) - Visual Aware Hierarchy Based Food Recognition [10.194167945992938]
We propose a new two-step food recognition system using Convolutional Neural Networks (CNNs) as the backbone architecture.
The food localization step is based on an implementation of the Faster R-CNN method to identify food regions.
In the food classification step, visually similar food categories can be clustered together automatically to generate a hierarchical structure.
arXiv Detail & Related papers (2020-12-06T20:25:31Z) - Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images
and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another.
We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities.
We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
arXiv Detail & Related papers (2020-03-09T07:41:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.