FIRE: Food Image to REcipe generation

• URL: http://arxiv.org/abs/2308.14391v2
• Date: Sun, 12 May 2024 18:45:52 GMT
• Title: FIRE: Food Image to REcipe generation
• Authors: Prateek Chhikara, Dhiraj Chaurasia, Yifan Jiang, Omkar Masur, Filip Ilievski,
• Abstract summary: Food computing aims to develop end-to-end intelligent systems capable of autonomously producing recipe information for a food image. This paper proposes FIRE, a novel methodology tailored to recipe generation in the food computing domain. We showcase two practical applications that can benefit from integrating FIRE with large language model prompting.
• Score: 10.45344523054623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Food computing has emerged as a prominent multidisciplinary field of research in recent years. An ambitious goal of food computing is to develop end-to-end intelligent systems capable of autonomously producing recipe information for a food image. Current image-to-recipe methods are retrieval-based and their success depends heavily on the dataset size and diversity, as well as the quality of learned embeddings. Meanwhile, the emergence of powerful attention-based vision and language models presents a promising avenue for accurate and generalizable recipe generation, which has yet to be extensively explored. This paper proposes FIRE, a novel multimodal methodology tailored to recipe generation in the food computing domain, which generates the food title, ingredients, and cooking instructions based on input food images. FIRE leverages the BLIP model to generate titles, utilizes a Vision Transformer with a decoder for ingredient extraction, and employs the T5 model to generate recipes incorporating titles and ingredients as inputs. We showcase two practical applications that can benefit from integrating FIRE with large language model prompting: recipe customization to fit recipes to user preferences and recipe-to-code transformation to enable automated cooking processes. Our experimental findings validate the efficacy of our proposed approach, underscoring its potential for future advancements and widespread adoption in food computing.

Related papers

• RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models [96.43285670458803]
  Uni-Food is a unified food dataset that comprises over 100,000 images with various food labels. Uni-Food is designed to provide a more holistic approach to food data analysis. We introduce a novel Linear Rectification Mixture of Diverse Experts (RoDE) approach to address the inherent challenges of food-related multitasking.
  arXiv  Detail & Related papers  (2024-07-17T16:49:34Z)
• Deep Image-to-Recipe Translation [0.0]
  Deep Image-to-Recipe Translation aims to bridge the gap between cherished food memories and the art of culinary creation. Our primary objective involves predicting ingredients from a given food image. Our approach emphasizes the importance of metrics such as Intersection over Union (IoU) and F1 score in scenarios where accuracy alone might be misleading.
  arXiv  Detail & Related papers  (2024-07-01T02:33:07Z)
• Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes [6.666528076345153]
  Food computation models have become increasingly popular in assisting people in maintaining healthy eating habits. In this study, we explore the use of generative AI methods to extend current food computation models to include cooking actions. We propose novel aggregation-based generative AI methods, Cook-Gen, that reliably generate cooking actions from recipes.
  arXiv  Detail & Related papers  (2023-06-01T18:49:47Z)
• A Rich Recipe Representation as Plan to Support Expressive Multi Modal Queries on Recipe Content and Preparation Process [24.94173789568803]
  We discuss the construction of a machine-understandable rich recipe representation (R3) R3 is infused with additional knowledge such as information about allergens and images of ingredients. We also present TREAT, a tool for recipe retrieval which uses R3 to perform multi-modal reasoning on the recipe's content.
  arXiv  Detail & Related papers  (2022-03-31T15:29:38Z)
• Learning Program Representations for Food Images and Cooking Recipes [26.054436410924737]
  We propose to represent cooking recipes and food images as cooking programs. A model is trained to learn a joint embedding between recipes and food images via self-supervision. We show that projecting the image-recipe embeddings into programs leads to better cross-modal retrieval results.
  arXiv  Detail & Related papers  (2022-03-30T05:52:41Z)
• Learning Structural Representations for Recipe Generation and Food Retrieval [101.97397967958722]
  We propose a novel framework of Structure-aware Generation Network (SGN) to tackle the food recipe generation task. Our proposed model can produce high-quality and coherent recipes, and achieve the state-of-the-art performance on the benchmark Recipe1M dataset.
  arXiv  Detail & Related papers  (2021-10-04T06:36:31Z)
• Towards Building a Food Knowledge Graph for Internet of Food [66.57235827087092]
  We review the evolution of food knowledge organization, from food classification to food to food knowledge graphs. Food knowledge graphs play an important role in food search and Question Answering (QA), personalized dietary recommendation, food analysis and visualization. Future directions for food knowledge graphs cover several fields such as multimodal food knowledge graphs and food intelligence.
  arXiv  Detail & Related papers  (2021-07-13T06:26:53Z)
• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning [17.42688184238741]
  Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives. We propose a simplified end-to-end model based on well established and high performing encoders for text and images. Our proposed method achieves state-of-the-art performance in the cross-modal recipe retrieval task on the Recipe1M dataset.
  arXiv  Detail & Related papers  (2021-03-24T10:17:09Z)
• Structure-Aware Generation Network for Recipe Generation from Images [142.047662926209]
  We investigate an open research task of generating cooking instructions based on only food images and ingredients. Target recipes are long-length paragraphs and do not have annotations on structure information. We propose a novel framework of Structure-aware Generation Network (SGN) to tackle the food recipe generation task.
  arXiv  Detail & Related papers  (2020-09-02T10:54:25Z)
• Multi-modal Cooking Workflow Construction for Food Recipes [147.4435186953995]
  We build MM-ReS, the first large-scale dataset for cooking workflow construction. We propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow.
  arXiv  Detail & Related papers  (2020-08-20T18:31:25Z)
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
  We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
  arXiv  Detail & Related papers  (2020-03-09T07:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.