CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion

• URL: http://arxiv.org/abs/2501.09042v2
• Date: Sun, 09 Feb 2025 15:33:20 GMT
• Title: CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion
• Authors: Yuan Wang, Bin Zhu, Yanbin Hao, Chong-Wah Ngo, Yi Tan, Xiang Wang,
• Abstract summary: We present textbfCookingDiffusion, a novel approach to generate photo-realistic images of cooking steps. These prompts encompass text prompts, image prompts, and multi-modal prompts, ensuring the consistent generation of cooking procedural images. Our experimental results demonstrate that our model excels at generating high-quality cooking procedural images.
• Score: 58.92430755180394
• License:
• Abstract: Recent advancements in text-to-image generation models have excelled in creating diverse and realistic images. This success extends to food imagery, where various conditional inputs like cooking styles, ingredients, and recipes are utilized. However, a yet-unexplored challenge is generating a sequence of procedural images based on cooking steps from a recipe. This could enhance the cooking experience with visual guidance and possibly lead to an intelligent cooking simulation system. To fill this gap, we introduce a novel task called \textbf{cooking procedural image generation}. This task is inherently demanding, as it strives to create photo-realistic images that align with cooking steps while preserving sequential consistency. To collectively tackle these challenges, we present \textbf{CookingDiffusion}, a novel approach that leverages Stable Diffusion and three innovative Memory Nets to model procedural prompts. These prompts encompass text prompts (representing cooking steps), image prompts (corresponding to cooking images), and multi-modal prompts (mixing cooking steps and images), ensuring the consistent generation of cooking procedural images. To validate the effectiveness of our approach, we preprocess the YouCookII dataset, establishing a new benchmark. Our experimental results demonstrate that our model excels at generating high-quality cooking procedural images with remarkable consistency across sequential cooking steps, as measured by both the FID and the proposed Average Procedure Consistency metrics. Furthermore, CookingDiffusion demonstrates the ability to manipulate ingredients and cooking methods in a recipe. We will make our code, models, and dataset publicly accessible.

Related papers

• Retrieval Augmented Recipe Generation [96.43285670458803]
  We propose a retrieval augmented large multimodal model for recipe generation. It retrieves recipes semantically related to the image from an existing datastore as a supplement. It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation.
  arXiv  Detail & Related papers  (2024-11-13T15:58:50Z)
• FIRE: Food Image to REcipe generation [10.45344523054623]
  Food computing aims to develop end-to-end intelligent systems capable of autonomously producing recipe information for a food image. This paper proposes FIRE, a novel methodology tailored to recipe generation in the food computing domain. We showcase two practical applications that can benefit from integrating FIRE with large language model prompting.
  arXiv  Detail & Related papers  (2023-08-28T08:14:20Z)
• Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions [65.50975507723827]
  Food image segmentation is an important task that has ubiquitous applications, such as estimating the nutritional value of a plate of food. One challenge is that food items can overlap and mix, making them difficult to distinguish. Two models are trained and compared, one based on convolutional neural networks and the other on Bidirectional representation for Image Transformers (BEiT) The BEiT model outperforms the previous state-of-the-art model by achieving a mean intersection over union of 49.4 on FoodSeg103.
  arXiv  Detail & Related papers  (2023-06-15T15:38:10Z)
• Learning Program Representations for Food Images and Cooking Recipes [26.054436410924737]
  We propose to represent cooking recipes and food images as cooking programs. A model is trained to learn a joint embedding between recipes and food images via self-supervision. We show that projecting the image-recipe embeddings into programs leads to better cross-modal retrieval results.
  arXiv  Detail & Related papers  (2022-03-30T05:52:41Z)
• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning [17.42688184238741]
  Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives. We propose a simplified end-to-end model based on well established and high performing encoders for text and images. Our proposed method achieves state-of-the-art performance in the cross-modal recipe retrieval task on the Recipe1M dataset.
  arXiv  Detail & Related papers  (2021-03-24T10:17:09Z)
• Structure-Aware Generation Network for Recipe Generation from Images [142.047662926209]
  We investigate an open research task of generating cooking instructions based on only food images and ingredients. Target recipes are long-length paragraphs and do not have annotations on structure information. We propose a novel framework of Structure-aware Generation Network (SGN) to tackle the food recipe generation task.
  arXiv  Detail & Related papers  (2020-09-02T10:54:25Z)
• Multi-modal Cooking Workflow Construction for Food Recipes [147.4435186953995]
  We build MM-ReS, the first large-scale dataset for cooking workflow construction. We propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow.
  arXiv  Detail & Related papers  (2020-08-20T18:31:25Z)
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
  We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
  arXiv  Detail & Related papers  (2020-03-09T07:41:17Z)
• CookGAN: Meal Image Synthesis from Ingredients [24.295634252929112]
  We propose a new computational framework, based on generative deep models, for synthesis of photo-realistic food meal images from textual list of its ingredients. CookGAN builds an attention-based ingredients-image association model, which is then used to condition a generative neural network tasked with synthesizing meal images.
  arXiv  Detail & Related papers  (2020-02-25T00:54:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.