Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval

• URL: http://arxiv.org/abs/2510.20393v1
• Date: Thu, 23 Oct 2025 09:43:43 GMT
• Title: Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval
• Authors: Qing Wang, Chong-Wah Ngo, Yu Cao, Ee-Peng Lim,
• Abstract summary: Cross-modal representations to bridge the modality gap between images and recipes tend to ignore subtle recipe-specific details.<n>This paper proposes a novel causal approach that predicts the culinary elements potentially overlooked in images.<n> Experiments are conducted on the standard monolingual Recipe1M dataset and a newly curated multilingual multicultural cuisine dataset.
• Score: 33.17028372962136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing approaches for image-to-recipe retrieval have the implicit assumption that a food image can fully capture the details textually documented in its recipe. However, a food image only reflects the visual outcome of a cooked dish and not the underlying cooking process. Consequently, learning cross-modal representations to bridge the modality gap between images and recipes tends to ignore subtle, recipe-specific details that are not visually apparent but are crucial for recipe retrieval. Specifically, the representations are biased to capture the dominant visual elements, resulting in difficulty in ranking similar recipes with subtle differences in use of ingredients and cooking methods. The bias in representation learning is expected to be more severe when the training data is mixed of images and recipes sourced from different cuisines. This paper proposes a novel causal approach that predicts the culinary elements potentially overlooked in images, while explicitly injecting these elements into cross-modal representation learning to mitigate biases. Experiments are conducted on the standard monolingual Recipe1M dataset and a newly curated multilingual multicultural cuisine dataset. The results indicate that the proposed causal representation learning is capable of uncovering subtle ingredients and cooking actions and achieves impressive retrieval performance on both monolingual and multilingual multicultural datasets.

Related papers

• Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval [33.21317747745805]
  This paper addresses the challenges of learning representations for recipes and food images in the cross-modal retrieval problem.<n>As the relationship between a recipe and its cooked dish is cause-and-effect, treating a recipe as a text source will create bias misleading image-and-recipe similarity judgment.<n>We propose a plug-and-play neural module, which is essentially a multi-label ingredient for debiasing.
  arXiv  Detail & Related papers  (2025-11-19T07:39:53Z)
• CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion [58.92430755180394]
  We present textbfCookingDiffusion, a novel approach to generate photo-realistic images of cooking steps.<n>These prompts encompass text prompts, image prompts, and multi-modal prompts, ensuring the consistent generation of cooking procedural images.<n>Our experimental results demonstrate that our model excels at generating high-quality cooking procedural images.
  arXiv  Detail & Related papers  (2025-01-15T06:58:53Z)
• Retrieval Augmented Recipe Generation [96.43285670458803]
  We propose a retrieval augmented large multimodal model for recipe generation.<n>It retrieves recipes semantically related to the image from an existing datastore as a supplement.<n>It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation.
  arXiv  Detail & Related papers  (2024-11-13T15:58:50Z)
• MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval [6.582204441933583]
  We propose a mask-augmentation-based local matching network (MALM) for image-to-recipe retrieval. Experimental results on Recipe1M dataset show our method can clearly outperform state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2023-05-18T22:25:50Z)
• Cross-lingual Adaptation for Recipe Retrieval with Mixup [56.79360103639741]
  Cross-modal recipe retrieval has attracted research attention in recent years, thanks to the availability of large-scale paired data for training. This paper studies unsupervised domain adaptation for image-to-recipe retrieval, where recipes in source and target domains are in different languages. A novel recipe mixup method is proposed to learn transferable embedding features between the two domains.
  arXiv  Detail & Related papers  (2022-05-08T15:04:39Z)
• A Large-Scale Benchmark for Food Image Segmentation [62.28029856051079]
  We build a new food image dataset FoodSeg103 (and its extension FoodSeg154) containing 9,490 images. We annotate these images with 154 ingredient classes and each image has an average of 6 ingredient labels and pixel-wise masks. We propose a multi-modality pre-training approach called ReLeM that explicitly equips a segmentation model with rich and semantic food knowledge.
  arXiv  Detail & Related papers  (2021-05-12T03:00:07Z)
• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning [17.42688184238741]
  Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives. We propose a simplified end-to-end model based on well established and high performing encoders for text and images. Our proposed method achieves state-of-the-art performance in the cross-modal recipe retrieval task on the Recipe1M dataset.
  arXiv  Detail & Related papers  (2021-03-24T10:17:09Z)
• Cross-modal Retrieval and Synthesis (X-MRS): Closing the modality gap in shared subspace [21.33710150033949]
  We propose a simple yet novel architecture for shared subspace learning, which is used to tackle the food image-to-recipe retrieval problem. Experimental analysis on the public Recipe1M dataset shows that the subspace learned via the proposed method outperforms the current state-of-the-arts. In order to demonstrate the representational power of the learned subspace, we propose a generative food image synthesis model conditioned on the embeddings of recipes.
  arXiv  Detail & Related papers  (2020-12-02T17:27:00Z)
• Multi-modal Cooking Workflow Construction for Food Recipes [147.4435186953995]
  We build MM-ReS, the first large-scale dataset for cooking workflow construction. We propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow.
  arXiv  Detail & Related papers  (2020-08-20T18:31:25Z)
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
  We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
  arXiv  Detail & Related papers  (2020-03-09T07:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.