You are what you eat? Feeding foundation models a regionally diverse food dataset of World Wide Dishes

• URL: http://arxiv.org/abs/2406.09496v2
• Date: Tue, 01 Oct 2024 23:11:00 GMT
• Title: You are what you eat? Feeding foundation models a regionally diverse food dataset of World Wide Dishes
• Authors: Jabez Magomere, Shu Ishida, Tejumade Afonja, Aya Salama, Daniel Kochin, Foutse Yuehgoh, Imane Hamzaoui, Raesetje Sefala, Aisha Alaagib, Elizaveta Semenova, Lauren Crais, Siobhan Mackenzie Hall,
• Abstract summary: We present World Wide Dishes, a mixed text and image dataset consisting of 765 dishes, with dish names collected in 131 local languages. We demonstrate a novel means of operationalising capability and representational biases in foundation models such as language models and text-to-image generative models. We find that these models generally do not produce quality text and image outputs of dishes specific to different regions.
• Score: 3.1402605498916514
• License:
• Abstract: Foundation models are increasingly ubiquitous in our daily lives, used in everyday tasks such as text-image searches, interactions with chatbots, and content generation. As use increases, so does concern over the disparities in performance and fairness of these models for different people in different parts of the world. To assess these growing regional disparities, we present World Wide Dishes, a mixed text and image dataset consisting of 765 dishes, with dish names collected in 131 local languages. World Wide Dishes has been collected purely through human contribution and decentralised means, by creating a website widely distributed through social networks. Using the dataset, we demonstrate a novel means of operationalising capability and representational biases in foundation models such as language models and text-to-image generative models. We enrich these studies with a pilot community review to understand, from a first-person perspective, how these models generate images for people in five African countries and the United States. We find that these models generally do not produce quality text and image outputs of dishes specific to different regions. This is true even for the US, which is typically considered to be more well-resourced in training data - though the generation of US dishes does outperform that of the investigated African countries. The models demonstrate a propensity to produce outputs that are inaccurate as well as culturally misrepresentative, flattening, and insensitive. These failures in capability and representational bias have the potential to further reinforce stereotypes and disproportionately contribute to erasure based on region. The dataset and code are available at https://github.com/oxai/world-wide-dishes/.

Related papers

• Multilingual Diversity Improves Vision-Language Representations [66.41030381363244]
  Pre-training on this dataset outperforms using English-only or English-dominated datasets on ImageNet. On a geographically diverse task like GeoDE, we also observe improvements across all regions, with the biggest gain coming from Africa.
  arXiv  Detail & Related papers  (2024-05-27T08:08:51Z)
• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do. We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models. Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• FoodFusion: A Latent Diffusion Model for Realistic Food Image Generation [69.91401809979709]
  Current state-of-the-art image generation models such as Latent Diffusion Models (LDMs) have demonstrated the capacity to produce visually striking food-related images. We introduce FoodFusion, a Latent Diffusion model engineered specifically for the faithful synthesis of realistic food images from textual descriptions. The development of the FoodFusion model involves harnessing an extensive array of open-source food datasets, resulting in over 300,000 curated image-caption pairs.
  arXiv  Detail & Related papers  (2023-12-06T15:07:12Z)
• Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision [25.414154497482162]
  We demonstrate that diffusion models can be leveraged to create such a dataset. We benchmark several vision-language models on a multi-class occupation classification task. We find that images generated with non-Caucasian labels have a significantly higher occupation misclassification rate than images generated with Caucasian labels.
  arXiv  Detail & Related papers  (2023-11-25T19:40:13Z)
• All Should Be Equal in the Eyes of Language Models: Counterfactually Aware Fair Text Generation [16.016546693767403]
  We propose a framework that dynamically compares the model understanding of diverse demographics to generate more equitable sentences. CAFIE produces fairer text and strikes the best balance between fairness and language modeling capability.
  arXiv  Detail & Related papers  (2023-11-09T15:39:40Z)
• Computer Vision Datasets and Models Exhibit Cultural and Linguistic Diversity in Perception [28.716435050743957]
  We study how people from different cultural backgrounds observe vastly different concepts even when viewing the same visual stimuli. By comparing textual descriptions generated across 7 languages for the same images, we find significant differences in the semantic content and linguistic expression. Our work points towards the need to accounttuning for and embrace the diversity of human perception in the computer vision community.
  arXiv  Detail & Related papers  (2023-10-22T16:51:42Z)
• Exposing Bias in Online Communities through Large-Scale Language Models [3.04585143845864]
  This work uses the flaw of bias in language models to explore the biases of six different online communities. The bias of the resulting models is evaluated by prompting the models with different demographics and comparing the sentiment and toxicity values of these generations. This work not only affirms how easily bias is absorbed from training data but also presents a scalable method to identify and compare the bias of different datasets or communities.
  arXiv  Detail & Related papers  (2023-06-04T08:09:26Z)
• Inspecting the Geographical Representativeness of Images from Text-to-Image Models [52.80961012689933]
  We measure the geographical representativeness of generated images using a crowdsourced study comprising 540 participants across 27 countries. For deliberately underspecified inputs without country names, the generated images most reflect the surroundings of the United States followed by India. The overall scores for many countries still remain low, highlighting the need for future models to be more geographically inclusive.
  arXiv  Detail & Related papers  (2023-05-18T16:08:11Z)
• Analyzing Bias in Diffusion-based Face Generation Models [75.80072686374564]
  Diffusion models are increasingly popular in synthetic data generation and image editing applications. We investigate the presence of bias in diffusion-based face generation models with respect to attributes such as gender, race, and age. We examine how dataset size affects the attribute composition and perceptual quality of both diffusion and Generative Adversarial Network (GAN) based face generation models.
  arXiv  Detail & Related papers  (2023-05-10T18:22:31Z)
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models [73.12069620086311]
  We investigate the visual reasoning capabilities and social biases of text-to-image models. First, we measure three visual reasoning skills: object recognition, object counting, and spatial relation understanding. Second, we assess the gender and skin tone biases by measuring the gender/skin tone distribution of generated images.
  arXiv  Detail & Related papers  (2022-02-08T18:36:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.