Cross-Lingual and Cross-Cultural Variation in Image Descriptions

• URL: http://arxiv.org/abs/2409.16646v3
• Date: Sat, 12 Oct 2024 13:05:56 GMT
• Title: Cross-Lingual and Cross-Cultural Variation in Image Descriptions
• Authors: Uri Berger, Edoardo M. Ponti,
• Abstract summary: We conduct the first large-scale empirical study of cross-lingual variation in image descriptions. We use a multimodal dataset with 31 languages and images from diverse locations. Our analysis reveals that pairs of languages that are geographically or genetically closer tend to mention the same entities more frequently.
• Score: 2.8664758928324883
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Do speakers of different languages talk differently about what they see? Behavioural and cognitive studies report cultural effects on perception; however, these are mostly limited in scope and hard to replicate. In this work, we conduct the first large-scale empirical study of cross-lingual variation in image descriptions. Using a multimodal dataset with 31 languages and images from diverse locations, we develop a method to accurately identify entities mentioned in captions and present in the images, then measure how they vary across languages. Our analysis reveals that pairs of languages that are geographically or genetically closer tend to mention the same entities more frequently. We also identify entity categories whose saliency is universally high (such as animate beings), low (clothing accessories) or displaying high variance across languages (landscape). In a case study, we measure the differences in a specific language pair (e.g., Japanese mentions clothing far more frequently than English). Furthermore, our method corroborates previous small-scale studies, including 1) Rosch et al. (1976)'s theory of basic-level categories, demonstrating a preference for entities that are neither too generic nor too specific, and 2) Miyamoto et al. (2006)'s hypothesis that environments afford patterns of perception, such as entity counts. Overall, our work reveals the presence of both universal and culture-specific patterns in entity mentions.

Related papers

• Computer Vision Datasets and Models Exhibit Cultural and Linguistic Diversity in Perception [28.716435050743957]
  We study how people from different cultural backgrounds observe vastly different concepts even when viewing the same visual stimuli. By comparing textual descriptions generated across 7 languages for the same images, we find significant differences in the semantic content and linguistic expression. Our work points towards the need to accounttuning for and embrace the diversity of human perception in the computer vision community.
  arXiv  Detail & Related papers  (2023-10-22T16:51:42Z)
• How Different Is Stereotypical Bias Across Languages? [1.0467550794914122]
  Recent studies have demonstrated how to assess the stereotypical bias in pre-trained English language models. We make use of the English StereoSet data set (Nadeem et al., 2021), which we semi-automatically translate into German, French, Spanish, and Turkish. The main takeaways from our analysis are that mGPT-2 shows surprising anti-stereotypical behavior across languages, English (monolingual) models exhibit the strongest bias, and the stereotypes reflected in the data set are least present in Turkish models.
  arXiv  Detail & Related papers  (2023-07-14T13:17:11Z)
• Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity [64.18762301574954]
  Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings. This seems to be true for both monolingual and multilingual models, although much less work has been done on the multilingual context. We investigate outlier dimensions and their relationship to anisotropy in multiple pre-trained multilingual language models.
  arXiv  Detail & Related papers  (2023-06-01T09:01:48Z)
• Multi-lingual and Multi-cultural Figurative Language Understanding [69.47641938200817]
  Figurative language permeates human communication, but is relatively understudied in NLP. We create a dataset for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data.
  arXiv  Detail & Related papers  (2023-05-25T15:30:31Z)
• Comparing Biases and the Impact of Multilingual Training across Multiple Languages [70.84047257764405]
  We present a bias analysis across Italian, Chinese, English, Hebrew, and Spanish on the downstream sentiment analysis task. We adapt existing sentiment bias templates in English to Italian, Chinese, Hebrew, and Spanish for four attributes: race, religion, nationality, and gender. Our results reveal similarities in bias expression such as favoritism of groups that are dominant in each language's culture.
  arXiv  Detail & Related papers  (2023-05-18T18:15:07Z)
• Measuring Geographic Performance Disparities of Offensive Language Classifiers [12.545108947857802]
  We ask two questions: Does language, dialect, and topical content vary across geographical regions?'' and If there are differences across the regions, do they impact model performance?'' We find that current models do not generalize across locations. Likewise, we show that while offensive language models produce false positives on African American English, model performance is not correlated with each city's minority population proportions.
  arXiv  Detail & Related papers  (2022-09-15T15:08:18Z)
• Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models [84.86942006830772]
  We conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe.
  arXiv  Detail & Related papers  (2022-05-04T12:22:31Z)
• Analyzing Gender Representation in Multilingual Models [59.21915055702203]
  We focus on the representation of gender distinctions as a practical case study. We examine the extent to which the gender concept is encoded in shared subspaces across different languages.
  arXiv  Detail & Related papers  (2022-04-20T00:13:01Z)
• Deception detection in text and its relation to the cultural dimension of individualism/collectivism [6.17866386107486]
  We investigate if differences in the usage of specific linguistic features of deception across cultures can be confirmed and attributed to norms in respect to the individualism/collectivism divide. We create culture/language-aware classifiers by experimenting with a wide range of n-gram features based on phonology, morphology and syntax. We conducted our experiments over 11 datasets from 5 languages i.e., English, Dutch, Russian, Spanish and Romanian, from six countries (US, Belgium, India, Russia, Mexico and Romania)
  arXiv  Detail & Related papers  (2021-05-26T13:09:47Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.