Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models

• URL: http://arxiv.org/abs/2407.02623v3
• Date: Mon, 14 Oct 2024 14:11:42 GMT
• Title: Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models
• Authors: Joan Nwatu, Oana Ignat, Rada Mihalcea,
• Abstract summary: We propose and evaluate several prompting strategies using non-English, geographic, and socioeconomic attributes. We show that these geographic and socioeconomic integrated prompts favor retrieving topic appearances commonly found in data from low-income households across different countries.
• Score: 28.3552578648979
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has demonstrated that the unequal representation of cultures and socioeconomic groups in training data leads to biased Large Multi-modal (LMM) models. To improve LMM model performance on underrepresented data, we propose and evaluate several prompting strategies using non-English, geographic, and socioeconomic attributes. We show that these geographic and socioeconomic integrated prompts favor retrieving topic appearances commonly found in data from low-income households across different countries leading to improved LMM model performance on lower-income data. Our analyses identify and highlight contexts where these strategies yield the most improvements.

Related papers

• Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives [10.753785813662654]
  We present an approach to modeling annotator disagreement in subjective NLP tasks through both architectural and data-centric innovations.<n>Our modelworks based on annotator demographics, enabling it to better represent structured, group-level variation.<n>We propose and evaluate approaches for blending real and synthetic data using strategies tailored to dataset structure.
  arXiv  Detail & Related papers  (2025-08-04T19:27:17Z)
• GeoReg: Weight-Constrained Few-Shot Regression for Socio-Economic Estimation using LLM [12.740560201477798]
  This research introduces GeoReg a regression model that integrates diverse data sources to estimate socio-economic indicators.<n> Experiments across three countries at different stages of development demonstrate that our model outperforms baselines in estimating socio-economic indicators.
  arXiv  Detail & Related papers  (2025-07-17T17:42:29Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Social Debiasing for Fair Multi-modal LLMs [55.8071045346024]
  Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities. However, these models often inherit severe social biases from their training datasets, leading to unfair predictions based on attributes like race and gender. This paper addresses the issue of social biases in MLLMs by i) Introducing a comprehensive Counterfactual dataset with Multiple Social Concepts (CMSC) and ii) Proposing an Anti-Stereotype Debiasing strategy (ASD)
  arXiv  Detail & Related papers  (2024-08-13T02:08:32Z)
• Understanding Intrinsic Socioeconomic Biases in Large Language Models [4.276697874428501]
  We introduce a novel dataset of one million English sentences to quantify socioeconomic biases. Our findings reveal pervasive socioeconomic biases in both established models like GPT-2 and state-of-the-art models like Llama 2 and Falcon.
  arXiv  Detail & Related papers  (2024-05-28T23:54:44Z)
• Chain-of-Thought Prompting for Demographic Inference with Large Multimodal Models [58.58594658683919]
  Large multimodal models (LMMs) have shown transformative potential across various research tasks. Our findings indicate LMMs possess advantages in zero-shot learning, interpretability, and handling uncurated 'in-the-wild' inputs. We propose a Chain-of-Thought augmented prompting approach, which effectively mitigates the off-target prediction issue.
  arXiv  Detail & Related papers  (2024-05-24T16:26:56Z)
• Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models [31.868468221653025]
  We evaluate the performance of a vision-language model (CLIP) on a geo-diverse dataset containing household images associated with different income values. Our results indicate that performance for the poorer groups is consistently lower than the wealthier groups across various topics and countries.
  arXiv  Detail & Related papers  (2023-11-09T21:10:52Z)
• Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks [65.21536453075275]
  We focus on the summarization task and investigate the membership inference (MI) attack. We exploit text similarity and the model's resistance to document modifications as potential MI signals. We discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.
  arXiv  Detail & Related papers  (2023-10-20T05:44:39Z)
• Survey of Social Bias in Vision-Language Models [65.44579542312489]
  Survey aims to provide researchers with a high-level insight into the similarities and differences of social bias studies in pre-trained models across NLP, CV, and VL. The findings and recommendations presented here can benefit the ML community, fostering the development of fairer and non-biased AI models.
  arXiv  Detail & Related papers  (2023-09-24T15:34:56Z)
• An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models [116.50367506746713]
  We present an empirical study of scaling LLaVA up to 33B and 65B/70B. We find that scaling LMM consistently enhances model performance and improves language capabilities. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible.
  arXiv  Detail & Related papers  (2023-09-18T17:30:46Z)
• Non-Invasive Fairness in Learning through the Lens of Data Drift [88.37640805363317]
  We show how to improve the fairness of Machine Learning models without altering the data or the learning algorithm. We use a simple but key insight: the divergence of trends between different populations, and, consecutively, between a learned model and minority populations, is analogous to data drift. We explore two strategies (model-splitting and reweighing) to resolve this drift, aiming to improve the overall conformance of models to the underlying data.
  arXiv  Detail & Related papers  (2023-03-30T17:30:42Z)
• Learning Economic Indicators by Aggregating Multi-Level Geospatial Information [20.0397537179667]
  This research presents a deep learning model to predict economic indicators via aggregating traits observed from multiple levels of geographical units. Our new multi-level learning model substantially outperforms strong baselines in predicting key indicators such as population, purchasing power, and energy consumption. We discuss the multi-level model's implications for measuring inequality, which is the essential first step in policy and social science research on inequality and poverty.
  arXiv  Detail & Related papers  (2022-05-03T13:05:39Z)
• Aggregate Learning for Mixed Frequency Data [0.0]
  We propose a mixed-temporal aggregate learning model that predicts economic indicators for smaller areas in real-time. We find that the proposed model predicts (i) the regional heterogeneity of the labor market condition and (ii) the rapidly changing economic status. The model can be applied to various tasks, especially economic analysis.
  arXiv  Detail & Related papers  (2021-05-20T08:12:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.