Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models

• URL: http://arxiv.org/abs/2304.13855v1
• Date: Wed, 26 Apr 2023 22:53:31 GMT
• Title: Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models
• Authors: Abhishek Mandal, Susan Leavy, Suzanne Little
• Abstract summary: Multimodal Composite Association Score (MCAS) is a new method of measuring gender bias in multimodal generative models. MCAS is an accessible and scalable method of quantifying potential bias for models with different modalities and a range of potential biases.
• Score: 6.369985818712948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative multimodal models based on diffusion models have seen tremendous growth and advances in recent years. Models such as DALL-E and Stable Diffusion have become increasingly popular and successful at creating images from texts, often combining abstract ideas. However, like other deep learning models, they also reflect social biases they inherit from their training data, which is often crawled from the internet. Manually auditing models for biases can be very time and resource consuming and is further complicated by the unbounded and unconstrained nature of inputs these models can take. Research into bias measurement and quantification has generally focused on small single-stage models working on a single modality. Thus the emergence of multistage multimodal models requires a different approach. In this paper, we propose Multimodal Composite Association Score (MCAS) as a new method of measuring gender bias in multimodal generative models. Evaluating both DALL-E 2 and Stable Diffusion using this approach uncovered the presence of gendered associations of concepts embedded within the models. We propose MCAS as an accessible and scalable method of quantifying potential bias for models with different modalities and a range of potential biases.

Related papers

• Diffusion Models for Multi-Task Generative Modeling [32.61765315067488]
  We propose a principled way to define a diffusion model by constructing a unified multi-modal diffusion model in a common diffusion space. We propose several multimodal generation settings to verify our framework, including image transition, masked-image training, joint image-label and joint image-representation generative modeling.
  arXiv  Detail & Related papers  (2024-07-24T18:04:17Z)
• DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception [66.88792390480343]
  We propose DEEM, a simple and effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. DEEM exhibits enhanced robustness and a superior capacity to alleviate hallucinations while utilizing fewer trainable parameters, less pre-training data, and a smaller base model size.
  arXiv  Detail & Related papers  (2024-05-24T05:46:04Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• MAFIA: Multi-Adapter Fused Inclusive LanguAge Models [13.793816113015513]
  Pretrained Language Models (PLMs) are widely used in NLP for various tasks. Recent studies have identified various biases that such models exhibit and have proposed methods to correct these biases. We propose a debiasing model that exploits the synergy amongst various societal biases and enables multi-bias debiasing simultaneously.
  arXiv  Detail & Related papers  (2024-02-12T09:41:00Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Generalized Multimodal ELBO [11.602089225841631]
  Multiple data types naturally co-occur when describing real-world phenomena and learning from them is a long-standing goal in machine learning research. Existing self-supervised generative models approximating an ELBO are not able to fulfill all desired requirements of multimodal models. We propose a new, generalized ELBO formulation for multimodal data that overcomes these limitations.
  arXiv  Detail & Related papers  (2021-05-06T07:05:00Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [55.28436972267793]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)
• Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models [86.9292779620645]
  We develop a contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. Under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
  arXiv  Detail & Related papers  (2020-07-02T15:08:11Z)
• MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning [8.70928211339504]
  We contribute the Multimodal Hierarchical Vari Auto-encoder (MHVAE), a hierarchical multimodal generative model for representation learning. Inspired by human cognitive models, the MHVAE is able to learn modality-specific distributions and a joint-modality distribution, responsible for cross-modality inference. Our model performs on par with other state-of-the-art generative models regarding joint-modality reconstruction from arbitrary input modalities and cross-modality inference.
  arXiv  Detail & Related papers  (2020-06-04T16:24:00Z)
• When Ensembling Smaller Models is More Efficient than Single Large Models [52.38997176317532]
  We show that ensembles can outperform single models with both higher accuracy and requiring fewer total FLOPs to compute. This presents an interesting observation that output diversity in ensembling can often be more efficient than training larger models.
  arXiv  Detail & Related papers  (2020-05-01T18:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.