FairViT: Fair Vision Transformer via Adaptive Masking

• URL: http://arxiv.org/abs/2407.14799v1
• Date: Sat, 20 Jul 2024 08:10:37 GMT
• Title: FairViT: Fair Vision Transformer via Adaptive Masking
• Authors: Bowei Tian, Ruijie Du, Yanning Shen,
• Abstract summary: Vision Transformer (ViT) has achieved excellent performance and demonstrated its promising potential in various computer vision tasks. However, most ViT-based works do not take fairness into account and it is unclear whether directly applying CNN-oriented debiased algorithm to ViT is feasible. We propose FairViT, a novel accurate and fair ViT framework.
• Score: 12.623905443515802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformer (ViT) has achieved excellent performance and demonstrated its promising potential in various computer vision tasks. The wide deployment of ViT in real-world tasks requires a thorough understanding of the societal impact of the model. However, most ViT-based works do not take fairness into account and it is unclear whether directly applying CNN-oriented debiased algorithm to ViT is feasible. Moreover, previous works typically sacrifice accuracy for fairness. Therefore, we aim to develop an algorithm that improves accuracy without sacrificing fairness. In this paper, we propose FairViT, a novel accurate and fair ViT framework. To this end, we introduce a novel distance loss and deploy adaptive fairness-aware masks on attention layers updating with model parameters. Experimental results show \sys can achieve accuracy better than other alternatives, even with competitive computational efficiency. Furthermore, \sys achieves appreciable fairness results.

Related papers

• A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysis [9.687982148528187]
  Convolutional Neural Networks (CNNs) are currently among the best texture analysis approaches. Vision Transformers (ViTs) have been surpassing the performance of CNNs on tasks such as object recognition. This work explores various pre-trained ViT architectures when transferred to tasks that rely on textures.
  arXiv  Detail & Related papers  (2024-06-10T09:48:13Z)
• Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design [84.34416126115732]
  Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute.
  arXiv  Detail & Related papers  (2023-05-22T13:39:28Z)
• Fairness-aware Vision Transformer via Debiased Self-Attention [12.406960223371959]
  Debiased Self-Attention (DSA) is a fairness-through-blindness approach that enforces Vision Transformer (ViT) to eliminate spurious features correlated with the sensitive label for bias mitigation. Our framework leads to improved fairness guarantees over prior works on multiple prediction tasks without compromising target prediction performance.
  arXiv  Detail & Related papers  (2023-01-31T17:44:59Z)
• Semi-supervised Vision Transformers at Scale [93.0621675558895]
  We study semi-supervised learning (SSL) for vision transformers (ViT) We propose a new SSL pipeline, consisting of first un/self-supervised pre-training, followed by supervised fine-tuning, and finally semi-supervised fine-tuning. Our proposed method, dubbed Semi-ViT, achieves comparable or better performance than the CNN counterparts in the semi-supervised classification setting.
  arXiv  Detail & Related papers  (2022-08-11T08:11:54Z)
• EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers [88.52500757894119]
  Self-attention based vision transformers (ViTs) have emerged as a very competitive architecture alternative to convolutional neural networks (CNNs) in computer vision. We introduce EdgeViTs, a new family of light-weight ViTs that, for the first time, enable attention-based vision models to compete with the best light-weight CNNs.
  arXiv  Detail & Related papers  (2022-05-06T18:17:19Z)
• AdaViT: Adaptive Tokens for Efficient Vision Transformer [91.88404546243113]
  We introduce AdaViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. AdaViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
  arXiv  Detail & Related papers  (2021-12-14T18:56:07Z)
• A Unified Pruning Framework for Vision Transformers [40.7622551128182]
  Vision transformer (ViT) and its variants have achieved promising performances in various computer vision tasks. We propose a unified framework for structural pruning of both ViTs and its variants, namely UP-ViTs. Our method focuses on pruning all ViTs components while maintaining the consistency of the model structure.
  arXiv  Detail & Related papers  (2021-11-30T05:01:02Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.