Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification

• URL: http://arxiv.org/abs/2505.08173v1
• Date: Tue, 13 May 2025 02:23:55 GMT
• Title: Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification
• Authors: Xiaoshuo Yan, Zhaochuan Li, Lei Meng, Zhuang Qi, Wei Wu, Zixuan Li, Xiangxu Meng,
• Abstract summary: Causal inference has emerged as a promising approach to mitigate long-tail classification by handling the biases introduced by class imbalance.<n>This paper investigates the influence of existing causal models on CNNs and ViT variants.<n>It proposes TSCNet, a two-stage causal modeling method to discover fine-grained causal associations.
• Score: 12.122203089278738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Causal inference has emerged as a promising approach to mitigate long-tail classification by handling the biases introduced by class imbalance. However, along with the change of advanced backbone models from Convolutional Neural Networks (CNNs) to Visual Transformers (ViT), existing causal models may not achieve an expected performance gain. This paper investigates the influence of existing causal models on CNNs and ViT variants, highlighting that ViT's global feature representation makes it hard for causal methods to model associations between fine-grained features and predictions, which leads to difficulties in classifying tail classes with similar visual appearance. To address these issues, this paper proposes TSCNet, a two-stage causal modeling method to discover fine-grained causal associations through multi-scale causal interventions. Specifically, in the hierarchical causal representation learning stage (HCRL), it decouples the background and objects, applying backdoor interventions at both the patch and feature level to prevent model from using class-irrelevant areas to infer labels which enhances fine-grained causal representation. In the counterfactual logits bias calibration stage (CLBC), it refines the optimization of model's decision boundary by adaptive constructing counterfactual balanced data distribution to remove the spurious associations in the logits caused by data distribution. Extensive experiments conducted on various long-tail benchmarks demonstrate that the proposed TSCNet can eliminate multiple biases introduced by data imbalance, which outperforms existing methods.

Related papers

• Mitigating Spurious Correlations with Causal Logit Perturbation [22.281052412112263]
  This study introduces a novel Causal Logit Perturbation (CLP) framework to train classifiers with generated causal logit perturbations for individual samples.<n>The framework is optimized by an online meta-learning-based learning algorithm and leverages human causal knowledge by augmenting metadata in both counterfactual and factual manners.
  arXiv  Detail & Related papers  (2025-05-21T08:21:02Z)
• Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalization [25.181305392387802]
  Attribute bias in federated learning (FL) typically leads local models to optimize inconsistently due to the learning of non-causal associations.<n>We propose the underlineFederated underlineDeconfounding and underlineDebiasing underlineLearning (FedDDL) method.<n>It constructs a structured causal graph to analyze the model inference process, and performs backdoor adjustment to eliminate confounding paths.
  arXiv  Detail & Related papers  (2025-05-08T06:32:59Z)
• A Causal Adjustment Module for Debiasing Scene Graph Generation [28.44150555570101]
  We employ causal inference techniques to model the causality among skewed distributions.<n>Our method enables the composition of zero-shot relationships, thereby enhancing the model's ability to recognize such relationships.
  arXiv  Detail & Related papers  (2025-03-22T20:44:01Z)
• Fighting Spurious Correlations in Text Classification via a Causal Learning Perspective [2.7813683000222653]
  We propose the Causally Calibrated Robust ( CCR) to reduce models' reliance on spurious correlations.<n> CCR integrates a causal feature selection method based on counterfactual reasoning, along with an inverse propensity weighting (IPW) loss function.<n>We show that CCR state-of-the-art performance among methods without group labels, and in some cases, it can compete with the models that utilize group labels.
  arXiv  Detail & Related papers  (2024-11-01T21:29:07Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• De-Biasing Generative Models using Counterfactual Methods [0.0]
  We propose a new decoder based framework named the Causal Counterfactual Generative Model (CCGM) Our proposed method combines a causal latent space VAE model with specific modification to emphasize causal fidelity. We explore how better disentanglement of causal learning and encoding/decoding generates higher causal intervention quality.
  arXiv  Detail & Related papers  (2022-07-04T16:53:20Z)
• Treatment Learning Causal Transformer for Noisy Image Classification [62.639851972495094]
  In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy. Motivated from causal variational inference, we propose a transformer-based architecture, that uses a latent generative model to estimate robust feature representations for noise image classification. We also create new noisy image datasets incorporating a wide range of noise factors for performance benchmarking.
  arXiv  Detail & Related papers  (2022-03-29T13:07:53Z)
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective [72.55093886515824]
  We introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables. We devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a causal graph. Experiment results on synthetic and real datasets show that our three proposed components significantly improve the robustness and reusability of the learned motion representations.
  arXiv  Detail & Related papers  (2021-11-29T18:59:09Z)
• Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization [89.73665256847858]
  We show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet. We also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS.
  arXiv  Detail & Related papers  (2021-07-09T19:48:23Z)
• Semantic Correspondence with Transformers [68.37049687360705]
  We propose Cost Aggregation with Transformers (CATs) to find dense correspondences between semantically similar images. We include appearance affinity modelling to disambiguate the initial correlation maps and multi-level aggregation. We conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods and provide extensive ablation studies.
  arXiv  Detail & Related papers  (2021-06-04T14:39:03Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.