Quantile Activation: Correcting a Failure Mode of ML Models

• URL: http://arxiv.org/abs/2405.11573v2
• Date: Tue, 24 Dec 2024 05:16:49 GMT
• Title: Quantile Activation: Correcting a Failure Mode of ML Models
• Authors: Aditya Challa, Sravan Danda, Laurent Najman, Snehanshu Saha,
• Abstract summary: An established failure mode for machine learning models occurs when the same features are equally likely to belong to class 0 and class 1. A solvable case emerges when the probabilities of class 0 and 1 vary with the context distribution. We propose a simple activation function, quantile activation (QACT), that addresses this problem without significantly increasing computational costs.
• Score: 4.035209200949511
• License:
• Abstract: An established failure mode for machine learning models occurs when the same features are equally likely to belong to class 0 and class 1. In such cases, existing ML models cannot correctly classify the sample. However, a solvable case emerges when the probabilities of class 0 and 1 vary with the context distribution. To the best of our knowledge, standard neural network architectures like MLPs or CNNs are not equipped to handle this. In this article, we propose a simple activation function, quantile activation (QACT), that addresses this problem without significantly increasing computational costs. The core idea is to adapt the outputs of each neuron to its context distribution. The proposed quantile activation, QACT, produces the relative quantile of the sample in its context distribution, rather than the actual values, as in traditional networks. A practical example where the same sample can have different labels arises in cases of inherent distribution shift. We validate the proposed activation function under such shifts, using datasets designed to test robustness against distortions : CIFAR10C, CIFAR100C, MNISTC, TinyImagenetC. Our results demonstrate significantly better generalization across distortions compared to conventional classifiers, across various architectures. Although this paper presents a proof of concept, we find that this approach unexpectedly outperforms DINOv2 (small) under large distortions, despite DINOv2 being trained with a much larger network and dataset.

Related papers

• GHOST: Gaussian Hypothesis Open-Set Technique [10.426399605773083]
  Evaluations of large-scale recognition methods typically focus on overall performance. addressing fairness in Open-Set Recognition (OSR), we demonstrate that per-class performance can vary dramatically. We apply Z-score normalization to logits to mitigate the impact of feature magnitudes that deviate from the model's expectations.
  arXiv  Detail & Related papers  (2025-02-05T16:56:14Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Probabilistic Invariant Learning with Randomized Linear Classifiers [24.485477981244593]
  We show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, we propose a class of binary classification models called Randomized Linears (RLCs)
  arXiv  Detail & Related papers  (2023-08-08T17:18:04Z)
• Neural Dependencies Emerging from Learning Massive Categories [94.77992221690742]
  This work presents two astonishing findings on neural networks learned for large-scale image classification. 1) Given a well-trained model, the logits predicted for some category can be directly obtained by linearly combining the predictions of a few other categories. 2) Neural dependencies exist not only within a single model, but even between two independently learned models.
  arXiv  Detail & Related papers  (2022-11-21T09:42:15Z)
• Revisiting Sparse Convolutional Model for Visual Recognition [40.726494290922204]
  This paper revisits the sparse convolutional modeling for image classification. We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100, and ImageNet datasets.
  arXiv  Detail & Related papers  (2022-10-24T04:29:21Z)
• Characterizing and Understanding the Behavior of Quantized Models for Reliable Deployment [32.01355605506855]
  Quantization-aware training can produce more stable models than standard, adversarial, and Mixup training. Disagreements often have closer top-1 and top-2 output probabilities, and $Margin$ is a better indicator than the other uncertainty metrics to distinguish disagreements. We opensource our code and models as a new benchmark for further studying the quantized models.
  arXiv  Detail & Related papers  (2022-04-08T11:19:16Z)
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module [60.63525456640462]
  Dropout has been demonstrated as a simple and effective module to regularize the training process of deep neural networks. We propose contextual dropout with an efficient structural design as a simple and scalable sample-dependent dropout module. Our experimental results show that the proposed method outperforms baseline methods in terms of both accuracy and quality of uncertainty estimation.
  arXiv  Detail & Related papers  (2021-03-06T19:30:32Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Variational Mixture of Normalizing Flows [0.0]
  Deep generative models, such as generative adversarial networks autociteGAN, variational autoencoders autocitevaepaper, and their variants, have seen wide adoption for the task of modelling complex data distributions. Normalizing flows have overcome this limitation by leveraging the change-of-suchs formula for probability density functions. The present work overcomes this by using normalizing flows as components in a mixture model and devising an end-to-end training procedure for such a model.
  arXiv  Detail & Related papers  (2020-09-01T17:20:08Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Model Fusion via Optimal Transport [64.13185244219353]
  We present a layer-wise model fusion algorithm for neural networks. We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
  arXiv  Detail & Related papers  (2019-10-12T22:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.