Black-Box Anomaly Attribution
- URL: http://arxiv.org/abs/2305.18440v1
- Date: Mon, 29 May 2023 01:42:32 GMT
- Title: Black-Box Anomaly Attribution
- Authors: Tsuyoshi Id\'e, Naoki Abe
- Abstract summary: When a black-box machine learning model deviates from the true observation, what can be said about the reason behind that deviation?
This is a fundamental and ubiquitous question that the end user in a business or industrial AI application often asks.
We propose a novel likelihood-based attribution framework we call the likelihood compensation''
- Score: 2.22999837458579
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: When the prediction of a black-box machine learning model deviates from the
true observation, what can be said about the reason behind that deviation? This
is a fundamental and ubiquitous question that the end user in a business or
industrial AI application often asks. The deviation may be due to a sub-optimal
black-box model, or it may be simply because the sample in question is an
outlier. In either case, one would ideally wish to obtain some form of
attribution score -- a value indicative of the extent to which an input
variable is responsible for the anomaly.
In the present paper we address this task of ``anomaly attribution,''
particularly in the setting in which the model is black-box and the training
data are not available. Specifically, we propose a novel likelihood-based
attribution framework we call the ``likelihood compensation (LC),'' in which
the responsibility score is equated with the correction on each input variable
needed to attain the highest possible likelihood. We begin by showing formally
why mainstream model-agnostic explanation methods, such as the local linear
surrogate modeling and Shapley values, are not designed to explain anomalies.
In particular, we show that they are ``deviation-agnostic,'' namely, that their
explanations are blind to the fact that there is a deviation in the model
prediction for the sample of interest. We do this by positioning these existing
methods under the unified umbrella of a function family we call the
``integrated gradient family.'' We validate the effectiveness of the proposed
LC approach using publicly available data sets. We also conduct a case study
with a real-world building energy prediction task and confirm its usefulness in
practice based on expert feedback.
Related papers
- Rejection via Learning Density Ratios [50.91522897152437]
Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.
We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
Our framework is tested empirically over clean and noisy datasets.
arXiv Detail & Related papers (2024-05-29T01:32:17Z) - Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation
of Prediction Rationale [53.152460508207184]
Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data.
This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis.
To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
arXiv Detail & Related papers (2024-02-02T05:53:22Z) - Calibrating Neural Simulation-Based Inference with Differentiable
Coverage Probability [50.44439018155837]
We propose to include a calibration term directly into the training objective of the neural model.
By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation.
It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference.
arXiv Detail & Related papers (2023-10-20T10:20:45Z) - Generative Perturbation Analysis for Probabilistic Black-Box Anomaly
Attribution [2.22999837458579]
We address the task of probabilistic anomaly attribution in the black-box regression setting.
This task differs from the standard XAI (explainable AI) scenario, since we wish to explain the anomalous deviation from a black-box prediction rather than the black-box model itself.
We propose a novel framework for probabilistic anomaly attribution that allows us to not only compute attribution scores as the predictive mean but also quantify the uncertainty of those scores.
arXiv Detail & Related papers (2023-08-09T04:59:06Z) - BELLA: Black box model Explanations by Local Linear Approximations [10.05944106581306]
We present BELLA, a deterministic model-agnostic post-hoc approach for explaining the individual predictions of regression black-box models.
BELLA provides explanations in the form of a linear model trained in the feature space.
BELLA can produce both factual and counterfactual explanations.
arXiv Detail & Related papers (2023-05-18T21:22:23Z) - Exploring validation metrics for offline model-based optimisation with
diffusion models [50.404829846182764]
In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle.
While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples.
This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation.
arXiv Detail & Related papers (2022-11-19T16:57:37Z) - Anomaly Attribution with Likelihood Compensation [14.99385222547436]
This paper addresses the task of explaining anomalous predictions of a black-box regression model.
Given model deviation from the expected value, infer the responsibility score of each of the input variables.
To the best of our knowledge, this is the first principled framework that computes a responsibility score for real valued anomalous model deviations.
arXiv Detail & Related papers (2022-08-23T02:00:20Z) - Explain, Edit, and Understand: Rethinking User Study Design for
Evaluating Model Explanations [97.91630330328815]
We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews.
We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
arXiv Detail & Related papers (2021-12-17T18:29:56Z) - Scalable Marginal Likelihood Estimation for Model Selection in Deep
Learning [78.83598532168256]
Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties.
Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
arXiv Detail & Related papers (2021-04-11T09:50:24Z) - Estimating g-Leakage via Machine Learning [34.102705643128004]
This paper considers the problem of estimating the information leakage of a system in the black-box scenario.
It is assumed that the system's internals are unknown to the learner, or anyway too complicated to analyze.
We propose a novel approach to perform black-box estimation of the g-vulnerability using Machine Learning (ML) algorithms.
arXiv Detail & Related papers (2020-05-09T09:26:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.