Don't guess what's true: choose what's optimal. A probability transducer
for machine-learning classifiers
- URL: http://arxiv.org/abs/2302.10578v1
- Date: Tue, 21 Feb 2023 10:14:13 GMT
- Title: Don't guess what's true: choose what's optimal. A probability transducer
for machine-learning classifiers
- Authors: K. Dyrland, A. S. Lundervold, P.G.L. Porta Mana
- Abstract summary: In medicine and drug discovery, the ultimate goal of a classification is not to guess a class, but to choose the optimal course of action among a set of possible ones.
The main idea of the present work is to calculate probabilities conditional not on the features, but on the trained classifier's output.
This calculation is cheap, needs to be made only once, and provides an output-to-probability "transducer" that can be applied to all future outputs.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In fields such as medicine and drug discovery, the ultimate goal of a
classification is not to guess a class, but to choose the optimal course of
action among a set of possible ones, usually not in one-one correspondence with
the set of classes. This decision-theoretic problem requires sensible
probabilities for the classes. Probabilities conditional on the features are
computationally almost impossible to find in many important cases. The main
idea of the present work is to calculate probabilities conditional not on the
features, but on the trained classifier's output. This calculation is cheap,
needs to be made only once, and provides an output-to-probability "transducer"
that can be applied to all future outputs of the classifier. In conjunction
with problem-dependent utilities, the probabilities of the transducer allow us
to find the optimal choice among the classes or among a set of more general
decisions, by means of expected-utility maximization. This idea is demonstrated
in a simplified drug-discovery problem with a highly imbalanced dataset. The
transducer and utility maximization together always lead to improved results,
sometimes close to theoretical maximum, for all sets of problem-dependent
utilities. The one-time-only calculation of the transducer also provides,
automatically: (i) a quantification of the uncertainty about the transducer
itself; (ii) the expected utility of the augmented algorithm (including its
uncertainty), which can be used for algorithm selection; (iii) the possibility
of using the algorithm in a "generative mode", useful if the training dataset
is biased.
Related papers
- Accelerated zero-order SGD under high-order smoothness and overparameterized regime [79.85163929026146]
We present a novel gradient-free algorithm to solve convex optimization problems.
Such problems are encountered in medicine, physics, and machine learning.
We provide convergence guarantees for the proposed algorithm under both types of noise.
arXiv Detail & Related papers (2024-11-21T10:26:17Z) - When can you trust feature selection? -- I: A condition-based analysis
of LASSO and generalised hardness of approximation [49.1574468325115]
We show how no (randomised) algorithm can determine the correct support sets (with probability $> 1/2$) of minimisers of LASSO when reading approximate input.
For ill-posed inputs, the algorithm runs forever, hence, it will never produce a wrong answer.
For any algorithm defined on an open set containing a point with infinite condition number, there is an input for which the algorithm will either run forever or produce a wrong answer.
arXiv Detail & Related papers (2023-12-18T18:29:01Z) - Nonparametric active learning for cost-sensitive classification [2.1756081703276]
We design a generic nonparametric active learning algorithm for cost-sensitive classification.
We prove the near-optimality of obtained upper bounds by providing matching (up to logarithmic factor) lower bounds.
arXiv Detail & Related papers (2023-09-30T22:19:21Z) - Experience in Engineering Complex Systems: Active Preference Learning
with Multiple Outcomes and Certainty Levels [1.5257326975704795]
Black-box optimization refers to the problem whose objective function and/or constraint sets are either unknown, inaccessible, or non-existent.
The algorithm so-called Active Preference Learning has been developed to exploit this specific information.
Our approach aims to extend the algorithm in such a way that can exploit further information effectively.
arXiv Detail & Related papers (2023-02-27T15:55:37Z) - Who Should Predict? Exact Algorithms For Learning to Defer to Humans [40.22768241509553]
We show that prior approaches can fail to find a human-AI system with low misclassification error.
We give a mixed-integer-linear-programming (MILP) formulation that can optimally solve the problem in the linear setting.
We provide a novel surrogate loss function that is realizable-consistent and performs well empirically.
arXiv Detail & Related papers (2023-01-15T21:57:36Z) - Reliability analysis of discrete-state performance functions via
adaptive sequential sampling with detection of failure surfaces [0.0]
The paper presents a new efficient and robust method for rare event probability estimation.
The method can estimate the probabilities of multiple failure types.
It can accommodate this information to increase the accuracy of the estimated probabilities.
arXiv Detail & Related papers (2022-08-04T05:59:25Z) - Efficient and Differentiable Conformal Prediction with General Function
Classes [96.74055810115456]
We propose a generalization of conformal prediction to multiple learnable parameters.
We show that it achieves approximate valid population coverage and near-optimal efficiency within class.
Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
arXiv Detail & Related papers (2022-02-22T18:37:23Z) - Offline Model-Based Optimization via Normalized Maximum Likelihood
Estimation [101.22379613810881]
We consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points.
This problem setting emerges in many domains where function evaluation is a complex and expensive process.
We propose a tractable approximation that allows us to scale our method to high-capacity neural network models.
arXiv Detail & Related papers (2021-02-16T06:04:27Z) - Online Model Selection for Reinforcement Learning with Function
Approximation [50.008542459050155]
We present a meta-algorithm that adapts to the optimal complexity with $tildeO(L5/6 T2/3)$ regret.
We also show that the meta-algorithm automatically admits significantly improved instance-dependent regret bounds.
arXiv Detail & Related papers (2020-11-19T10:00:54Z) - Theoretical Insights Into Multiclass Classification: A High-dimensional
Asymptotic View [82.80085730891126]
We provide the first modernally precise analysis of linear multiclass classification.
Our analysis reveals that the classification accuracy is highly distribution-dependent.
The insights gained may pave the way for a precise understanding of other classification algorithms.
arXiv Detail & Related papers (2020-11-16T05:17:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.