Marginal Effects for Non-Linear Prediction Functions
- URL: http://arxiv.org/abs/2201.08837v1
- Date: Fri, 21 Jan 2022 18:47:38 GMT
- Title: Marginal Effects for Non-Linear Prediction Functions
- Authors: Christian A. Scholbeck, Giuseppe Casalicchio, Christoph Molnar, Bernd
Bischl, Christian Heumann
- Abstract summary: We introduce a new class of marginal effects termed forward marginal effects.
We argue against summarizing feature effects of a non-linear prediction function in a single metric such as the average marginal effect.
- Score: 0.7349727826230864
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Beta coefficients for linear regression models represent the ideal form of an
interpretable feature effect. However, for non-linear models and especially
generalized linear models, the estimated coefficients cannot be interpreted as
a direct feature effect on the predicted outcome. Hence, marginal effects are
typically used as approximations for feature effects, either in the shape of
derivatives of the prediction function or forward differences in prediction due
to a change in a feature value. While marginal effects are commonly used in
many scientific fields, they have not yet been adopted as a model-agnostic
interpretation method for machine learning models. This may stem from their
inflexibility as a univariate feature effect and their inability to deal with
the non-linearities found in black box models. We introduce a new class of
marginal effects termed forward marginal effects. We argue to abandon
derivatives in favor of better-interpretable forward differences. Furthermore,
we generalize marginal effects based on forward differences to multivariate
changes in feature values. To account for the non-linearity of prediction
functions, we introduce a non-linearity measure for marginal effects. We argue
against summarizing feature effects of a non-linear prediction function in a
single metric such as the average marginal effect. Instead, we propose to
partition the feature space to compute conditional average marginal effects on
feature subspaces, which serve as conditional feature effect estimates.
Related papers
- Scaling and renormalization in high-dimensional regression [72.59731158970894]
This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models.
We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
arXiv Detail & Related papers (2024-05-01T15:59:00Z) - Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression [0.0]
This study explores the predictive properties of over parameterized nonlinear regression within the Bayesian framework.
Posterior contraction is established for generalized linear and single-neuron models with Lipschitz continuous activation functions.
The proposed method was validated via numerical simulations and a real data application.
arXiv Detail & Related papers (2024-04-06T04:22:48Z) - Double Machine Learning for Static Panel Models with Fixed Effects [0.0]
We use double machine learning (DML) to approximate high-dimensional and non-linear functions of confounders.
We propose new estimators by adapting correlated random effects, within-group and first-difference estimation for linear models.
arXiv Detail & Related papers (2023-12-13T14:34:12Z) - Optimal Nonlinearities Improve Generalization Performance of Random
Features [0.9790236766474201]
Random feature model with a nonlinear activation function has been shown to performally equivalent to a Gaussian model in terms of training and generalization errors.
We show that acquired parameters from the Gaussian model enable us to define a set of optimal nonlinearities.
Our numerical results validate that the optimized nonlinearities achieve better generalization performance than widely-used nonlinear functions such as ReLU.
arXiv Detail & Related papers (2023-09-28T20:55:21Z) - Monotonicity and Double Descent in Uncertainty Estimation with Gaussian
Processes [52.92110730286403]
It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions.
We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension.
We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
arXiv Detail & Related papers (2022-10-14T08:09:33Z) - Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing.
We study the case where probability distributions are not known a priori but need to be estimated from data.
arXiv Detail & Related papers (2022-08-29T16:16:22Z) - Improving Generalization via Uncertainty Driven Perturbations [107.45752065285821]
We consider uncertainty-driven perturbations of the training data points.
Unlike loss-driven perturbations, uncertainty-guided perturbations do not cross the decision boundary.
We show that UDP is guaranteed to achieve the robustness margin decision on linear models.
arXiv Detail & Related papers (2022-02-11T16:22:08Z) - LQF: Linear Quadratic Fine-Tuning [114.3840147070712]
We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning.
LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification.
arXiv Detail & Related papers (2020-12-21T06:40:20Z) - Non-parametric Models for Non-negative Functions [48.7576911714538]
We provide the first model for non-negative functions from the same good linear models.
We prove that it admits a representer theorem and provide an efficient dual formulation for convex problems.
arXiv Detail & Related papers (2020-07-08T07:17:28Z) - Assumption-lean inference for generalised linear model parameters [0.0]
We propose nonparametric definitions of main effect estimands and effect modification estimands.
These reduce to standard main effect and effect modification parameters in generalised linear models when these models are correctly specified.
We achieve an assumption-lean inference for these estimands.
arXiv Detail & Related papers (2020-06-15T13:49:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.