Loss as the Inconsistency of a Probabilistic Dependency Graph: Choose
Your Model, Not Your Loss Function
- URL: http://arxiv.org/abs/2202.11862v1
- Date: Thu, 24 Feb 2022 01:51:21 GMT
- Title: Loss as the Inconsistency of a Probabilistic Dependency Graph: Choose
Your Model, Not Your Loss Function
- Authors: Oliver E Richardson
- Abstract summary: We show that many standard loss functions arise as the inconsistency of a natural PDG describing the appropriate scenario.
We also show that the PDG inconsistency captures a large class of statistical divergences.
We observe that inconsistency becomes the log partition function (free energy) in the setting where PDGs are factor graphs.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: In a world blessed with a great diversity of loss functions, we argue that
that choice between them is not a matter of taste or pragmatics, but of model.
Probabilistic depencency graphs (PDGs) are probabilistic models that come
equipped with a measure of "inconsistency". We prove that many standard loss
functions arise as the inconsistency of a natural PDG describing the
appropriate scenario, and use the same approach to justify a well-known
connection between regularizers and priors. We also show that the PDG
inconsistency captures a large class of statistical divergences, and detail
benefits of thinking of them in this way, including an intuitive visual
language for deriving inequalities between them. In variational inference, we
find that the ELBO, a somewhat opaque objective for latent variable models, and
variants of it arise for free out of uncontroversial modeling assumptions -- as
do simple graphical proofs of their corresponding bounds. Finally, we observe
that inconsistency becomes the log partition function (free energy) in the
setting where PDGs are factor graphs.
Related papers
- Inference for Probabilistic Dependency Graphs [42.03917543423699]
Probabilistic dependency graphs (PDGs) are a flexible class of probabilistic models.
We present the first tractable inference algorithm for PDGs with discrete variables.
arXiv Detail & Related papers (2023-11-09T18:40:12Z) - Nonparametric Identifiability of Causal Representations from Unknown
Interventions [63.1354734978244]
We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables.
Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
arXiv Detail & Related papers (2023-06-01T10:51:58Z) - Gaussian Process Probes (GPP) for Uncertainty-Aware Probing [61.91898698128994]
We introduce a unified and simple framework for probing and measuring uncertainty about concepts represented by models.
Our experiments show it can (1) probe a model's representations of concepts even with a very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are to the model), and (3) detect out of distribution data using those uncertainty measures as well as classic methods do.
arXiv Detail & Related papers (2023-05-29T17:00:16Z) - Causal Transportability for Visual Recognition [70.13627281087325]
We show that standard classifiers fail because the association between images and labels is not transportable across settings.
We then show that the causal effect, which severs all sources of confounding, remains invariant across domains.
This motivates us to develop an algorithm to estimate the causal effect for image classification.
arXiv Detail & Related papers (2022-04-26T15:02:11Z) - Discovering Invariant Rationales for Graph Neural Networks [104.61908788639052]
Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features.
We propose a new strategy of discovering invariant rationale (DIR) to construct intrinsically interpretable GNNs.
arXiv Detail & Related papers (2022-01-30T16:43:40Z) - Identification of Latent Variables From Graphical Model Residuals [0.0]
We present a novel method to control for the latent space when estimating a DAG by iteratively deriving proxies for the latent space from the residuals of the inferred model.
We show that any improvement of prediction of an outcome is intrinsically capped and cannot rise beyond a certain limit as compared to the confounded model.
arXiv Detail & Related papers (2021-01-07T02:28:49Z) - Probabilistic Dependency Graphs [14.505867475659274]
We introduce Probabilistic Dependency Graphs (PDGs)
PDGs can capture inconsistent beliefs in a natural way.
We show how PDGs are an especially natural modeling tool.
arXiv Detail & Related papers (2020-12-19T22:29:49Z) - Probabilistic Circuits for Variational Inference in Discrete Graphical
Models [101.28528515775842]
Inference in discrete graphical models with variational methods is difficult.
Many sampling-based methods have been proposed for estimating Evidence Lower Bound (ELBO)
We propose a new approach that leverages the tractability of probabilistic circuit models, such as Sum Product Networks (SPN)
We show that selective-SPNs are suitable as an expressive variational distribution, and prove that when the log-density of the target model is aweighted the corresponding ELBO can be computed analytically.
arXiv Detail & Related papers (2020-10-22T05:04:38Z) - Accounting for Unobserved Confounding in Domain Generalization [107.0464488046289]
This paper investigates the problem of learning robust, generalizable prediction models from a combination of datasets.
Part of the challenge of learning robust models lies in the influence of unobserved confounders.
We demonstrate the empirical performance of our approach on healthcare data from different modalities.
arXiv Detail & Related papers (2020-07-21T08:18:06Z) - Query Training: Learning a Worse Model to Infer Better Marginals in
Undirected Graphical Models with Hidden Variables [11.985433487639403]
Probabilistic graphical models (PGMs) provide a compact representation of knowledge that can be queried in a flexible way.
We introduce query training (QT), a mechanism to learn a PGM that is optimized for the approximate inference algorithm that will be paired with it.
We demonstrate experimentally that QT can be used to learn a challenging 8-connected grid Markov random field with hidden variables.
arXiv Detail & Related papers (2020-06-11T20:34:32Z) - Interventions and Counterfactuals in Tractable Probabilistic Models:
Limitations of Contemporary Transformations [12.47276164048813]
We show that when transforming SPNs to a causal graph interventional reasoning reduces to computing marginal distributions.
We first provide an algorithm for constructing a causal graph from a PSDD, which introduces augmented variables.
arXiv Detail & Related papers (2020-01-29T15:45:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.