Entropy, Information, and the Updating of Probabilities

• URL: http://arxiv.org/abs/2107.04529v1
• Date: Fri, 9 Jul 2021 16:27:23 GMT
• Title: Entropy, Information, and the Updating of Probabilities
• Authors: Ariel Caticha
• Abstract summary: This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The ME method goes beyond the mere selection of a single posterior, but also addresses the question of how much less probable other distributions might be.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes the pragmatic elements in the derivation. An epistemic notion of information is defined in terms of its relation to the Bayesian beliefs of ideally rational agents. The method of updating from a prior to a posterior probability distribution is designed through an eliminative induction process. The logarithmic relative entropy is singled out as the unique tool for updating that (a) is of universal applicability; (b) that recognizes the value of prior information; and (c) that recognizes the privileged role played by the notion of independence in science. The resulting framework -- the ME method -- can handle arbitrary priors and arbitrary constraints. It includes MaxEnt and Bayes' rule as special cases and, therefore, it unifies entropic and Bayesian methods into a single general inference scheme. The ME method goes beyond the mere selection of a single posterior, but also addresses the question of how much less probable other distributions might be, which provides a direct bridge to the theories of fluctuations and large deviations.

Related papers

• Uncertainty Quantification with Bayesian Higher Order ReLU KANs [0.0]
  We introduce the first method of uncertainty quantification in the domain of Kolmogorov-Arnold Networks, specifically focusing on (Higher Order) ReLUKANs. We validate our method through a series of closure tests, including simple one-dimensional functions. We demonstrate the method's ability to correctly identify functional dependencies introduced through the inclusion of a term.
  arXiv  Detail & Related papers  (2024-10-02T15:57:18Z)
• Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models [0.0]
  We show how diffusion-based models can be repurposed for performing principled, identifiable Bayesian inference. We show how such maps can be learned via standard DBM training using a novel noise schedule. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space.
  arXiv  Detail & Related papers  (2024-07-11T19:58:19Z)
• Bayesian Renormalization [68.8204255655161]
  We present a fully information theoretic approach to renormalization inspired by Bayesian statistical inference. The main insight of Bayesian Renormalization is that the Fisher metric defines a correlation length that plays the role of an emergent RG scale. We provide insight into how the Bayesian Renormalization scheme relates to existing methods for data compression and data generation.
  arXiv  Detail & Related papers  (2023-05-17T18:00:28Z)
• Inferential Moments of Uncertain Multivariable Systems [0.0]
  We treat Bayesian probability updating as a random process and uncover intrinsic quantitative features of joint probability distributions called inferential moments. Inferential moments quantify shape information about how a prior distribution is expected to update in response to yet to be obtained information. We find a power series expansion of the mutual information in terms of inferential moments, which implies a connection between inferential theoretic logic and elements of information theory.
  arXiv  Detail & Related papers  (2023-05-03T00:56:12Z)
• Information-Theoretic Safe Exploration with Gaussian Processes [89.31922008981735]
  We consider a sequential decision making task where we are not allowed to evaluate parameters that violate an unknown (safety) constraint. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. We propose an information-theoretic safe exploration criterion that directly exploits the GP posterior to identify the most informative safe parameters to evaluate.
  arXiv  Detail & Related papers  (2022-12-09T15:23:58Z)
• Information Theory with Kernel Methods [0.0]
  We show that the von Neumann entropy and relative entropy of these operators are intimately related to the usual notions of Shannon entropy and relative entropy. They come together with efficient estimation algorithms from various oracles on the probability distributions.
  arXiv  Detail & Related papers  (2022-02-17T09:42:29Z)
• Variational Causal Networks: Approximate Bayesian Inference over Causal Structures [132.74509389517203]
  We introduce a parametric variational family modelled by an autoregressive distribution over the space of discrete DAGs. In experiments, we demonstrate that the proposed variational posterior is able to provide a good approximation of the true posterior.
  arXiv  Detail & Related papers  (2021-06-14T17:52:49Z)
• BayesIMP: Uncertainty Quantification for Causal Data Fusion [52.184885680729224]
  We study the causal data fusion problem, where datasets pertaining to multiple causal graphs are combined to estimate the average treatment effect of a target variable. We introduce a framework which combines ideas from probabilistic integration and kernel mean embeddings to represent interventional distributions in the reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2021-06-07T10:14:18Z)
• Parametrization invariant interpretation of priors and posteriors [0.0]
  We move away from the idea that "a prior distribution establishes a probability distribution over the parameters of our model" to the idea that "a prior distribution establishes a probability distribution over probability distributions" Under this mindset, any distribution over probability distributions should be "intrinsic", that is, invariant to the specific parametrization which is selected for the manifold.
  arXiv  Detail & Related papers  (2021-05-18T06:45:05Z)
• Causal Expectation-Maximisation [70.45873402967297]
  We show that causal inference is NP-hard even in models characterised by polytree-shaped graphs. We introduce the causal EM algorithm to reconstruct the uncertainty about the latent variables from data about categorical manifest variables. We argue that there appears to be an unnoticed limitation to the trending idea that counterfactual bounds can often be computed without knowledge of the structural equations.
  arXiv  Detail & Related papers  (2020-11-04T10:25:13Z)
• Polynomial-Time Exact MAP Inference on Discrete Models with Global Dependencies [83.05591911173332]
  junction tree algorithm is the most general solution for exact MAP inference with run-time guarantees. We propose a new graph transformation technique via node cloning which ensures a run-time for solving our target problem independently of the form of a corresponding clique tree.
  arXiv  Detail & Related papers  (2019-12-27T13:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.