Identifying Information from Observations with Uncertainty and Novelty

• URL: http://arxiv.org/abs/2501.09331v2
• Date: Wed, 09 Apr 2025 01:11:06 GMT
• Title: Identifying Information from Observations with Uncertainty and Novelty
• Abstract summary: This paper formalizes the notion of identifiable information that arises from the language used to express the relationship between distinct states.<n>We prove that computable PAC-Bayes learners' sample complexity distribution is determined by its moments in terms of the prior probability distribution over a fixed finite hypothesis set.
• Score: 7.088298584035036
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A machine learning tasks from observations must encounter and process uncertainty and novelty, especially when it is to maintain performance when observing new information and to choose the hypothesis that best fits the current observations. In this context, some key questions arise: what and how much information did the observations provide, how much information is required to identify the data-generating process, how many observations remain to get that information, and how does a predictor determine that it has observed novel information? This paper strengthens existing answers to these questions by formalizing the notion of identifiable information that arises from the language used to express the relationship between distinct states. Model identifiability and sample complexity are defined via computation of an indicator function over a set of hypotheses, bridging algorithmic and probabilistic information. Their properties and asymptotic statistics are described for data-generating processes ranging from deterministic processes to ergodic stationary stochastic processes. This connects the notion of identifying information in finite steps with asymptotic statistics and PAC-learning. The indicator function's computation naturally formalizes novel information and its identification from observations with respect to a hypothesis set. We also proved that computable PAC-Bayes learners' sample complexity distribution is determined by its moments in terms of the prior probability distribution over a fixed finite hypothesis set.

Related papers

• Speed Limit for Information Acquisition in Stochastic Learning Dynamics [0.0]
  We derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information.<n>The resulting inequality decomposes the information flow into drift and noise contributions.
  arXiv  Detail & Related papers  (2026-09-08T04:02:52Z)
• Computational Identifiability [58.12019389497595]
  'Identical identifiability' assumes properties, infinite data, or other mathematically idealized conditions.<n>'Computational identifiability' is to instead define a finite computational search procedure for an empirical estimator.<n>We show how this framework allows us to answer fine-grained, practical identification questions.
  arXiv  Detail & Related papers  (2026-06-08T19:39:59Z)
• Causal Representation Meets Stochastic Modeling under Generic Geometry [49.24293444627916]
  We develop causal representation learning for continuous-time latent point processes.<n>We develop MUTATE, an identifiable variational autoencoder framework with a time-adaptive transition module.<n>Across simulated and empirical studies, we find that MUTATE can effectively answer scientific questions.
  arXiv  Detail & Related papers  (2026-02-04T20:40:53Z)
• Simulation-based Bayesian inference with ameliorative learned summary statistics -- Part I [0.0]
  This paper considers a simulation-based inference with learned summary statistics.<n>The exact likelihood function associated with the observation data and the simulation model is difficult to obtain in a closed form or computationally intractable.
  arXiv  Detail & Related papers  (2026-01-30T01:21:11Z)
• From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence [91.54446789584826]
  Epiplexity is a formalization of information capturing what computationally bounded observers can learn from data.<n>We show how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself.
  arXiv  Detail & Related papers  (2026-01-06T18:04:03Z)
• Rademacher learning rates for iterated random functions [0.0]
  We consider the case where the training dataset is generated by an iterated random function that is not necessarily irreducible or aperiodic.<n>Under the assumption that the governing function is contractive with respect to its first argument, we first establish a uniform convergence result for the corresponding sample error.<n>We then demonstrate the learnability of the approximate empirical risk minimization algorithm and derive its learning rate bound.
  arXiv  Detail & Related papers  (2025-06-16T19:36:13Z)
• Comparing the information content of probabilistic representation spaces [3.7277730514654555]
  Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. We propose two information-theoretic measures to compare general probabilistic representation spaces. We demonstrate the utility of these measures in three case studies.
  arXiv  Detail & Related papers  (2024-05-31T17:33:07Z)
• On the Identification of Temporally Causal Representation with Instantaneous Dependence [50.14432597910128]
  Temporally causal representation learning aims to identify the latent causal process from time series observations. Most methods require the assumption that the latent causal processes do not have instantaneous relations. We propose an textbfIDentification framework for instantanetextbfOus textbfLatent dynamics.
  arXiv  Detail & Related papers  (2024-05-24T08:08:05Z)
• On the Properties and Estimation of Pointwise Mutual Information Profiles [49.877314063833296]
  The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. We introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods.
  arXiv  Detail & Related papers  (2023-10-16T10:02:24Z)
• LaPLACE: Probabilistic Local Model-Agnostic Causal Explanations [1.0370398945228227]
  We introduce LaPLACE-explainer, designed to provide probabilistic cause-and-effect explanations for machine learning models. The LaPLACE-Explainer component leverages the concept of a Markov blanket to establish statistical boundaries between relevant and non-relevant features. Our approach offers causal explanations and outperforms LIME and SHAP in terms of local accuracy and consistency of explained features.
  arXiv  Detail & Related papers  (2023-10-01T04:09:59Z)
• On the Joint Interaction of Models, Data, and Features [82.60073661644435]
  We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
  arXiv  Detail & Related papers  (2023-06-07T21:35:26Z)
• Mutual Information Estimation via $f$-Divergence and Data Derangements [6.43826005042477]
  We propose a novel class of discrimi mutual information estimators based on the variational representation of the $f$-divergence. The proposed estimator is flexible since it exhibits an excellent bias/ variance trade-off.
  arXiv  Detail & Related papers  (2023-05-31T16:54:25Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• Inferential Moments of Uncertain Multivariable Systems [0.0]
  We treat Bayesian probability updating as a random process and uncover intrinsic quantitative features of joint probability distributions called inferential moments. Inferential moments quantify shape information about how a prior distribution is expected to update in response to yet to be obtained information. We find a power series expansion of the mutual information in terms of inferential moments, which implies a connection between inferential theoretic logic and elements of information theory.
  arXiv  Detail & Related papers  (2023-05-03T00:56:12Z)
• Identifying Weight-Variant Latent Causal Models [82.14087963690561]
  We find that transitivity acts as a key role in impeding the identifiability of latent causal representations. Under some mild assumptions, we can show that the latent causal representations can be identified up to trivial permutation and scaling. We propose a novel method, termed Structural caUsAl Variational autoEncoder, which directly learns latent causal representations and causal relationships among them.
  arXiv  Detail & Related papers  (2022-08-30T11:12:59Z)
• Visualizing Confidence Intervals for Critical Point Probabilities in 2D Scalar Field Ensembles [7.484221280249876]
  We present an approach for the computation and visual representation of confidence intervals for the occurrence probabilities of critical points in ensemble data sets. We demonstrate the added value of our approach over existing methods for critical point prediction in uncertain data on a synthetic data set and show its applicability to a data set from climate research.
  arXiv  Detail & Related papers  (2022-07-13T12:54:27Z)
• A Bayesian Framework for Information-Theoretic Probing [51.98576673620385]
  We argue that probing should be seen as approximating a mutual information. This led to the rather unintuitive conclusion that representations encode exactly the same information about a target task as the original sentences. This paper proposes a new framework to measure what we term Bayesian mutual information.
  arXiv  Detail & Related papers  (2021-09-08T18:08:36Z)
• Efficient nonparametric statistical inference on population feature importance using Shapley values [7.6146285961466]
  We present a procedure for estimating and obtaining valid statistical inference on the Shapley Population Variable Importance Measure (SPVIM) Although the computational complexity of the true SPVIM exponentially with the number of variables, we propose an estimator based on randomly sampling only $Theta(n)$ feature subsets given $n$ observations. Our procedure has good finite-sample performance in simulations, and for an in-hospital prediction task produces similar variable importance estimates when different machine learning algorithms are applied.
  arXiv  Detail & Related papers  (2020-06-16T19:47:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.