Information-Theoretic Measures on Lattices for High-Order Interactions

• URL: http://arxiv.org/abs/2408.07533v2
• Date: Thu, 10 Oct 2024 18:17:20 GMT
• Title: Information-Theoretic Measures on Lattices for High-Order Interactions
• Authors: Zhaolu Liu, Mauricio Barahona, Robert L. Peach,
• Abstract summary: We present a systematic framework that derives higher-order information-theoretic measures using lattice and operator function pairs. We show that many commonly used measures can be derived within this framework, however they are often restricted to sublattices of the partition lattice. To fully characterise all interactions among $d$ variables, we introduce the Streitberg Information, using generalisations of KL divergence as an operator function.
• Score: 0.7373617024876725
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Traditional models based solely on pairwise associations often fail to capture the complex statistical structure of multivariate data. Existing approaches for identifying information shared among groups of $d>3$ variables are frequently computationally intractable, asymmetric with respect to a target variable, or unable to account for all factorisations of the joint probability distribution. We present a systematic framework that derives higher-order information-theoretic measures using lattice and operator function pairs, whereby the lattice representing the algebraic relationships among variables, with operator functions that compute the measures over the lattice. We show that many commonly used measures can be derived within this framework, however they are often restricted to sublattices of the partition lattice, which prevents them from capturing all interactions when $d>3$. We also demonstrate that KL divergence, when used as an operator function, leads to unwanted cancellation of interactions for $d>3$. To fully characterise all interactions among $d$ variables, we introduce the Streitberg Information, using generalisations of KL divergence as an operator function, and defined over the full partition lattice. We validate Streitberg Information numerically on synthetic data, and illustrate its application in analysing complex interactions among stocks, decoding neural signals, and performing feature selection in machine learning.

Related papers

• Learning Divergence Fields for Shift-Robust Graph Representations [73.11818515795761]
  In this work, we propose a geometric diffusion model with learnable divergence fields for the challenging problem with interdependent data. We derive a new learning objective through causal inference, which can guide the model to learn generalizable patterns of interdependence that are insensitive across domains.
  arXiv  Detail & Related papers  (2024-06-07T14:29:21Z)
• iSCAN: Identifying Causal Mechanism Shifts among Nonlinear Additive Noise Models [48.33685559041322]
  This paper focuses on identifying the causal mechanism shifts in two or more related datasets over the same set of variables. Code implementing the proposed method is open-source and publicly available at https://github.com/kevinsbello/iSCAN.
  arXiv  Detail & Related papers  (2023-06-30T01:48:11Z)
• Interaction Measures, Partition Lattices and Kernel Tests for High-Order Interactions [1.9457612782595313]
  Non-trivial dependencies between groups of more than two variables can play a significant role in the analysis and modelling of such systems. We introduce a hierarchy of $d$-order ($d geq 2$) interaction measures, increasingly inclusive of possible factorisations of the joint probability distribution. We also establish mathematical links with lattice theory, which elucidate the derivation of the interaction measures and their composite permutation tests.
  arXiv  Detail & Related papers  (2023-06-01T16:59:37Z)
• High-Dimensional Undirected Graphical Models for Arbitrary Mixed Data [2.2871867623460207]
  In many applications data span variables of different types, whose principled joint analysis is nontrivial. Recent advances have shown how the binary-continuous case can be tackled, but the general mixed variable type regime remains challenging. We propose flexible and scalable methodology for data with variables of entirely general mixed type.
  arXiv  Detail & Related papers  (2022-11-21T18:21:31Z)
• Partial Counterfactual Identification from Observational and Experimental Data [83.798237968683]
  We develop effective Monte Carlo algorithms to approximate the optimal bounds from an arbitrary combination of observational and experimental data. Our algorithms are validated extensively on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-10-12T02:21:30Z)
• Link Prediction on N-ary Relational Data Based on Relatedness Evaluation [61.61555159755858]
  We propose a method called NaLP to conduct link prediction on n-ary relational data. We represent each n-ary relational fact as a set of its role and role-value pairs. Experimental results validate the effectiveness and merits of the proposed methods.
  arXiv  Detail & Related papers  (2021-04-21T09:06:54Z)
• The Role of Mutual Information in Variational Classifiers [47.10478919049443]
  We study the generalization error of classifiers relying on encodings trained on the cross-entropy loss. We derive bounds to the generalization error showing that there exists a regime where the generalization error is bounded by the mutual information.
  arXiv  Detail & Related papers  (2020-10-22T12:27:57Z)
• Tractable Inference in Credal Sentential Decision Diagrams [116.6516175350871]
  Probabilistic sentential decision diagrams are logic circuits where the inputs of disjunctive gates are annotated by probability values. We develop the credal sentential decision diagrams, a generalisation of their probabilistic counterpart that allows for replacing the local probabilities with credal sets of mass functions. For a first empirical validation, we consider a simple application based on noisy seven-segment display images.
  arXiv  Detail & Related papers  (2020-08-19T16:04:34Z)
• Multi-Partition Embedding Interaction with Block Term Format for Knowledge Graph Completion [3.718476964451589]
  Knowledge graph embedding methods perform the task by representing entities and relations as embedding vectors. Previous work has usually treated each embedding as a whole and has modeled the interactions between these whole embeddings. We propose the multi- partition embedding interaction (MEI) model with block term format to address this problem.
  arXiv  Detail & Related papers  (2020-06-29T20:37:11Z)
• Robust Generalization via $\alpha$-Mutual Information [24.40306100502023]
  bounds connecting two probability measures of the same event using R'enyi $alpha$-Divergences and Sibson's $alpha$-Mutual Information. Results have broad applications, from bounding the generalization error of learning algorithms to the more general framework of adaptive data analysis.
  arXiv  Detail & Related papers  (2020-01-14T11:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.