S$Ω$I: Score-based O-INFORMATION Estimation

• URL: http://arxiv.org/abs/2402.05667v3
• Date: Fri, 7 Jun 2024 15:31:53 GMT
• Title: S$Ω$I: Score-based O-INFORMATION Estimation
• Authors: Mustapha Bounoua, Giulio Franzese, Pietro Michiardi,
• Abstract summary: We introduce S$Omega$I, which allows for the first time to compute O-information without restrictive assumptions about the system. Our experiments validate our approach on synthetic data, and demonstrate the effectiveness of S$Omega$I in the context of a real-world use case.
• Score: 7.399561232927219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The analysis of scientific data and complex multivariate systems requires information quantities that capture relationships among multiple random variables. Recently, new information-theoretic measures have been developed to overcome the shortcomings of classical ones, such as mutual information, that are restricted to considering pairwise interactions. Among them, the concept of information synergy and redundancy is crucial for understanding the high-order dependencies between variables. One of the most prominent and versatile measures based on this concept is O-information, which provides a clear and scalable way to quantify the synergy-redundancy balance in multivariate systems. However, its practical application is limited to simplified cases. In this work, we introduce S$\Omega$I, which allows for the first time to compute O-information without restrictive assumptions about the system. Our experiments validate our approach on synthetic data, and demonstrate the effectiveness of S$\Omega$I in the context of a real-world use case.

Related papers

• SyMANTIC: An Efficient Symbolic Regression Method for Interpretable and Parsimonious Model Discovery in Science and Beyond [3.4191590966148824]
  We introduce SyMANTIC, a novel Symbolic Regression (SR) algorithm. SyMANTIC efficiently identifies low-dimensional descriptors from a large set of candidates. We show that SyMANTIC uncovers similar or more accurate models at a fraction of the cost of existing SR methods.
  arXiv  Detail & Related papers  (2025-02-05T17:05:25Z)
• REMEDI: Corrective Transformations for Improved Neural Entropy Estimation [0.7488108981865708]
  We introduce $textttREMEDI$ for efficient and accurate estimation of differential entropy. Our approach demonstrates improvement across a broad spectrum of estimation tasks. It can be naturally extended to information theoretic supervised learning models.
  arXiv  Detail & Related papers  (2024-02-08T14:47:37Z)
• Max-Sliced Mutual Information [17.667315953598788]
  Quantifying the dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure. This work proposes a middle ground in the form of a scalable information-theoretic generalization of CCA, termed max-sliced mutual information (mSMI)
  arXiv  Detail & Related papers  (2023-09-28T06:49:25Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• Mutual Information Estimation via $f$-Divergence and Data Derangements [6.43826005042477]
  We propose a novel class of discrimi mutual information estimators based on the variational representation of the $f$-divergence. The proposed estimator is flexible since it exhibits an excellent bias/ variance trade-off.
  arXiv  Detail & Related papers  (2023-05-31T16:54:25Z)
• Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework [89.8609061423685]
  We propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks. We demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies.
  arXiv  Detail & Related papers  (2023-02-23T18:59:05Z)
• Variational Distillation for Multi-View Learning [104.17551354374821]
  We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
  arXiv  Detail & Related papers  (2022-06-20T03:09:46Z)
• Tracing Information Flow from Open Quantum Systems [52.77024349608834]
  We use photons in a waveguide array to implement a quantum simulation of the coupling of a qubit with a low-dimensional discrete environment. Using the trace distance between quantum states as a measure of information, we analyze different types of information transfer.
  arXiv  Detail & Related papers  (2021-03-22T16:38:31Z)
• Brain Image Synthesis with Unsupervised Multivariate Canonical CSC$\ell_4$Net [122.8907826672382]
  We propose to learn dedicated features that cross both intre- and intra-modal variations using a novel CSC$ell_4$Net.
  arXiv  Detail & Related papers  (2021-03-22T05:19:40Z)
• Information Theory Measures via Multidimensional Gaussianization [7.788961560607993]
  Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications. However, obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality.
  arXiv  Detail & Related papers  (2020-10-08T07:22:16Z)
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates [70.9701218475002]
  We introduce a unified convergence analysis of decentralized communication methods. We derive universal convergence rates for several applications. Our proofs rely on weak assumptions.
  arXiv  Detail & Related papers  (2020-03-23T17:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.