Learning Discrete Concepts in Latent Hierarchical Models

• URL: http://arxiv.org/abs/2406.00519v1
• Date: Sat, 1 Jun 2024 18:01:03 GMT
• Title: Learning Discrete Concepts in Latent Hierarchical Models
• Authors: Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P. Xing, Yuejie Chi, Kun Zhang,
• Abstract summary: Learning concepts from natural high-dimensional data holds potential in building human-aligned and interpretable machine learning models. We formalize concepts as discrete latent causal variables that are related via a hierarchical causal model. We substantiate our theoretical claims with synthetic data experiments.
• Score: 73.01229236386148
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights.

Related papers

• Cross-Entropy Is All You Need To Invert the Data Generating Process [29.94396019742267]
  Empirical phenomena suggest that supervised models can learn interpretable factors of variation in a linear fashion. Recent advances in self-supervised learning have shown that these methods can recover latent structures by inverting the data generating process. We prove that even in standard classification tasks, models learn representations of ground-truth factors of variation up to a linear transformation.
  arXiv  Detail & Related papers  (2024-10-29T09:03:57Z)
• Towards Compositional Interpretability for XAI [3.3768167170511587]
  We present an approach to defining AI models and their interpretability based on category theory. We compare a wide range of AI models as compositional models. We find that what makes the standard 'intrinsically interpretable' models so transparent is brought out most clearly diagrammatically.
  arXiv  Detail & Related papers  (2024-06-25T14:27:03Z)
• Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective [60.64922606733441]
  We introduce a mathematical model that formalizes relational learning as hypergraph recovery to study pre-training of Foundation Models (FMs) In our framework, the world is represented as a hypergraph, with data abstracted as random samples from hyperedges. We theoretically examine the feasibility of a Pre-Trained Model (PTM) to recover this hypergraph and analyze the data efficiency in a minimax near-optimal style.
  arXiv  Detail & Related papers  (2024-06-17T06:20:39Z)
• A Recursive Bateson-Inspired Model for the Generation of Semantic Formal Concepts from Spatial Sensory Data [77.34726150561087]
  This paper presents a new symbolic-only method for the generation of hierarchical concept structures from complex sensory data. The approach is based on Bateson's notion of difference as the key to the genesis of an idea or a concept. The model is able to produce fairly rich yet human-readable conceptual representations without training.
  arXiv  Detail & Related papers  (2023-07-16T15:59:13Z)
• On the Joint Interaction of Models, Data, and Features [82.60073661644435]
  We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
  arXiv  Detail & Related papers  (2023-06-07T21:35:26Z)
• Provably Learning Object-Centric Representations [25.152680199034215]
  We analyze when object-centric representations can provably be learned without supervision. We prove that the ground-truth object representations can be identified by an invertible and compositional inference model. We provide evidence that our theory holds predictive power for existing object-centric models.
  arXiv  Detail & Related papers  (2023-05-23T16:44:49Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Contrastive Topographic Models: Energy-based density models applied to the understanding of sensory coding and cortical topography [9.555150216958246]
  We address the problem of building theoretical models that help elucidate the function of the visual brain at computational/algorithmic and structural/mechanistic levels.
  arXiv  Detail & Related papers  (2020-11-05T16:36:43Z)
• Causal Discovery in Physical Systems from Videos [123.79211190669821]
  Causal discovery is at the core of human cognition. We consider the task of causal discovery from videos in an end-to-end fashion without supervision on the ground-truth graph structure.
  arXiv  Detail & Related papers  (2020-07-01T17:29:57Z)
• A theory of independent mechanisms for extrapolation in generative models [20.794692397859755]
  Generative models can be trained to emulate complex empirical data, but are they useful to make predictions in the context of previously unobserved environments? We develop a theoretical framework to address this challenging situation by defining a weaker form of identifiability, based on the principle of independence of mechanisms. We demonstrate on toy examples that classical gradient descent can hinder the model's extrapolation capabilities, suggesting independence of mechanisms should be enforced explicitly during training.
  arXiv  Detail & Related papers  (2020-04-01T01:01:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.