When do neural networks learn world models?

• URL: http://arxiv.org/abs/2502.09297v2
• Date: Thu, 20 Feb 2025 07:33:20 GMT
• Title: When do neural networks learn world models?
• Authors: Tianren Zhang, Guanyu Chen, Feng Chen,
• Abstract summary: We study whether neural networks can learn similar world models. We show that models with a low-degree bias provably recover latent data-generating variables under mild assumptions.
• Score: 5.998374495575507
• License:
• Abstract: Humans develop world models that capture the underlying generation process of data. Whether neural networks can learn similar world models remains an open problem. In this work, we provide the first theoretical results for this problem, showing that in a multi-task setting, models with a low-degree bias provably recover latent data-generating variables under mild assumptions -- even if proxy tasks involve complex, non-linear functions of the latents. However, such recovery is also sensitive to model architecture. Our analysis leverages Boolean models of task solutions via the Fourier-Walsh transform and introduces new techniques for analyzing invertible Boolean transforms, which may be of independent interest. We illustrate the algorithmic implications of our results and connect them to related research areas, including self-supervised learning, out-of-distribution generalization, and the linear representation hypothesis in large language models.

Related papers

• Latent Variable Sequence Identification for Cognitive Models with Neural Network Estimators [7.7227297059345466]
  We present an approach that extends neural Bayes estimation to learn a direct mapping between experimental data and the targeted latent variable space. Our work underscores that combining recurrent neural networks and simulation-based inference to identify latent variable sequences can enable researchers to access a wider class of cognitive models.
  arXiv  Detail & Related papers  (2024-06-20T21:13:39Z)
• Learning Divergence Fields for Shift-Robust Graph Representations [73.11818515795761]
  In this work, we propose a geometric diffusion model with learnable divergence fields for the challenging problem with interdependent data. We derive a new learning objective through causal inference, which can guide the model to learn generalizable patterns of interdependence that are insensitive across domains.
  arXiv  Detail & Related papers  (2024-06-07T14:29:21Z)
• Generative Learning of Continuous Data by Tensor Networks [45.49160369119449]
  We introduce a new family of tensor network generative models for continuous data. We benchmark the performance of this model on several synthetic and real-world datasets. Our methods give important theoretical and empirical evidence of the efficacy of quantum-inspired methods for the rapidly growing field of generative learning.
  arXiv  Detail & Related papers  (2023-10-31T14:37:37Z)
• On sparse regression, Lp-regularization, and automated model discovery [0.0]
  We show that Lp regularized neural networks can simultaneously discover both, interpretable models and physically meaningful parameters. Our ability to automatically discover material models from data could have tremendous applications in generative material design.
  arXiv  Detail & Related papers  (2023-10-09T05:34:21Z)
• On the Trade-off Between Efficiency and Precision of Neural Abstraction [62.046646433536104]
  Neural abstractions have been recently introduced as formal approximations of complex, nonlinear dynamical models. We employ formal inductive synthesis procedures to generate neural abstractions that result in dynamical models with these semantics.
  arXiv  Detail & Related papers  (2023-07-28T13:22:32Z)
• EINNs: Epidemiologically-Informed Neural Networks [75.34199997857341]
  We introduce a new class of physics-informed neural networks-EINN-crafted for epidemic forecasting. We investigate how to leverage both the theoretical flexibility provided by mechanistic models as well as the data-driven expressability afforded by AI models.
  arXiv  Detail & Related papers  (2022-02-21T18:59:03Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Closed-form Continuous-Depth Models [99.40335716948101]
  Continuous-depth neural models rely on advanced numerical differential equation solvers. We present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster.
  arXiv  Detail & Related papers  (2021-06-25T22:08:51Z)
• Learning Variational Data Assimilation Models and Solvers [34.22350850350653]
  We introduce end-to-end neural network architectures for data assimilation. A key feature of the proposed end-to-end learning architecture is that we may train the NN models using both supervised and unsupervised strategies.
  arXiv  Detail & Related papers  (2020-07-25T14:28:48Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.