Abstraction requires breadth: a renormalisation group approach

• URL: http://arxiv.org/abs/2407.01656v3
• Date: Wed, 19 Feb 2025 10:27:03 GMT
• Title: Abstraction requires breadth: a renormalisation group approach
• Authors: Carlo Orientale Caputo, Elias Seiffert, Matteo Marsili,
• Abstract summary: We argue that the level of abstraction depends crucially on how broad the training set is. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for an abstract representation.
• Score: 0.0
• License:
• Abstract: Abstraction is the process of extracting the essential features from raw data while ignoring irrelevant details. This is similar to the process of focusing on large-scale properties, systematically removing irrelevant small-scale details, implemented in the renormalisation group of statistical physics. This analogy is suggestive because the fixed points of the renormalisation group offer an ideal candidate of a truly abstract -- i.e. data independent -- representation. It has been observed that abstraction emerges with depth in neural networks. Deep layers of neural network capture abstract characteristics of data, such as "cat-ness" or "dog-ness" in images, by combining the lower level features encoded in shallow layers (e.g. edges). Yet we argue that depth alone is not enough to develop truly abstract representations. We advocate that the level of abstraction crucially depends on how broad the training set is. We address the issue within a renormalisation group approach where a representation is expanded to encompass a broader set of data. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for an abstract representation. This theoretical picture is tested in numerical experiments based on Deep Belief Networks trained on data of different breadth. These show that representations in deep layers of neural networks approach the Hierarchical Feature Model as the data gets broader, in agreement with theoretical predictions.

Related papers

• Robust Shape Fitting for 3D Scene Abstraction [33.84212609361491]
  In particular, we can describe man-made environments using volumetric primitives such as cuboids or cylinders. We propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.
  arXiv  Detail & Related papers  (2024-03-15T16:37:43Z)
• Neural Causal Abstractions [63.21695740637627]
  We develop a new family of causal abstractions by clustering variables and their domains. We show that such abstractions are learnable in practical settings through Neural Causal Models. Our experiments support the theory and illustrate how to scale causal inferences to high-dimensional settings involving image data.
  arXiv  Detail & Related papers  (2024-01-05T02:00:27Z)
• Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
  Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We show that linear networks make provably optimal predictions at infinite depth. We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
  arXiv  Detail & Related papers  (2022-12-29T20:57:46Z)
• ALSO: Automotive Lidar Self-supervision by Occupancy estimation [70.70557577874155]
  We propose a new self-supervised method for pre-training the backbone of deep perception models operating on point clouds. The core idea is to train the model on a pretext task which is the reconstruction of the surface on which the 3D points are sampled. The intuition is that if the network is able to reconstruct the scene surface, given only sparse input points, then it probably also captures some fragments of semantic information.
  arXiv  Detail & Related papers  (2022-12-12T13:10:19Z)
• Primitive-based Shape Abstraction via Nonparametric Bayesian Inference [29.7543198254021]
  We propose a novel non-parametric Bayesian statistical method to infer an abstraction, consisting of an unknown number of geometric primitives, from a point cloud. Our method outperforms the state-of-the-art in terms of accuracy and is generalizable to various types of objects.
  arXiv  Detail & Related papers  (2022-03-28T13:00:06Z)
• Online Deep Learning based on Auto-Encoder [4.128388784932455]
  We propose a two-phase Online Deep Learning based on Auto-Encoder (ODLAE) Based on auto-encoder, considering reconstruction loss, we extract abstract hierarchical latent representations of instances. We devise two fusion strategies: the output-level fusion strategy, which is obtained by fusing the classification results of each hidden layer; and feature-level fusion strategy, which is leveraged self-attention mechanism to fusion every hidden layer output.
  arXiv  Detail & Related papers  (2022-01-19T02:14:57Z)
• Learning Debiased and Disentangled Representations for Semantic Segmentation [52.35766945827972]
  We propose a model-agnostic and training scheme for semantic segmentation. By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes. Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-10-31T16:15:09Z)
• A Light-weight Interpretable CompositionalNetwork for Nuclei Detection and Weakly-supervised Segmentation [10.196621315018884]
  Deep neural networks usually require large numbers of annotated data to train vast parameters. We propose to build a data-efficient model, which only requires partial annotation, specifically on isolated nucleus.
  arXiv  Detail & Related papers  (2021-10-26T16:44:08Z)
• Reasoning-Modulated Representations [85.08205744191078]
  We study a common setting where our task is not purely opaque. Our approach paves the way for a new class of data-efficient representation learning.
  arXiv  Detail & Related papers  (2021-07-19T13:57:13Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• Extracting Semantic Indoor Maps from Occupancy Grids [2.4214518935746185]
  We focus on the semantic mapping of indoor environments. We propose a method to extract an abstracted floor plan from typical grid maps using Bayesian reasoning. We demonstrate the effectiveness of the approach using real-world data.
  arXiv  Detail & Related papers  (2020-02-19T18:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.