Group Crosscoders for Mechanistic Analysis of Symmetry

• URL: http://arxiv.org/abs/2410.24184v2
• Date: Fri, 01 Nov 2024 03:29:29 GMT
• Title: Group Crosscoders for Mechanistic Analysis of Symmetry
• Authors: Liv Gorton,
• Abstract summary: Group crosscoders systematically discover and analyse symmetrical features in neural networks. We show that group crosscoders can provide systematic insights into how neural networks represent symmetry.
• Score: 0.0
• License:
• Abstract: We introduce group crosscoders, an extension of crosscoders that systematically discover and analyse symmetrical features in neural networks. While neural networks often develop equivariant representations without explicit architectural constraints, understanding these emergent symmetries has traditionally relied on manual analysis. Group crosscoders automate this process by performing dictionary learning across transformed versions of inputs under a symmetry group. Applied to InceptionV1's mixed3b layer using the dihedral group $\mathrm{D}_{32}$, our method reveals several key insights: First, it naturally clusters features into interpretable families that correspond to previously hypothesised feature types, providing more precise separation than standard sparse autoencoders. Second, our transform block analysis enables the automatic characterisation of feature symmetries, revealing how different geometric features (such as curves versus lines) exhibit distinct patterns of invariance and equivariance. These results demonstrate that group crosscoders can provide systematic insights into how neural networks represent symmetry, offering a promising new tool for mechanistic interpretability.

Related papers

• Symmetry Discovery for Different Data Types [52.2614860099811]
  Equivariant neural networks incorporate symmetries into their architecture, achieving higher generalization performance. We propose LieSD, a method for discovering symmetries via trained neural networks which approximate the input-output mappings of the tasks. We validate the performance of LieSD on tasks with symmetries such as the two-body problem, the moment of inertia matrix prediction, and top quark tagging.
  arXiv  Detail & Related papers  (2024-10-13T13:39:39Z)
• Symmetry From Scratch: Group Equivariance as a Supervised Learning Task [1.8570740863168362]
  In machine learning datasets with symmetries, the paradigm for backward compatibility with symmetry-breaking has been to relax equivariant architectural constraints. We introduce symmetry-cloning, a method for inducing equivariance in machine learning models.
  arXiv  Detail & Related papers  (2024-10-05T00:44:09Z)
• Stochastic Neural Network Symmetrisation in Markov Categories [2.0668277618112203]
  We consider the problem of symmetrising a neural network along a group homomorphism. We obtain a flexible, compositional, and generic framework for symmetrisation.
  arXiv  Detail & Related papers  (2024-06-17T17:54:42Z)
• Enhancing lattice kinetic schemes for fluid dynamics with Lattice-Equivariant Neural Networks [79.16635054977068]
  We present a new class of equivariant neural networks, dubbed Lattice-Equivariant Neural Networks (LENNs) Our approach develops within a recently introduced framework aimed at learning neural network-based surrogate models Lattice Boltzmann collision operators. Our work opens towards practical utilization of machine learning-augmented Lattice Boltzmann CFD in real-world simulations.
  arXiv  Detail & Related papers  (2024-05-22T17:23:15Z)
• Unsupervised Learning of Invariance Transformations [105.54048699217668]
  We develop an algorithmic framework for finding approximate graph automorphisms. We discuss how this framework can be used to find approximate automorphisms in weighted graphs in general.
  arXiv  Detail & Related papers  (2023-07-24T17:03:28Z)
• Learning Lie Group Symmetry Transformations with Neural Networks [17.49001206996365]
  This work focuses on discovering and characterizing unknown symmetries present in the dataset, namely, Lie group symmetry transformations. Our goal is to characterize the transformation group and the distribution of the parameter values. Results showcase the effectiveness of the approach in both these settings.
  arXiv  Detail & Related papers  (2023-07-04T09:23:24Z)
• Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles [55.41644538483948]
  We design a deep-learning algorithm for the discovery and identification of the continuous group of symmetries present in a labeled dataset. We use fully connected neural networks to model the transformations symmetry and the corresponding generators. Our study also opens the door for using a machine learning approach in the mathematical study of Lie groups and their properties.
  arXiv  Detail & Related papers  (2023-01-13T16:25:25Z)
• Equivariance with Learned Canonicalization Functions [77.32483958400282]
  We show that learning a small neural network to perform canonicalization is better than using predefineds. Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
  arXiv  Detail & Related papers  (2022-11-11T21:58:15Z)
• On the Symmetries of Deep Learning Models and their Internal Representations [1.418465438044804]
  We seek to connect the symmetries arising from the architecture of a family of models with the symmetries of that family's internal representation of data. Our work suggests that the symmetries of a network are propagated into the symmetries in that network's representation of data.
  arXiv  Detail & Related papers  (2022-05-27T22:29:08Z)
• Primal-Dual Mesh Convolutional Neural Networks [62.165239866312334]
  We propose a primal-dual framework drawn from the graph-neural-network literature to triangle meshes. Our method takes features for both edges and faces of a 3D mesh as input and dynamically aggregates them. We provide theoretical insights of our approach using tools from the mesh-simplification literature.
  arXiv  Detail & Related papers  (2020-10-23T14:49:02Z)
• Detecting Symmetries with Neural Networks [0.0]
  We make extensive use of the structure in the embedding layer of the neural network. We identify whether a symmetry is present and to identify orbits of the symmetry in the input. For this example we present a novel data representation in terms of graphs.
  arXiv  Detail & Related papers  (2020-03-30T17:58:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.