Koopman Autoencoders Learn Neural Representation Dynamics

• URL: http://arxiv.org/abs/2505.12809v1
• Date: Mon, 19 May 2025 07:35:43 GMT
• Title: Koopman Autoencoders Learn Neural Representation Dynamics
• Authors: Nishant Suresh Aswani, Saif Eddin Jabari,
• Abstract summary: We introduce Koopman autoencoders to capture how neural representations evolve through network layers.<n>Our approach learns a surrogate model that predicts how neural representations transform from input to output.<n>As a practical application, we show how our approach enables targeted class unlearning in the Yin-Yang and MNIST classification tasks.
• Score: 6.393645655578601
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper explores a simple question: can we model the internal transformations of a neural network using dynamical systems theory? We introduce Koopman autoencoders to capture how neural representations evolve through network layers, treating these representations as states in a dynamical system. Our approach learns a surrogate model that predicts how neural representations transform from input to output, with two key advantages. First, by way of lifting the original states via an autoencoder, it operates in a linear space, making editing the dynamics straightforward. Second, it preserves the topologies of the original representations by regularizing the autoencoding objective. We demonstrate that these surrogate models naturally replicate the progressive topological simplification observed in neural networks. As a practical application, we show how our approach enables targeted class unlearning in the Yin-Yang and MNIST classification tasks.

Related papers

• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Toward Neuromic Computing: Neurons as Autoencoders [0.0]
  This paper presents the idea that neural backpropagation is using dendritic processing to enable individual neurons to perform autoencoding. Using a very simple connection weight search and artificial neural network model, the effects of interleaving autoencoding for each neuron in a hidden layer of a feedforward network are explored.
  arXiv  Detail & Related papers  (2024-03-04T18:58:09Z)
• Neural Network Diffusion [45.851945143942885]
  A diffusion model is trained to synthesize latent representations from random noise.<n>This model then generates new representations, which are passed through the autoencoder's decoder to produce new subsets of high-performing network parameters.
  arXiv  Detail & Related papers  (2024-02-20T16:59:03Z)
• Generalizable Neural Fields as Partially Observed Neural Processes [16.202109517569145]
  We propose a new paradigm that views the large-scale training of neural representations as a part of a partially-observed neural process framework. We demonstrate that this approach outperforms both state-of-the-art gradient-based meta-learning approaches and hypernetwork approaches.
  arXiv  Detail & Related papers  (2023-09-13T01:22:16Z)
• NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning [25.197394237526865]
  We propose a modified Transformer-based universal neural network representation learning model NAR-Former V2. Specifically, we take the network as a graph and design a straightforward tokenizer to encode the network into a sequence. We incorporate the inductive representation learning capability of GNN into Transformer, enabling Transformer to generalize better when encountering unseen architecture.
  arXiv  Detail & Related papers  (2023-06-19T09:11:04Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• ConCerNet: A Contrastive Learning Based Framework for Automated Conservation Law Discovery and Trustworthy Dynamical System Prediction [82.81767856234956]
  This paper proposes a new learning framework named ConCerNet to improve the trustworthiness of the DNN based dynamics modeling. We show that our method consistently outperforms the baseline neural networks in both coordinate error and conservation metrics.
  arXiv  Detail & Related papers  (2023-02-11T21:07:30Z)
• Gaussian Process Surrogate Models for Neural Networks [6.8304779077042515]
  In science and engineering, modeling is a methodology used to understand complex systems whose internal processes are opaque. We construct a class of surrogate models for neural networks using Gaussian processes. We demonstrate our approach captures existing phenomena related to the spectral bias of neural networks, and then show that our surrogate models can be used to solve practical problems.
  arXiv  Detail & Related papers  (2022-08-11T20:17:02Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Neural Cellular Automata Manifold [84.08170531451006]
  We show that the neural network architecture of the Neural Cellular Automata can be encapsulated in a larger NN. This allows us to propose a new model that encodes a manifold of NCA, each of them capable of generating a distinct image. In biological terms, our approach would play the role of the transcription factors, modulating the mapping of genes into specific proteins that drive cellular differentiation.
  arXiv  Detail & Related papers  (2020-06-22T11:41:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.