Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks

• URL: http://arxiv.org/abs/2505.09326v1
• Date: Wed, 14 May 2025 12:24:22 GMT
• Title: Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks
• Authors: Vincent Abbott, Kotaro Kamiya, Gerard Glowacki, Yu Atsumi, Gioele Zardini, Yoshihiro Maruyama,
• Abstract summary: We use neural circuit diagrams to prove a general theorem related to deep learning algorithms.<n>We guide the development of a novel attention algorithm catered to the domain of gene regulatory networks.<n>Our kernel achieves comparable performance to the state-of-the-art, fine-tuned FlashAttention algorithm.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: How do we enable artificial intelligence models to improve themselves? This is central to exponentially improving generalized artificial intelligence models, which can improve their own architecture to handle new problem domains in an efficient manner that leverages the latest hardware. However, current automated compilation methods are poor, and efficient algorithms require years of human development. In this paper, we use neural circuit diagrams, based in category theory, to prove a general theorem related to deep learning algorithms, guide the development of a novel attention algorithm catered to the domain of gene regulatory networks, and produce a corresponding efficient kernel. The algorithm we propose, spherical attention, shows that neural circuit diagrams enable a principled and systematic method for reasoning about deep learning architectures and providing high-performance code. By replacing SoftMax with an $L^2$ norm as suggested by diagrams, it overcomes the special function unit bottleneck of standard attention while retaining the streaming property essential to high-performance. Our diagrammatically derived \textit{FlashSign} kernel achieves comparable performance to the state-of-the-art, fine-tuned FlashAttention algorithm on an A100, and $3.6\times$ the performance of PyTorch. Overall, this investigation shows neural circuit diagrams' suitability as a high-level framework for the automated development of efficient, novel artificial intelligence architectures.

Related papers

• Learning neuro-symbolic convergent term rewriting systems [47.129504708849446]
  We introduce a general framework for learning convergent term rewriting systems using a neuro-symbolic architecture inspired by the rewriting algorithm itself.<n>We present two modular implementations of such architecture: the Neural Rewriting System (NRS) and the Fast Neural Rewriting System (FastNRS)
  arXiv  Detail & Related papers  (2025-07-25T15:24:56Z)
• NAR-*ICP: Neural Execution of Classical ICP-based Pointcloud Registration Algorithms [7.542220697870245]
  This study explores the intersection of neural networks and classical robotics algorithms through the Neural Algorithmic Reasoning framework. We propose a Graph Neural Network (GNN)-based learning framework, NAR-*ICP, which learns the intermediate algorithmic steps of classical ICP-based pointcloud registration algorithms. We evaluate our approach across diverse datasets, from real-world to synthetic, demonstrating its flexibility in handling complex and noisy inputs.
  arXiv  Detail & Related papers  (2024-10-14T19:33:46Z)
• Principled Architecture-aware Scaling of Hyperparameters [69.98414153320894]
  Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture. We demonstrate that network rankings can be easily changed by better training networks in benchmarks.
  arXiv  Detail & Related papers  (2024-02-27T11:52:49Z)
• Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
  We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations. Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
  arXiv  Detail & Related papers  (2024-02-20T15:23:24Z)
• Input Convex Lipschitz RNN: A Fast and Robust Approach for Engineering Tasks [14.835081385422653]
  We introduce a novel network architecture, termed Input Convex Lipschitz Recurrent Neural Networks (ICLRNNs)<n>This architecture seamlessly integrates the benefits of convexity and Lipschitz continuity, enabling fast and robust neural network-based modeling and optimization.<n>It has been successfully applied to practical engineering scenarios, such as modeling and control of chemical process and the modeling and real-world solar irradiance prediction for solar PV system planning.
  arXiv  Detail & Related papers  (2024-01-15T06:26:53Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Neuroevolution of Recurrent Architectures on Control Tasks [3.04585143845864]
  We implement a massively parallel evolutionary algorithm and run experiments on all 19 OpenAI Gym state-based reinforcement learning control tasks. We find that dynamic agents match or exceed the performance of gradient-based agents while utilizing orders of magnitude fewer parameters.
  arXiv  Detail & Related papers  (2023-04-03T16:29:18Z)
• Inducing Gaussian Process Networks [80.40892394020797]
  We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains. We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-21T05:27:09Z)
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence [3.874729481138221]
  Moore's law has driven exponential computing power expectations, its nearing end calls for new avenues for improving the overall system performance. One of these avenues is the exploration of alternative brain-inspired computing architectures that aim at achieving the flexibility and computational efficiency of biological neural processing systems. We provide a comprehensive overview of the field, highlighting the different levels of granularity at which this paradigm shift is realized.
  arXiv  Detail & Related papers  (2021-06-02T16:51:45Z)
• Graph Neural Networks for Scalable Radio Resource Management: Architecture Design and Theoretical Analysis [31.372548374969387]
  We propose to apply graph neural networks (GNNs) to solve large-scale radio resource management problems. The proposed method is highly scalable and can solve the beamforming problem in an interference channel with $1000$ transceiver pairs within $6$ milliseconds on a single GPU.
  arXiv  Detail & Related papers  (2020-07-15T11:43:32Z)
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning [56.83172249278467]
  We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces. We train and validate our approach directly on the Intel NNP-I chip for inference. We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
  arXiv  Detail & Related papers  (2020-07-14T18:50:12Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)
• Spiking Neural Networks Hardware Implementations and Challenges: a Survey [53.429871539789445]
  Spiking Neural Networks are cognitive algorithms mimicking neuron and synapse operational principles. We present the state of the art of hardware implementations of spiking neural networks. We discuss the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level.
  arXiv  Detail & Related papers  (2020-05-04T13:24:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.