Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinement

• URL: http://arxiv.org/abs/2505.18131v1
• Date: Fri, 23 May 2025 17:41:18 GMT
• Title: Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinement
• Authors: Jonas A. Actor, Graham Harper, Ben Southworth, Eric C. Cyr,
• Abstract summary: Multilayer perceptrons (MLPs) are a workhorse machine learning architecture, used in a variety of modern deep learning frameworks.<n>Recently Kolmogorov-Arnold Networks (KANs) have become increasingly popular due to their success on a range of problems.<n>In this paper, we exploit the relationship between KANs and multichannels to gain structural insight into how to trains faster.
• Score: 2.249916681499244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilayer perceptrons (MLPs) are a workhorse machine learning architecture, used in a variety of modern deep learning frameworks. However, recently Kolmogorov-Arnold Networks (KANs) have become increasingly popular due to their success on a range of problems, particularly for scientific machine learning tasks. In this paper, we exploit the relationship between KANs and multichannel MLPs to gain structural insight into how to train MLPs faster. We demonstrate the KAN basis (1) provides geometric localized support, and (2) acts as a preconditioned descent in the ReLU basis, overall resulting in expedited training and improved accuracy. Our results show the equivalence between free-knot spline KAN architectures, and a class of MLPs that are refined geometrically along the channel dimension of each weight tensor. We exploit this structural equivalence to define a hierarchical refinement scheme that dramatically accelerates training of the multi-channel MLP architecture. We show further accuracy improvements can be had by allowing the $1$D locations of the spline knots to be trained simultaneously with the weights. These advances are demonstrated on a range of benchmark examples for regression and scientific machine learning.

Related papers

• Improving Memory Efficiency for Training KANs via Meta Learning [55.24089119864207]
  We propose to generate weights for KANs via a smaller meta-learner, called MetaKANs.<n>By training KANs and MetaKANs in an end-to-end differentiable manner, MetaKANs achieve comparable or even superior performance.
  arXiv  Detail & Related papers  (2025-06-09T08:38:26Z)
• KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics [1.8434042562191815]
  Kurkova-Kolmogorov-Arnold Network (KKAN) is a new two-block architecture that combines robust multi-layer perceptron (MLP) based inner functions with flexible linear combinations of basis functions as outer functions.<n> benchmark results show that KKANs outperform original Kolmogorov-Arnold Networks (KANs) in function approximation and operator learning tasks.
  arXiv  Detail & Related papers  (2024-12-21T19:01:38Z)
• On the expressiveness and spectral bias of KANs [17.42614039265962]
  KANs were recently proposed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP)<n>KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demostrated in function regression, PDE solving, and many more scientific problems.
  arXiv  Detail & Related papers  (2024-10-02T17:57:38Z)
• Incorporating Arbitrary Matrix Group Equivariance into KANs [69.30866522377694]
  We propose Equivariant Kolmogorov-Arnold Networks (EKAN), a method for incorporating arbitrary matrix group equivariants into KANs.<n>EKAN achieves higher accuracy with smaller datasets or fewer parameters on symmetry-related tasks, such as particle scattering and the three-body problem.
  arXiv  Detail & Related papers  (2024-10-01T06:34:58Z)
• A preliminary study on continual learning in computer vision using Kolmogorov-Arnold Networks [43.70716358136333]
  Kolmogorov- Networks (KAN) are based on a fundamentally different mathematical framework. KANs address several major issues insio, such as forgetting in continual learning scenarios. We extend the investigation by evaluating the performance of KANs in continual learning tasks within computer vision.
  arXiv  Detail & Related papers  (2024-09-20T14:49:21Z)
• KAN v.s. MLP for Offline Reinforcement Learning [4.3621896506713185]
  Kolmogorov-Arnold Networks (KAN) is an emerging neural network architecture in machine learning. In this paper, we explore the incorporation of KAN into the actor and critic networks for offline reinforcement learning.
  arXiv  Detail & Related papers  (2024-09-15T07:52:44Z)
• Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction [53.88231294380083]
  We introduce a novel Multi-Epoch learning with Data Augmentation (MEDA) framework, suitable for both non-continual and continual learning scenarios. MEDA minimizes overfitting by reducing the dependency of the embedding layer on subsequent training data. Our findings confirm that pre-trained layers can adapt to new embedding spaces, enhancing performance without overfitting.
  arXiv  Detail & Related papers  (2024-06-27T04:00:15Z)
• Equivariant Architectures for Learning in Deep Weight Spaces [54.61765488960555]
  We present a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trainedvariant. We show how these layers can be implemented using three basic operations.
  arXiv  Detail & Related papers  (2023-01-30T10:50:33Z)
• Back to MLP: A Simple Baseline for Human Motion Prediction [59.18776744541904]
  This paper tackles the problem of human motion prediction, consisting in forecasting future body poses from historically observed sequences. We show that the performance of these approaches can be surpassed by a light-weight and purely architectural architecture with only 0.14M parameters. An exhaustive evaluation on Human3.6M, AMASS and 3DPW datasets shows that our method, which we dub siMLPe, consistently outperforms all other approaches.
  arXiv  Detail & Related papers  (2022-07-04T16:35:58Z)
• Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models [68.9288651177564]
  We present a novel MoE architecture based on matrix product operators (MPO) from quantum many-body physics. With the decomposed MPO structure, we can reduce the parameters of the original MoE architecture. Experiments on the three well-known downstream natural language datasets based on GPT2 show improved performance and efficiency in increasing model capacity.
  arXiv  Detail & Related papers  (2022-03-02T13:44:49Z)
• Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
  We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn. We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
  arXiv  Detail & Related papers  (2020-12-01T11:46:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.