Bridging Algorithmic Information Theory and Machine Learning: A New Approach to Kernel Learning

• URL: http://arxiv.org/abs/2311.12624v3
• Date: Wed, 10 Apr 2024 11:35:14 GMT
• Title: Bridging Algorithmic Information Theory and Machine Learning: A New Approach to Kernel Learning
• Authors: Boumediene Hamzi, Marcus Hutter, Houman Owhadi,
• Abstract summary: We adopt an AIT perspective on the problem of learning kernels from data, through the method of Sparse Kernel Flows. This approach opens the door to reformulating algorithms in machine learning using tools from AIT.
• Score: 12.848057726330714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine Learning (ML) and Algorithmic Information Theory (AIT) look at Complexity from different points of view. We explore the interface between AIT and Kernel Methods (that are prevalent in ML) by adopting an AIT perspective on the problem of learning kernels from data, in kernel ridge regression, through the method of Sparse Kernel Flows. In particular, by looking at the differences and commonalities between Minimal Description Length (MDL) and Regularization in Machine Learning (RML), we prove that the method of Sparse Kernel Flows is the natural approach to adopt to learn kernels from data. This approach aligns naturally with the MDL principle, offering a more robust theoretical basis than the existing reliance on cross-validation. The study reveals that deriving Sparse Kernel Flows does not require a statistical approach; instead, one can directly engage with code-lengths and complexities, concepts central to AIT. Thereby, this approach opens the door to reformulating algorithms in machine learning using tools from AIT, with the aim of providing them a more solid theoretical foundation.

Related papers

• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Blind Super-Resolution via Meta-learning and Markov Chain Monte Carlo Simulation [46.5310645609264]
  We propose a Meta-learning and Markov Chain Monte Carlo based SISR approach to learn kernel priors from organized randomness. A lightweight network is adopted as kernel generator, and is optimized via learning from the MCMC simulation on random Gaussian distributions. A meta-learning-based alternating optimization procedure is proposed to optimize the kernel generator and image restorer.
  arXiv  Detail & Related papers  (2024-06-13T07:50:15Z)
• Kernel Correlation-Dissimilarity for Multiple Kernel k-Means Clustering [21.685153346752124]
  Current methods enhance information diversity and reduce redundancy by exploiting interdependencies among multiple kernels based on correlations or dissimilarities. We introduce a novel method that systematically integrates both kernel correlation and dissimilarity. By emphasizing the coherence between kernel correlation and dissimilarity, our method offers a more objective and transparent strategy for extracting non-linear information.
  arXiv  Detail & Related papers  (2024-03-06T04:24:43Z)
• Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics [77.34726150561087]
  Recent developments in artificial neural networks, particularly deep learning (DL), are reviewed in detail. Both hybrid and pure machine learning (ML) methods are discussed. History and limitations of AI are recounted and discussed, with particular attention at pointing out misstatements or misconceptions of the classics.
  arXiv  Detail & Related papers  (2022-12-18T02:03:00Z)
• On Leave-One-Out Conditional Mutual Information For Generalization [122.2734338600665]
  We derive information theoretic generalization bounds for supervised learning algorithms based on a new measure of leave-one-out conditional mutual information (loo-CMI) Contrary to other CMI bounds, our loo-CMI bounds can be computed easily and can be interpreted in connection to other notions such as classical leave-one-out cross-validation. We empirically validate the quality of the bound by evaluating its predicted generalization gap in scenarios for deep learning.
  arXiv  Detail & Related papers  (2022-07-01T17:58:29Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Neural Generalization of Multiple Kernel Learning [2.064612766965483]
  Multiple Kernel Learning is a conventional way to learn the kernel function in kernel-based methods. Deep learning models can learn complex functions by applying nonlinear transformations to data through several layers. We show that a typical MKL algorithm can be interpreted as a one-layer neural network with linear activation functions.
  arXiv  Detail & Related papers  (2021-02-26T07:28:37Z)
• Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms [91.3755431537592]
  We analyze four broad meta-learning strategies which rely on plug-in estimation and pseudo-outcome regression. We highlight how this theoretical reasoning can be used to guide principled algorithm design and translate our analyses into practice.
  arXiv  Detail & Related papers  (2021-01-26T17:11:40Z)
• Self-organizing Democratized Learning: Towards Large-scale Distributed Learning Systems [71.14339738190202]
  democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this paper. The proposed algorithms demonstrate better results in the generalization performance of learning models in agents compared to the conventional FL algorithms.
  arXiv  Detail & Related papers  (2020-07-07T08:34:48Z)
• Gradient tracking and variance reduction for decentralized optimization and machine learning [19.54092620537586]
  Decentralized methods to solve finite-sum problems are important in many signal processing and machine learning tasks. We provide a unified algorithmic framework that combines variance-reduction with gradient tracking to achieve robust performance.
  arXiv  Detail & Related papers  (2020-02-13T07:17:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.