What is important about the No Free Lunch theorems?

• URL: http://arxiv.org/abs/2007.10928v1
• Date: Tue, 21 Jul 2020 16:42:36 GMT
• Title: What is important about the No Free Lunch theorems?
• Authors: David H. Wolpert
• Abstract summary: The No Free Lunch theorems prove that under a uniform distribution over induction problems, all induction algorithms perform equally. The importance of the theorems arises by using them to analyze scenarios involving non-uniform' distributions. They also motivate a dictionary'' between supervised learning and improve blackbox optimization.
• Score: 0.5874142059884521
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The No Free Lunch theorems prove that under a uniform distribution over induction problems (search problems or learning problems), all induction algorithms perform equally. As I discuss in this chapter, the importance of the theorems arises by using them to analyze scenarios involving {non-uniform} distributions, and to compare different algorithms, without any assumption about the distribution over problems at all. In particular, the theorems prove that {anti}-cross-validation (choosing among a set of candidate algorithms based on which has {worst} out-of-sample behavior) performs as well as cross-validation, unless one makes an assumption -- which has never been formalized -- about how the distribution over induction problems, on the one hand, is related to the set of algorithms one is choosing among using (anti-)cross validation, on the other. In addition, they establish strong caveats concerning the significance of the many results in the literature which establish the strength of a particular algorithm without assuming a particular distribution. They also motivate a ``dictionary'' between supervised learning and improve blackbox optimization, which allows one to ``translate'' techniques from supervised learning into the domain of blackbox optimization, thereby strengthening blackbox optimization algorithms. In addition to these topics, I also briefly discuss their implications for philosophy of science.

Related papers

• A quantitative Robbins-Siegmund theorem [0.0]
  We provide a quantitative version of the Robbins-Siegmund theorem, establishing a bound on how far one needs to look in order to locate a region of metastability in the sense of Tao. Our proof involves a metastable analogue of Doob's theorem for $L_$-supermartingales along with a series of technical lemmas that make precise how quantitative information propagates through sums and products of processes.
  arXiv  Detail & Related papers  (2024-10-21T13:16:29Z)
• Inference for an Algorithmic Fairness-Accuracy Frontier [0.9147443443422864]
  We provide a consistent estimator for a theoretical fairness-accuracy frontier put forward by Liang, Lu and Mu (2023) We propose inference methods to test hypotheses that have received much attention in the fairness literature. We show that the estimated support function converges to a tight process as the sample size increases.
  arXiv  Detail & Related papers  (2024-02-14T00:56:09Z)
• The Adversary Bound Revisited: From Optimal Query Algorithms to Optimal Control [0.0]
  This note complements the paper "One-Way Ticket to Las Vegas and the Quantum Adversary" I develop the ideas behind the adversary bound - universal algorithm duality therein in a different form, using the same perspective as Barnum-Saks-Szegedy.
  arXiv  Detail & Related papers  (2022-11-29T15:25:45Z)
• On a class of geodesically convex optimization problems solved via Euclidean MM methods [50.428784381385164]
  We show how a difference of Euclidean convexization functions can be written as a difference of different types of problems in statistics and machine learning. Ultimately, we helps the broader broader the broader the broader the broader the work.
  arXiv  Detail & Related papers  (2022-06-22T23:57:40Z)
• Optimal Algorithms for Decentralized Stochastic Variational Inequalities [113.43047601775453]
  This work concentrates on the decentralized setting, which is increasingly important but not well understood. We present lower bounds for both communication and local iterations and construct optimal algorithms that match these lower bounds. Our algorithms are the best among the available not only in the decentralized case, but also in the deterministic and non-distributed literature.
  arXiv  Detail & Related papers  (2022-02-06T13:14:02Z)
• Minimax Optimization: The Case of Convex-Submodular [50.03984152441271]
  Minimax problems extend beyond the continuous domain to mixed continuous-discrete domains or even fully discrete domains. We introduce the class of convex-submodular minimax problems, where the objective is convex with respect to the continuous variable and submodular with respect to the discrete variable. Our proposed algorithms are iterative and combine tools from both discrete and continuous optimization.
  arXiv  Detail & Related papers  (2021-11-01T21:06:35Z)
• Black-Box Optimization via Generative Adversarial Nets [6.46243851154653]
  We present agenerative adversarial nets (OPT-GAN) to guide search on black-box problems. Experiments demonstrate that OPT-GAN outperforms other classical BBO algorithms.
  arXiv  Detail & Related papers  (2021-02-07T19:12:09Z)
• Recent Theoretical Advances in Non-Convex Optimization [56.88981258425256]
  Motivated by recent increased interest in analysis of optimization algorithms for non- optimization in deep networks and other problems in data, we give an overview of recent results of theoretical optimization algorithms for non- optimization.
  arXiv  Detail & Related papers  (2020-12-11T08:28:51Z)
• Global Optimization of Objective Functions Represented by ReLU Networks [77.55969359556032]
  Neural networks can learn complex, non- adversarial functions, and it is challenging to guarantee their correct behavior in safety-critical contexts. Many approaches exist to find failures in networks (e.g., adversarial examples), but these cannot guarantee the absence of failures. We propose an approach that integrates the optimization process into the verification procedure, achieving better performance than the naive approach.
  arXiv  Detail & Related papers  (2020-10-07T08:19:48Z)
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms [67.67377846416106]
  We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We show that value-based methods such as TD($lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions.
  arXiv  Detail & Related papers  (2020-03-27T05:13:29Z)
• Scalable Approximate Inference and Some Applications [2.6541211006790983]
  In this thesis, we propose a new framework for approximate inference. Our proposed four algorithms are motivated by the recent computational progress of Stein's method. Results on simulated and real datasets indicate the statistical efficiency and wide applicability of our algorithm.
  arXiv  Detail & Related papers  (2020-03-07T04:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.