Adaptive Inference: Theoretical Limits and Unexplored Opportunities

• URL: http://arxiv.org/abs/2402.04359v1
• Date: Tue, 6 Feb 2024 19:49:23 GMT
• Title: Adaptive Inference: Theoretical Limits and Unexplored Opportunities
• Authors: Soheil Hor, Ying Qian, Mert Pilanci, Amin Arbabian
• Abstract summary: We provide new approximate and exact bounds for the achievable efficiency and performance gains. We offer insights on improving achievable efficiency gains through the optimal selection and design of adaptive inference state spaces.
• Score: 43.19515571173486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces the first theoretical framework for quantifying the efficiency and performance gain opportunity size of adaptive inference algorithms. We provide new approximate and exact bounds for the achievable efficiency and performance gains, supported by empirical evidence demonstrating the potential for 10-100x efficiency improvements in both Computer Vision and Natural Language Processing tasks without incurring any performance penalties. Additionally, we offer insights on improving achievable efficiency gains through the optimal selection and design of adaptive inference state spaces.

Related papers

• A Novel Unified Parametric Assumption for Nonconvex Optimization [53.943470475510196]
  Non optimization is central to machine learning, but the general framework non convexity enables weak convergence guarantees too pessimistic compared to the other hand. We introduce a novel unified assumption in non convex algorithms.
  arXiv  Detail & Related papers  (2025-02-17T21:25:31Z)
• Black-box Optimization with Simultaneous Statistical Inference for Optimal Performance [18.13513199455587]
  Black-box optimization is often encountered for decision-making in complex systems management. Our goal is to address the dual tasks of optimization and statistical inference for the optimal performance in an online fashion.
  arXiv  Detail & Related papers  (2025-01-14T02:37:09Z)
• Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training [3.668740611399284]
  Large-scale language models (LLM) have achieved remarkable results in a variety of tasks. This paper proposes an improved method based on adaptive optimization algorithm.
  arXiv  Detail & Related papers  (2024-12-06T02:17:30Z)
• Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks [6.596361762662328]
  Internal structure and operation mechanism of large-scale language models are analyzed theoretically. We evaluate the contribution of adaptive optimization algorithms (such as AdamW), massively parallel computing techniques, and mixed precision training strategies.
  arXiv  Detail & Related papers  (2024-05-20T00:10:00Z)
• Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties [49.351577714596544]
  We propose a human-AI collaborative Bayesian framework to incorporate expert preferences about unmeasured abstract properties into surrogate modeling. We provide an efficient strategy that can also handle any incorrect/misleading expert bias in preferential judgments.
  arXiv  Detail & Related papers  (2024-02-27T09:23:13Z)
• Towards Compute-Optimal Transfer Learning [82.88829463290041]
  We argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.
  arXiv  Detail & Related papers  (2023-04-25T21:49:09Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions [80.78077900288868]
  We decompose the design methodology of early-exit networks to its key components and survey the recent advances in each one of them. We position early-exiting against other efficient inference solutions and provide our insights on the current challenges and most promising future directions for research in the field.
  arXiv  Detail & Related papers  (2021-06-09T12:33:02Z)
• Effective End-to-End Learning Framework for Economic Dispatch [3.034038412630808]
  We adopt the notion of end-to-end machine learning and propose a task-specific learning criteria to conduct economic dispatch. We provide both theoretical analysis and empirical insights to highlight the effectiveness and efficiency of the proposed learning framework.
  arXiv  Detail & Related papers  (2020-02-22T08:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.