Betty: An Automatic Differentiation Library for Multilevel Optimization
- URL: http://arxiv.org/abs/2207.02849v1
- Date: Tue, 5 Jul 2022 14:01:15 GMT
- Title: Betty: An Automatic Differentiation Library for Multilevel Optimization
- Authors: Sang Keun Choe, Willie Neiswanger, Pengtao Xie, Eric Xing
- Abstract summary: Betty is a high-level software library for gradient-based multilevel optimization.
We develop an automatic differentiation procedure based on a novel interpretation of multilevel optimization as a dataflow graph.
We empirically demonstrate that Betty can be used as a high-level programming interface for an array of multilevel optimization programs.
- Score: 24.256787655657277
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multilevel optimization has been widely adopted as a mathematical foundation
for a myriad of machine learning problems, such as hyperparameter optimization,
meta-learning, and reinforcement learning, to name a few. Nonetheless,
implementing multilevel optimization programs oftentimes requires expertise in
both mathematics and programming, stunting research in this field. We take an
initial step towards closing this gap by introducing Betty, a high-level
software library for gradient-based multilevel optimization. To this end, we
develop an automatic differentiation procedure based on a novel interpretation
of multilevel optimization as a dataflow graph. We further abstract the main
components of multilevel optimization as Python classes, to enable easy,
modular, and maintainable programming. We empirically demonstrate that Betty
can be used as a high-level programming interface for an array of multilevel
optimization programs, while also observing up to 11\% increase in test
accuracy, 14\% decrease in GPU memory usage, and 20\% decrease in wall time
over existing implementations on multiple benchmarks. The code is available at
http://github.com/leopard-ai/betty .
Related papers
- Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization [71.35604981129838]
Traditional gradient-based bi-level optimization algorithms are ill-suited to meet the demands of large-scale applications.
We introduce $(textFG)2textU$, which achieves an unbiased approximation of the meta gradient for bi-level optimization.
$(textFG)2textU$ is inherently designed to support parallel computing, enabling it to effectively leverage large-scale distributed computing systems.
arXiv Detail & Related papers (2024-06-20T08:21:52Z) - Two Optimizers Are Better Than One: LLM Catalyst Empowers Gradient-Based Optimization for Prompt Tuning [69.95292905263393]
We show that gradient-based optimization and large language models (MsLL) are complementary to each other, suggesting a collaborative optimization approach.
Our code is released at https://www.guozix.com/guozix/LLM-catalyst.
arXiv Detail & Related papers (2024-05-30T06:24:14Z) - LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP [20.86307407685542]
Linear Probe (LP) has been often reported as a weak baseline for few-shot CLIP adaptation.
In this work, we examine from convex-optimization perspectives a generalization of the standard LP baseline.
Our image-language objective function, along with these non-trivial optimization insights and ingredients, yields, surprisingly, highly competitive few-shot CLIP performances.
arXiv Detail & Related papers (2024-04-02T20:23:10Z) - Contextual Stochastic Bilevel Optimization [50.36775806399861]
We introduce contextual bilevel optimization (CSBO) -- a bilevel optimization framework with the lower-level problem minimizing an expectation on some contextual information and the upper-level variable.
It is important for applications such as meta-learning, personalized learning, end-to-end learning, and Wasserstein distributionally robustly optimization with side information (WDRO-SI)
arXiv Detail & Related papers (2023-10-27T23:24:37Z) - Implicit Bilevel Optimization: Differentiating through Bilevel
Optimization Programming [7.310043452300735]
Bilevel Optimization Programming is used to model complex and conflicting interactions between agents.
BiGrad is applicable to both continuous and Bilevel optimization problems.
Experiments show that the BiGrad successfully extends existing single-level approaches to Bilevel Programming.
arXiv Detail & Related papers (2023-02-28T10:32:17Z) - Learning Performance-Improving Code Edits [107.21538852090208]
We introduce a framework for adapting large language models (LLMs) to high-level program optimization.
First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs.
For prompting, we propose retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play.
arXiv Detail & Related papers (2023-02-15T18:59:21Z) - Kernel methods through the roof: handling billions of points efficiently [94.31450736250918]
Kernel methods provide an elegant and principled approach to nonparametric learning, but so far could hardly be used in large scale problems.
Recent advances have shown the benefits of a number of algorithmic ideas, for example combining optimization, numerical linear algebra and random projections.
Here, we push these efforts further to develop and test a solver that takes full advantage of GPU hardware.
arXiv Detail & Related papers (2020-06-18T08:16:25Z) - Soft Gradient Boosting Machine [72.54062017726154]
We propose the soft Gradient Boosting Machine (sGBM) by wiring multiple differentiable base learners together.
Experimental results showed that, sGBM enjoys much higher time efficiency with better accuracy, given the same base learner in both on-line and off-line settings.
arXiv Detail & Related papers (2020-06-07T06:43:23Z) - pymoo: Multi-objective Optimization in Python [7.8140593450932965]
We have developed pymoo, a multi-objective optimization framework in Python.
We provide a guide to getting started with our framework by demonstrating the implementation of an exemplary constrained multi-objective optimization scenario.
The implementations in our framework are customizable and algorithms can be modified/extended by supplying custom operators.
arXiv Detail & Related papers (2020-01-22T16:04:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.