PyHopper -- Hyperparameter optimization
- URL: http://arxiv.org/abs/2210.04728v1
- Date: Mon, 10 Oct 2022 14:35:01 GMT
- Title: PyHopper -- Hyperparameter optimization
- Authors: Mathias Lechner, Ramin Hasani, Philipp Neubauer, Sophie Neubauer,
Daniela Rus
- Abstract summary: We present PyHopper, a black-box optimization platform for machine learning researchers.
PyHopper's goal is to integrate with existing code with minimal effort and run the optimization process with minimal necessary manual oversight.
With simplicity as the primary theme, PyHopper is powered by a single robust Markov-chain Monte-Carlo optimization algorithm.
- Score: 51.40201315676902
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Hyperparameter tuning is a fundamental aspect of machine learning research.
Setting up the infrastructure for systematic optimization of hyperparameters
can take a significant amount of time. Here, we present PyHopper, a black-box
optimization platform designed to streamline the hyperparameter tuning workflow
of machine learning researchers. PyHopper's goal is to integrate with existing
code with minimal effort and run the optimization process with minimal
necessary manual oversight. With simplicity as the primary theme, PyHopper is
powered by a single robust Markov-chain Monte-Carlo optimization algorithm that
scales to millions of dimensions. Compared to existing tuning packages,
focusing on a single algorithm frees the user from having to decide between
several algorithms and makes PyHopper easily customizable. PyHopper is publicly
available under the Apache-2.0 license at https://github.com/PyHopper/PyHopper.
Related papers
- PockEngine: Sparse and Efficient Fine-tuning in a Pocket [62.955793932377524]
We introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices.
PockEngine supports sparse backpropagation and sparsely updates the model with measured memory saving and latency reduction.
Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin at 550 tokens/s, 7.9$times$ faster than the PyTorch.
arXiv Detail & Related papers (2023-10-26T19:46:11Z) - PyBADS: Fast and robust black-box optimization in Python [11.4219428942199]
PyBADS is an implementation of the Adaptive Direct Search (BADS) algorithm for fast and robust black-box optimization.
It comes along with an easy-to-use Python interface for running the algorithm for running the results.
arXiv Detail & Related papers (2023-06-27T15:54:44Z) - PyTorch Hyperparameter Tuning - A Tutorial for spotPython [0.20305676256390928]
This tutorial includes a brief comparison with Ray Tune, a Python library for running experiments and tuning hyperparameters.
We show that spotPython achieves similar or even better results while being more flexible and transparent than Ray Tune.
arXiv Detail & Related papers (2023-05-19T17:47:50Z) - VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
We leverage the same scaling approach behind the success of deep learning to learn versatiles.
We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates.
We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
arXiv Detail & Related papers (2022-11-17T18:39:07Z) - Pre-training helps Bayesian optimization too [49.28382118032923]
We seek an alternative practice for setting functional priors.
In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori.
Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
arXiv Detail & Related papers (2022-07-07T04:42:54Z) - PyEPO: A PyTorch-based End-to-End Predict-then-Optimize Library for
Linear and Integer Programming [9.764407462807588]
We present the PyEPO package, a PyTorchbased end-to-end predict-then-optimize library in Python.
PyEPO is the first such generic tool for linear and integer programming with predicted objective function coefficients.
arXiv Detail & Related papers (2022-06-28T18:33:55Z) - Surrogate modeling for Bayesian optimization beyond a single Gaussian
process [62.294228304646516]
We propose a novel Bayesian surrogate model to balance exploration with exploitation of the search space.
To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model.
To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret.
arXiv Detail & Related papers (2022-05-27T16:43:10Z) - Tasks, stability, architecture, and compute: Training more effective
learned optimizers, and using them to train themselves [53.37905268850274]
We introduce a new, hierarchical, neural network parameterized, hierarchical with access to additional features such as validation loss to enable automatic regularization.
Most learneds have been trained on only a single task, or a small number of tasks.
We train ours on thousands of tasks, making use of orders of magnitude more compute, resulting in generalizes that perform better to unseen tasks.
arXiv Detail & Related papers (2020-09-23T16:35:09Z) - Sherpa: Robust Hyperparameter Optimization for Machine Learning [6.156647008180291]
Sherpa is a hyperparameter optimization library for machine learning models.
It is specifically designed for problems with computationally expensive, iterative function evaluations.
Sherpa can be run on either a single machine or in parallel on a cluster.
arXiv Detail & Related papers (2020-05-08T13:52:49Z) - PHS: A Toolbox for Parallel Hyperparameter Search [2.0305676256390934]
We introduce an open source python framework named PHS - Parallel Hyperparameter Search.
It enables hyperparameter optimization on numerous compute instances of any arbitrary python function.
arXiv Detail & Related papers (2020-02-26T12:17:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.