dagger: A Python Framework for Reproducible Machine Learning Experiment
Orchestration
- URL: http://arxiv.org/abs/2006.07484v1
- Date: Fri, 12 Jun 2020 21:42:48 GMT
- Title: dagger: A Python Framework for Reproducible Machine Learning Experiment
Orchestration
- Authors: Michela Paganini, Jessica Zosa Forde
- Abstract summary: Multi-stage experiments in machine learning often involve state-mutating operations acting on models along multiple paths of execution.
We present dagger, a framework to facilitate reproducible and reusable experiment orchestration.
- Score: 0.913755431537592
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Many research directions in machine learning, particularly in deep learning,
involve complex, multi-stage experiments, commonly involving state-mutating
operations acting on models along multiple paths of execution. Although machine
learning frameworks provide clean interfaces for defining model architectures
and unbranched flows, burden is often placed on the researcher to track
experimental provenance, that is, the state tree that leads to a final model
configuration and result in a multi-stage experiment. Originally motivated by
analysis reproducibility in the context of neural network pruning research,
where multi-stage experiment pipelines are common, we present dagger, a
framework to facilitate reproducible and reusable experiment orchestration. We
describe the design principles of the framework and example usage.
Related papers
- Automatic benchmarking of large multimodal models via iterative experiment programming [71.78089106671581]
We present APEx, the first framework for automatic benchmarking of LMMs.
Given a research question expressed in natural language, APEx leverages a large language model (LLM) and a library of pre-specified tools to generate a set of experiments for the model at hand.
The report drives the testing procedure: based on the current status of the investigation, APEx chooses which experiments to perform and whether the results are sufficient to draw conclusions.
arXiv Detail & Related papers (2024-06-18T06:43:46Z) - MLXP: A Framework for Conducting Replicable Experiments in Python [63.37350735954699]
We propose MLXP, an open-source, simple, and lightweight experiment management tool based on Python.
It streamlines the experimental process with minimal overhead while ensuring a high level of practitioner overhead.
arXiv Detail & Related papers (2024-02-21T14:22:20Z) - Learning Large-scale Universal User Representation with Sparse Mixture
of Experts [1.2722697496405464]
We propose SUPERMOE, a generic framework to obtain high quality user representation from multiple tasks.
Specifically, the user behaviour sequences are encoded by MoE transformer, and we can thus increase the model capacity to billions of parameters.
In order to deal with seesaw phenomenon when learning across multiple tasks, we design a new loss function with task indicators.
arXiv Detail & Related papers (2022-07-11T06:19:03Z) - Benchopt: Reproducible, efficient and collaborative optimization
benchmarks [67.29240500171532]
Benchopt is a framework to automate, reproduce and publish optimization benchmarks in machine learning.
Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments.
arXiv Detail & Related papers (2022-06-27T16:19:24Z) - Active learning in open experimental environments: selecting the right
information channel(s) based on predictability in deep kernel learning [0.0]
Key tasks in experimental studies is establishing which of these channels is predictive of the behaviors of interest.
Here we explore the problem of discovery of the optimal predictive channel for structure-property relationships in microscopy.
This approach can be directly applicable to similar active learning tasks in automated synthesis and the discovery of quantitative structure-activity relations in molecular systems.
arXiv Detail & Related papers (2022-03-18T22:36:40Z) - An Information-Theoretic Framework for Supervised Learning [22.280001450122175]
We propose a novel information-theoretic framework with its own notions of regret and sample complexity.
We study the sample complexity of learning from data generated by deep neural networks with ReLU activation units.
We conclude by corroborating our theoretical results with experimental analysis of random single-hidden-layer neural networks.
arXiv Detail & Related papers (2022-03-01T05:58:28Z) - Experiments as Code: A Concept for Reproducible, Auditable, Debuggable,
Reusable, & Scalable Experiments [7.557948558412152]
A common concern in experimental research is the auditability and of experiments.
We propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code is provided.
arXiv Detail & Related papers (2022-02-24T12:15:00Z) - Multi-Agent Imitation Learning with Copulas [102.27052968901894]
Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions.
In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems.
Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
arXiv Detail & Related papers (2021-07-10T03:49:41Z) - Conditional Generative Modeling via Learning the Latent Space [54.620761775441046]
We propose a novel framework for conditional generation in multimodal spaces.
It uses latent variables to model generalizable learning patterns.
At inference, the latent variables are optimized to find optimal solutions corresponding to multiple output modes.
arXiv Detail & Related papers (2020-10-07T03:11:34Z) - Deep Multimodal Neural Architecture Search [178.35131768344246]
We devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks.
Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone.
On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks.
arXiv Detail & Related papers (2020-04-25T07:00:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.