MLPro: A System for Hosting Crowdsourced Machine Learning Challenges for Open-Ended Research Problems

• URL: http://arxiv.org/abs/2204.01216v1
• Date: Mon, 4 Apr 2022 02:56:12 GMT
• Title: MLPro: A System for Hosting Crowdsourced Machine Learning Challenges for Open-Ended Research Problems
• Authors: Peter Washington, Aayush Nandkeolyar, Sam Yang
• Abstract summary: We develop a system which combines the notion of open-ended ML coding problems with the concept of an automatic online code judging platform. We find that for sufficiently unconstrained and complex problems, many experts submit similar solutions, but some experts provide unique solutions which outperform the "typical" solution class.
• Score: 1.3254304182988286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The task of developing a machine learning (ML) model for a particular problem is inherently open-ended, and there is an unbounded set of possible solutions. Steps of the ML development pipeline, such as feature engineering, loss function specification, data imputation, and dimensionality reduction, require the engineer to consider an extensive and often infinite array of possibilities. Successfully identifying high-performing solutions for an unfamiliar dataset or problem requires a mix of mathematical prowess and creativity applied towards inventing and repurposing novel ML methods. Here, we explore the feasibility of hosting crowdsourced ML challenges to facilitate a breadth-first exploration of open-ended research problems, thereby expanding the search space of problem solutions beyond what a typical ML team could viably investigate. We develop MLPro, a system which combines the notion of open-ended ML coding problems with the concept of an automatic online code judging platform. To conduct a pilot evaluation of this paradigm, we crowdsource several open-ended ML challenges to ML and data science practitioners. We describe results from two separate challenges. We find that for sufficiently unconstrained and complex problems, many experts submit similar solutions, but some experts provide unique solutions which outperform the "typical" solution class. We suggest that automated expert crowdsourcing systems such as MLPro have the potential to accelerate ML engineering creativity.

Related papers

• ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection [60.297079601066784]
  We introduce ErrorRadar, the first benchmark designed to assess MLLMs' capabilities in error detection. ErrorRadar evaluates two sub-tasks: error step identification and error categorization. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation.
  arXiv  Detail & Related papers  (2024-10-06T14:59:09Z)
• Matching Problems to Solutions: An Explainable Way of Solving Machine Learning Problems [1.7368964547487398]
  Domain experts from all fields are called upon, working with data scientists, to explore the use of ML techniques to solve their problems. This paper focuses on: 1) the representation of domain problems, ML problems, and the main ML solution artefacts, and 2) a matching function that helps identify the ML algorithm family that is most appropriate for the domain problem at hand.
  arXiv  Detail & Related papers  (2024-06-21T21:39:34Z)
• MacGyver: Are Large Language Models Creative Problem Solvers? [87.70522322728581]
  We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. We create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems. We present our collection to both LLMs and humans to compare and contrast their problem-solving abilities.
  arXiv  Detail & Related papers  (2023-11-16T08:52:27Z)
• MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks [31.733088105662876]
  We aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks.
  arXiv  Detail & Related papers  (2023-04-28T17:03:57Z)
• Tiny Robot Learning: Challenges and Directions for Machine Learning in Resource-Constrained Robots [57.27442333662654]
  Machine learning (ML) has become a pervasive tool across computing systems. Tiny robot learning is the deployment of ML on resource-constrained low-cost autonomous robots. Tiny robot learning is subject to challenges from size, weight, area, and power (SWAP) constraints. This paper gives a brief survey of the tiny robot learning space, elaborates on key challenges, and proposes promising opportunities for future work in ML system design.
  arXiv  Detail & Related papers  (2022-05-11T19:36:15Z)
• Panoramic Learning with A Standardized Machine Learning Formalism [116.34627789412102]
  This paper presents a standardized equation of the learning objective, that offers a unifying understanding of diverse ML algorithms. It also provides guidance for mechanic design of new ML solutions, and serves as a promising vehicle towards panoramic learning with all experiences.
  arXiv  Detail & Related papers  (2021-08-17T17:44:38Z)
• White Paper Machine Learning in Certified Systems [70.24215483154184]
  DEEL Project set-up the ML Certification 3 Workgroup (WG) set-up by the Institut de Recherche Technologique Saint Exup'ery de Toulouse (IRT)
  arXiv  Detail & Related papers  (2021-03-18T21:14:30Z)
• Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making [67.72855777115772]
  Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts -- who often have no expertise in ML or data science -- are asked to use ML predictions to make high-stakes decisions. We investigate the ML usability challenges present in the domain of child welfare screening through a series of collaborations with child welfare screeners.
  arXiv  Detail & Related papers  (2021-03-02T22:50:45Z)
• A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges [0.0]
  Advancements in machine learning (ML) lead to a shift from the traditional view of software development, where algorithms are hard-coded by humans, to ML systems materialized through learning from data. We need to revisit our ways of developing software systems and consider the particularities required by these new types of systems.
  arXiv  Detail & Related papers  (2020-12-14T20:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.