MLCommons Cloud Masking Benchmark with Early Stopping

• URL: http://arxiv.org/abs/2401.08636v1
• Date: Mon, 11 Dec 2023 19:06:06 GMT
• Title: MLCommons Cloud Masking Benchmark with Early Stopping
• Abstract summary: We report on work performed for the MLCommons Science Working Group on the cloud masking benchmark. The benchmarks are conducted on the High Performance Computing Clusters of New York University and University of Virginia. We provide a description of the cloud masking benchmark, as well as a summary of our submission to MLCommons on the benchmark experiment we conducted.
• Score: 0.837165016574332
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we report on work performed for the MLCommons Science Working Group on the cloud masking benchmark. MLCommons is a consortium that develops and maintains several scientific benchmarks that aim to benefit developments in AI. The benchmarks are conducted on the High Performance Computing (HPC) Clusters of New York University and University of Virginia, as well as a commodity desktop. We provide a description of the cloud masking benchmark, as well as a summary of our submission to MLCommons on the benchmark experiment we conducted. It includes a modification to the reference implementation of the cloud masking benchmark enabling early stopping. This benchmark is executed on the NYU HPC through a custom batch script that runs the various experiments through the batch queuing system while allowing for variation on the number of epochs trained. Our submission includes the modified code, a custom batch script to modify epochs, documentation, and the benchmark results. We report the highest accuracy (scientific metric) and the average time taken (performance metric) for training and inference that was achieved on NYU HPC Greene. We also provide a comparison of the compute capabilities between different systems by running the benchmark for one epoch. Our submission can be found in a Globus repository that is accessible to MLCommons Science Working Group.

Related papers

• Assessing and Improving the Representativeness of Code Generation Benchmarks Using Knowledge Units (KUs) of Programming Languages -- An Empirical Study [7.0773305889955616]
  Large Language Models (LLMs) have shown impressive performance in code generation.<n>LLMs must understand and apply a wide range of language concepts.<n>If the concepts exercised in benchmarks are not representative of those used in real-world projects, evaluations may yield incomplete.
  arXiv  Detail & Related papers  (2026-01-07T10:23:33Z)
• InfoSynth: Information-Guided Benchmark Synthesis for LLMs [69.80981631587501]
  Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation.<n>Traditional benchmark creation relies on manual human effort, a process that is both expensive and time-consuming.<n>This work introduces Info Synth, a novel framework for automatically generating and evaluating reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-01-02T05:26:27Z)
• How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation under the One-Time-Pad-Based Framework [8.76693832650115]
  Overestimation in evaluating large language models (LLMs) has become an increasing concern.<n>We propose ArxivRoll, a dynamic evaluation framework inspired by one-time pad encryption in cryptography.
  arXiv  Detail & Related papers  (2025-07-25T12:39:03Z)
• Improve Machine Learning carbon footprint using Nvidia GPU and Mixed Precision training for classification models -- Part I [0.0]
  This dissertation compares the power consumption using the default floating point (32bit) and Nvidia mixed precision (16bit and 32bit) while training a classification ML model.
  arXiv  Detail & Related papers  (2024-09-12T08:59:53Z)
• Benchmarking Predictive Coding Networks -- Made Simple [48.652114040426625]
  We tackle the problems of efficiency and scalability for predictive coding networks (PCNs) in machine learning. We propose a library, called PCX, that focuses on performance and simplicity, and use it to implement a large set of standard benchmarks. We perform extensive tests on such benchmarks using both existing algorithms for PCNs, as well as adaptations of other methods popular in the bio-plausible deep learning community.
  arXiv  Detail & Related papers  (2024-07-01T10:33:44Z)
• LiveBench: A Challenging, Contamination-Free LLM Benchmark [101.21578097087699]
  We release LiveBench, the first benchmark that contains frequently-updated questions from recent information sources. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 110B in size. Questions will be added and updated on a monthly basis, and we will release new tasks and harder versions of tasks over time.
  arXiv  Detail & Related papers  (2024-06-27T16:47:42Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification (UQ) is a critical component of machine learning (ML) applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across nine tasks, and identify the most promising approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Improvements & Evaluations on the MLCommons CloudMask Benchmark [11.115005057843579]
  We provide a description of the cloud-masking benchmark task, updated code, and the best model for this benchmark. Results include the highest accuracy achieved on the NYU system as well as the average time taken for both training and inference.
  arXiv  Detail & Related papers  (2024-03-07T14:48:48Z)
• How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark [60.72725673114168]
  We revisit the question of accurate BERT-pruning during fine-tuning on downstream datasets. We propose a set of general guidelines for successful pruning, even on the challenging SMC benchmark.
  arXiv  Detail & Related papers  (2023-12-21T03:11:30Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations [5.138982355658199]
  Molecular dynamics (MD) simulation techniques are widely used for various natural science applications. We benchmark a collection of state-of-the-art (SOTA) ML FF models and illustrate, in particular, how the commonly benchmarked force accuracy is not well aligned with relevant simulation metrics.
  arXiv  Detail & Related papers  (2022-10-13T17:59:03Z)
• PDEBENCH: An Extensive Benchmark for Scientific Machine Learning [20.036987098901644]
  We introduce PDEBench, a benchmark suite of time-dependent simulation tasks based on Partial Differential Equations (PDEs) PDEBench comprises both code and data to benchmark the performance of novel machine learning models against both classical numerical simulations and machine learning baselines.
  arXiv  Detail & Related papers  (2022-10-13T17:03:36Z)
• WRENCH: A Comprehensive Benchmark for Weak Supervision [66.82046201714766]
  benchmark consists of 22 varied real-world datasets for classification and sequence tagging. We use benchmark to conduct extensive comparisons over more than 100 method variants to demonstrate its efficacy as a benchmark platform.
  arXiv  Detail & Related papers  (2021-09-23T13:47:16Z)
• The Benchmark Lottery [114.43978017484893]
  "A benchmark lottery" describes the overall fragility of the machine learning benchmarking process. We show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks.
  arXiv  Detail & Related papers  (2021-07-14T21:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.