Misleading Microbenchmarks on the Java Virtual Machines

• URL: http://arxiv.org/abs/2605.23570v1
• Date: Fri, 22 May 2026 12:38:12 GMT
• Title: Misleading Microbenchmarks on the Java Virtual Machines
• Abstract summary: On the Java Virtual Machine (JVM), developers often use microbenchmarks to choose the most performant implementation of a method or a class.<n>This paper shows how using microbenchmarks under conditions that induce unrealistic profiles yields misleading results despite following existing guidelines.
• Score: 1.4885603159356078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developers often use microbenchmarks to choose the most performant implementation of a method or a class. On the Java Virtual Machine (JVM), this is commonly done using the Java Microbenchmark Harness (JMH) which addresses common pitfalls of measuring code performance on the JVM. However, even using JMH guidelines cannot overcome the fundamental issue of context. Microbenchmarks inherently execute code in isolation, without interference from other application code competing for CPU resources, such as cache or branch-predictor capacity. On managed runtimes with tiered dynamic compilation, such as the JVM, the speculative, profile-driven nature of compilation decisions means that code performance is highly dependent on profiles collected during early execution. Because profiles usually include also branch probabilities and receiver types (besides code hotness metrics), a badly designed microbenchmark may cause the JVM to collect an unrealistic profile, resulting in aggressive, yet misleading, optimizations, that would not occur in a real application. In this paper, we demonstrate how using microbenchmarks under conditions that induce the JVM to collect unrealistic profiles yields misleading results despite following existing guidelines. We also extend these guidelines by suggesting actions to make the microbenchmark results more representative.

Related papers

• JEDI: Java Evaluation of Declarative and Imperative Queries [1.6042394978941517]
  There is a lack of benchmarks specifically targeting stream-based applications.<n>In this work we present JEDI, a benchmark suite that targets the Stream API.
  arXiv  Detail & Related papers  (2026-05-22T12:07:55Z)
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory [50.857546269660276]
  We introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving large language models (LLMs) memory.<n>Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference.
  arXiv  Detail & Related papers  (2026-05-14T20:15:22Z)
• MapReplay: Trace-Driven Benchmark Generation for Java HashMap [1.198637189064676]
  We propose MapReplay, a benchmarking methodology that combines the realism of application benchmarks with the efficiency of microbenchmarks.<n>Applying MapReplay to DaCapo-Chopin and Renaissance, the resulting suite, MapReplayBench, reproduces application-level performance trends.
  arXiv  Detail & Related papers  (2026-03-14T16:46:09Z)
• EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation [9.472124187479915]
  In on-chain transaction scenarios, even a minor error can cause irreversible loss for users.<n>We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation.<n>We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion.
  arXiv  Detail & Related papers  (2026-01-10T13:25:27Z)
• Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts [49.99400612296149]
  We find that models can ace many benchmarks without strong visual understanding.<n>This is especially problematic for vision-centric benchmarks that are meant to require visual inputs.<n>We adopt a diagnostic principle for benchmark design: if a benchmark can be gamed, it will be gamed.
  arXiv  Detail & Related papers  (2025-11-06T18:43:21Z)
• MIB: A Mechanistic Interpretability Benchmark [77.35046700898326]
  We propose MIB, a Mechanistic Interpretability Benchmark, with two tracks spanning four tasks and five models.<n>Using MIB, we find that attribution and mask optimization methods perform best on circuit localization.<n>For causal variable localization, we find that the supervised DAS method performs best, while SAE features are not better than neurons.
  arXiv  Detail & Related papers  (2025-04-17T17:55:45Z)
• ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptions [4.852619858744873]
  Large Language Models (LLMs) have shown astounding capabilities of code understanding and synthesis.<n>We introduce ThrowBench, a benchmark consisting of over 2,400 short user-written programs written in four different programming languages.<n>We evaluate our benchmark on six state-of-the-art code LLMs and see modest performance ranging from 19 to 38% (F1 score)
  arXiv  Detail & Related papers  (2025-03-06T09:22:23Z)
• DOCE: Finding the Sweet Spot for Execution-Based Code Generation [69.5305729627198]
  We propose a comprehensive framework that includes candidate generation, $n$-best reranking, minimum Bayes risk (MBR) decoding, and self-ging as the core components. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods.
  arXiv  Detail & Related papers  (2024-08-25T07:10:36Z)
• How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark [60.72725673114168]
  We revisit the question of accurate BERT-pruning during fine-tuning on downstream datasets. We propose a set of general guidelines for successful pruning, even on the challenging SMC benchmark.
  arXiv  Detail & Related papers  (2023-12-21T03:11:30Z)
• Defining Standard Strategies for Quantum Benchmarks [0.1759008116536278]
  We define a set of characteristics that any benchmark should follow, and make a distinction between benchmarks and diagnostics. We discuss the issue of benchmark optimizations, detail when those optimizations are appropriate, and how they should be reported. We introduce a scalable mirror quantum volume benchmark.
  arXiv  Detail & Related papers  (2023-03-03T17:50:34Z)
• The Benchmark Lottery [114.43978017484893]
  "A benchmark lottery" describes the overall fragility of the machine learning benchmarking process. We show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks.
  arXiv  Detail & Related papers  (2021-07-14T21:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.