JEDI: Java Evaluation of Declarative and Imperative Queries

• URL: http://arxiv.org/abs/2605.23543v1
• Date: Fri, 22 May 2026 12:07:55 GMT
• Title: JEDI: Java Evaluation of Declarative and Imperative Queries
• Abstract summary: There is a lack of benchmarks specifically targeting stream-based applications.<n>In this work we present JEDI, a benchmark suite that targets the Stream API.
• Score: 1.6042394978941517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Java Stream API aims at increasing developer productivity thanks to an easy-to-read declarative syntax to express computations. It also simplifies parallel computing, providing a high-level abstraction on top of common parallelization aspects. Unfortunately, there is a lack of benchmarks specifically targeting stream-based applications. Such a lack of benchmarks makes it difficult for researchers and developers of the Java class library to optimize the Stream API. Moreover, in the absence of dedicated benchmarks, it is difficult to analyze the performance of streams to suggest developers how to write efficient code using the API. In this work we present JEDI, a benchmark suite that targets the Stream API. JEDI is automatically generated by converting SQL benchmarks into Java benchmarks. Our code generator supports targets different implementations (both stream-based and imperative) for the same query. The ultimate goal of our benchmark suite -- and the main contribution of this work -- is to analyze the performance of the different implementations to spot inefficient code structures and better alternatives, suggesting best practices to Java developers. Among the multiple implementations we generate, we focus on different parallelization strategies and explain the most efficient parallelization strategies based on characteristics of the processed data. Finally, the code generation producing imperative code defines of a baseline that can guide researchers and Java implementers to optimize the Stream API.

Related papers

• Misleading Microbenchmarks on the Java Virtual Machines [1.4885603159356078]
  On the Java Virtual Machine (JVM), developers often use microbenchmarks to choose the most performant implementation of a method or a class.<n>This paper shows how using microbenchmarks under conditions that induce unrealistic profiles yields misleading results despite following existing guidelines.
  arXiv  Detail & Related papers  (2026-05-22T12:38:12Z)
• PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization [43.72919370258115]
  We introduce PerfCodeBench, an executable benchmark for evaluating large language models (LLMs) on high-performance code optimization.<n>The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks.<n>Our evaluation shows a clear gap between model-generated code and expert-optimized implementations.
  arXiv  Detail & Related papers  (2026-05-13T08:10:26Z)
• AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion [55.21541958868449]
  We propose AlignCoder, a repository-level code completion framework.<n>Our framework generates an enhanced query that bridges the semantic gap between the initial query and the target code.<n>We employ reinforcement learning to train an AlignRetriever that learns to leverage inference information in the enhanced query for more accurate retrieval.
  arXiv  Detail & Related papers  (2026-01-27T15:23:14Z)
• CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments [1.3999481573773072]
  We introduce a multi-language benchmark that evaluates instruction-following capabilities and is to operate on any set of standalone coding problems.<n>Our benchmark evaluates instruction following in two key settings: adherence to pre-defined constraints specified with the initial problem, and the ability to perform refinements based on follow-up instructions.
  arXiv  Detail & Related papers  (2025-10-31T15:47:07Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models [14.665460257371164]
  Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation. We propose AutoAPIEval, a framework designed to evaluate the capabilities of LLMs in API-oriented code generation.
  arXiv  Detail & Related papers  (2024-09-23T17:22:09Z)
• DOCE: Finding the Sweet Spot for Execution-Based Code Generation [69.5305729627198]
  We propose a comprehensive framework that includes candidate generation, $n$-best reranking, minimum Bayes risk (MBR) decoding, and self-ging as the core components. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods.
  arXiv  Detail & Related papers  (2024-08-25T07:10:36Z)
• ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation [4.181146104301203]
  The paper demonstrates this methodology with large language models for code generation. A benchmark called ScenEval is constructed from problems in textbooks, an online tutorial website and Stack Overflow. Experiments found that the performance of ChatGPT decreases with the complexity of the coding task.
  arXiv  Detail & Related papers  (2024-06-18T14:02:20Z)
• Long Code Arena: a Set of Benchmarks for Long-Context Code Models [75.70507534322336]
  Long Code Arena is a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions.
  arXiv  Detail & Related papers  (2024-06-17T14:58:29Z)
• JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models [22.95865189208591]
  We propose JavaBench, a project-level Java benchmark that exercises OOP features. It comprises four Java projects with 389 methods in 106 Java classes. It is attested by 282 undergraduate students, reaching a 90.93/100 average score.
  arXiv  Detail & Related papers  (2024-06-10T06:43:25Z)
• ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
  We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
  arXiv  Detail & Related papers  (2022-03-15T08:25:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.