DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation

• URL: http://arxiv.org/abs/2608.30248v1
• Date: Mon, 31 Aug 2026 04:59:26 GMT
• Title: DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation
• Abstract summary: We introduce DSEffi-Bench, the first benchmark specifically targeting execution efficiency in LLM-generated DS code.<n>We evaluate 16 models across 3 tiers and find that correctness alone fails to characterize efficiency.<n>A human-annotated five-category taxonomy reveals that 79.1% of efficiency deficits extend beyond algorithmic complexity.
• Score: 13.90546553534012
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. We introduce DSEffi-Bench, the first benchmark specifically targeting execution efficiency in LLM-generated DS code, comprising 1,000 instances across 10+ DS libraries with stress-testing harnesses and human-validated references. Evaluating 16 models across 3 tiers, we find that correctness alone fails to characterize efficiency: GPT-5.4 leads in correctness (Pass, 66.9\%) but its efficiency score (B$|$P, 71.7\%) nearly matches GPT-5.4-mini (71.6\%), which solves 47 fewer tasks; Kimi-K2.5 ranks lowest in correctness among frontier models (40.2\%) yet achieves the highest efficiency score (73.6\%) across all 16 models. A human-annotated five-category taxonomy reveals that 79.1\% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes, with distinct failure profiles across model tiers and libraries. Two exploratory experiments provide initial evidence that these diagnostics can guide improvement, yielding up to +14.7\% efficiency gains via taxonomy-guided optimization and approaching Claude-Opus-4.6 Best@3 in efficiency at 13.0$\times$ lower cost via library-conditioned routing.

Related papers

• APEX: Automated Prompt Engineering eXpert with Dynamic Data Selection [60.504476571531]
  Large Language Models are highly sensitive to prompt formulation, necessitating automatic prompt optimization to unlock their full potential.<n>Current methods treat the development dataset as a static benchmark, wasting significant compute budget on uninformative data.<n>We introduce APEX (Automatic Prompt Engineering eXpert), a novel framework that optimize the data usage alongside the prompt search.
  arXiv  Detail & Related papers  (2026-06-09T21:22:06Z)
• CODESTRUCT: Code Agents over Structured Action Spaces [14.039220358416104]
  LLM-based code agents treat repositories as unstructured text, applying edits through brittle matching string.<n>We propose reframing the as a structured action space where agents operate on named AST entities rather than text spans.<n>Our framework, CODESTRUCT, provides readCode for retrieving complete syntactic units and editCode for applying syntax-validated transformations to semantic program elements.
  arXiv  Detail & Related papers  (2026-04-07T03:58:10Z)
• TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation [15.051178553018746]
  We present textbftextsctrace, the first benchmark to explicitly assess efficiency in Large Language Models (LLMs)<n>textsctrace includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency degradations often overlooked by small-scale tests.<n>Our results establish efficiency as an essential dimension of code translation and position textsctrace as a principled foundation for efficiency-oriented evaluation.
  arXiv  Detail & Related papers  (2026-03-17T13:05:54Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Code [2.8815599572682906]
  Pluto is a benchmark and evaluation framework designed to assess the efficiency of LLM-generated Verilog designs.<n>LLMs can achieve high functional correctness, reaching 78.3% at pass@1, but their synthesis efficiency still lags behind expert-crafted implementations.
  arXiv  Detail & Related papers  (2025-10-16T14:57:01Z)
• FGIT: Fault-Guided Fine-Tuning for Code Generation [11.557977684152233]
  We propose Fault-Guided Fine-Tuning (FGIT) to enhance large language models' code generation.<n>Our method achieves an average relative improvement of 6.9% on pass@1 benchmarks.<n>Our technique demonstrates strong generalization with performance improvements ranging from 3.8% to 19.1% across diverse instruction-tuned LLMs.
  arXiv  Detail & Related papers  (2025-03-21T07:23:26Z)
• Scattered Forest Search: Smarter Code Space Exploration with LLMs [55.71665969800222]
  We propose SCATTERED FOREST SEARCH (SFS), a novel approach that improves solution diversity and better exploits feedback during evolutionary search.<n>Our approach scales more efficiently than existing search techniques, including tree search, line search, and repeated sampling.
  arXiv  Detail & Related papers  (2024-10-22T01:58:29Z)
• KAN-RCBEVDepth: A multi-modal fusion algorithm in object detection for autonomous driving [2.382388777981433]
  This paper introduces the KAN-RCBEVDepth method to enhance 3D object detection in autonomous driving. Our unique Bird's Eye View-based approach significantly improves detection accuracy and efficiency. The code will be released in urlhttps://www.laitiamo.com/laitiamo/RCBEVDepth-KAN.
  arXiv  Detail & Related papers  (2024-08-04T16:54:49Z)
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [54.05511925104712]
  We propose a simple, effective, and data-efficient method called Step-DPO. Step-DPO treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters.
  arXiv  Detail & Related papers  (2024-06-26T17:43:06Z)
• Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data [122.282521548393]
  Contrastive Language-Image Pre-training (CLIP) has become the standard for cross-modal image-text representation learning.<n>We introduce HELIP, a cost-effective strategy that improves CLIP models by exploiting challenging text-image pairs within existing datasets in continuous training.
  arXiv  Detail & Related papers  (2023-05-09T07:00:17Z)
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning [91.13797346047984]
  We introduce ADAHESSIAN, a second order optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates. We show that ADAHESSIAN achieves new state-of-the-art results by a large margin as compared to other adaptive optimization methods.
  arXiv  Detail & Related papers  (2020-06-01T05:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.