ShapeBench: A Scalable Benchmark and Diagnostic Suite for Standardized Evaluation in Aerodynamic Shape Optimization

• URL: http://arxiv.org/abs/2605.20763v1
• Date: Wed, 20 May 2026 06:05:57 GMT
• Title: ShapeBench: A Scalable Benchmark and Diagnostic Suite for Standardized Evaluation in Aerodynamic Shape Optimization
• Abstract summary: We introduce ShapeBench, an open-source ASO benchmark with a unified API spanning 103 tasks across eight categories shape and multiple optimization regimes.<n>Each ShapeBench task includes a validated surrogate for fast search, enabling systematic fidelity-gap analysis.<n>Results on ShapeBench demonstrate substantial variance in rankings across shape categories and problem formulations, with mean pairwise Spearman $= 0.013$, so single-task conclusions do not reliably generalize across problem classes.<n>The benchmark is also far from saturation; classical methods are rarely applicable across all shape categories and tasks, further highlighting the need for more general-purpose approaches.
• Score: 9.151054983399924
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rapid progress in aerodynamic shape optimization (ASO) has outpaced currently-available standardized evaluation frameworks. Fair comparison requires a unified benchmark spanning diverse shape classes, objective formulations, and matched-budget state-of-the-art baselines. We introduce ShapeBench, an open-source ASO benchmark with a unified API spanning 103 tasks across eight shape categories and multiple optimization regimes. Each ShapeBench task includes a validated surrogate for fast search; when feasible, a high-fidelity Computational Fluid Dynamics (CFD) pipeline for final verification is available, enabling systematic fidelity-gap analysis. ShapeBench provides a reproducible protocol with well-configured baselines to compare fairly using a consistent budget metric, allowing for comparison among both classical and LLM-driven methods, including general-purpose optimizers and a new domain-specialized evolutionary LLM baseline, ShapeEvolve. Results on ShapeBench demonstrate substantial variance in optimizer rankings across shape categories and problem formulations, with mean pairwise Spearman $ρ= 0.013$, so single-task conclusions do not reliably generalize across problem classes. The benchmark is also far from saturation; classical methods are rarely applicable across all shape categories and tasks, further highlighting the need for more general-purpose approaches.

Related papers

• SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning [4.819678320271634]
  Specification-guided reinforcement learning (RL) provides a principled framework for encoding complex, temporally extended tasks.<n>We introduce SpecRLBench, a benchmark designed to evaluate the capabilities of generalization-based RL methods.
  arXiv  Detail & Related papers  (2026-04-27T17:40:18Z)
• POLCA: Stochastic Generative Optimization with LLM [26.342554900977003]
  We introduce Prioritized Optimization with Local Contextualion (POLCA)<n>POLCA is a scalable framework designed to handle parameterity in optimization.<n>We demonstrate POLCA achieves robust, sample and time-efficient performance.
  arXiv  Detail & Related papers  (2026-03-16T03:07:44Z)
• Principled Algorithms for Optimizing Generalized Metrics in Binary Classification [53.604375124674796]
  We introduce principled algorithms for optimizing generalized metrics, supported by $H$-consistency and finite-sample generalization bounds.<n>Our approach reformulates metric optimization as a generalized cost-sensitive learning problem.<n>We develop new algorithms, METRO, with strong theoretical performance guarantees.
  arXiv  Detail & Related papers  (2025-12-29T01:33:42Z)
• LLM Routing with Dueling Feedback [49.67815163970033]
  We study the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost.<n>We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores.<n>We introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting.
  arXiv  Detail & Related papers  (2025-10-01T12:52:25Z)
• LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning [56.273799410256075]
  The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path. The framework has been tested on general and advanced benchmarks, showing superior performance in terms of search efficiency and problem-solving capability.
  arXiv  Detail & Related papers  (2024-10-03T18:12:29Z)
• A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling [54.05517338122698]
  A popular similarity-based feature upsampling pipeline has been proposed, which utilizes a high-resolution feature as guidance.<n>We propose an explicitly controllable query-key feature alignment from both semantic-aware and detail-aware perspectives.<n>We develop a fine-grained neighbor selection strategy on HR features, which is simple yet effective for alleviating mosaic artifacts.
  arXiv  Detail & Related papers  (2024-07-02T14:12:21Z)
• Generalized Preference Optimization: A Unified Approach to Offline Alignment [54.97015778517253]
  We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases. Our results present new algorithmic toolkits and empirical insights to alignment practitioners.
  arXiv  Detail & Related papers  (2024-02-08T15:33:09Z)
• Improving LSHADE by means of a pre-screening mechanism [0.0]
  The paper introduces an extension to the well-known LSHADE algorithm in the form of a pre-screening mechanism (psLSHADE) The proposed pre-screening relies on the three following components: a specific initial sampling procedure, an archive of samples, and a global linear meta-model of a fitness function. The performance of psLSHADE is evaluated using the CEC2021 benchmark in an expensive scenario with an optimization budget of 102-104 FFEs per dimension.
  arXiv  Detail & Related papers  (2022-04-08T14:52:46Z)
• Batched Data-Driven Evolutionary Multi-Objective Optimization Based on Manifold Interpolation [6.560512252982714]
  We propose a framework for implementing batched data-driven evolutionary multi-objective optimization. It is so general that any off-the-shelf evolutionary multi-objective optimization algorithms can be applied in a plug-in manner. Our proposed framework is featured with a faster convergence and a stronger resilience to various PF shapes.
  arXiv  Detail & Related papers  (2021-09-12T23:54:26Z)
• Automatic selection of basis-adaptive sparse polynomial chaos expansions for engineering applications [0.0]
  We describe three state-of-the-art basis-adaptive approaches for sparse chaos expansions. We conduct an extensive benchmark in terms of global approximation accuracy on a large set of computational models. We introduce a novel solver and basis adaptivity selection scheme guided by cross-validation error.
  arXiv  Detail & Related papers  (2020-09-10T12:13:57Z)
• Stochastic batch size for adaptive regularization in deep network optimization [63.68104397173262]
  We propose a first-order optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework. We empirically demonstrate the effectiveness of our algorithm using an image classification task based on conventional network models applied to commonly used benchmark datasets.
  arXiv  Detail & Related papers  (2020-04-14T07:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.