Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild

• URL: http://arxiv.org/abs/2605.24213v1
• Date: Fri, 22 May 2026 20:54:30 GMT
• Title: Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild
• Abstract summary: Evaluation harnesses orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting.<n>We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause.
• Score: 18.82003667637388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluation harnesses are software systems that orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting. Despite their critical role in machine learning infrastructure, their operational challenges and engineering concerns have received limited attention so far. We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause. Most harness operational challenges concentrate in the Specification stage (41.4% of issues), where harnesses integrate external models, datasets, and scoring judges. The three most frequent root causes of operational challenges are unimplemented features (24.3%), documentation gaps (20.3%), and missing input validation (17.2%), which together account for 61.7% of classified issues, spanning both defects in existing functionality and capability gaps that block intended workflows. Root causes also vary by workflow stage: environment incompatibility and external dependency breakage account for 36.2% of provisioning issues, whereas algorithmic error (25.9%) and validation gap (22.5%) dominate assessment issues. Together, these contributions establish an empirical foundation for treating evaluation engineering as a distinct software engineering concern.

Related papers

• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents [56.400332595452966]
  We introduce SearchAuditBench, a benchmark that evaluates whether auditors can localize, attribute, and repair search-agent failures.<n>SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks.<n>We propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures.
  arXiv  Detail & Related papers  (2026-08-05T09:09:00Z)
• ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents [23.840537431318296]
  We present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves and how it achieves it.<n>ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services.<n>A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification) anchored by 12,541 task-specific items.
  arXiv  Detail & Related papers  (2026-07-30T11:18:47Z)
• Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs [86.49905745865038]
  Soohak is a 439-problem benchmark newly authored from scratch by 64 mathematicians.<n>The dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.
  arXiv  Detail & Related papers  (2026-05-09T17:14:22Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• Exploring Fairness Interventions in Open Source Projects [4.637328271312331]
  bias in machine learning models can have adverse effects in sectors such as criminal justice and healthcare.<n>A range of machine learning fairness interventions have been developed, aiming to mitigate bias and promote the creation of more equitable models.<n>Despite the growing availability of these interventions, their adoption in real-world applications remains limited.
  arXiv  Detail & Related papers  (2025-07-09T16:57:59Z)
• HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems [2.4241401076864]
  The HackerRank-ASTRA Benchmark introduces project-based coding problems that mirror real-world scenarios.<n>It evaluates model consistency through 32 runs (k = 32) and median standard deviation.<n>The top three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved comparable average scores of 75%.
  arXiv  Detail & Related papers  (2025-01-31T23:47:02Z)
• SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories [55.161075901665946]
  Super aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges, and 602 automatically generated problems for larger-scale development. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios.
  arXiv  Detail & Related papers  (2024-09-11T17:37:48Z)
• Generative AI for Requirements Engineering: A Systematic Literature Review [1.6986294649170766]
  Generative pretrained transformer models dominate current applications.<n>Industrial adoption remains nascent, with over 90% of studies corresponding to early stage development.<n>Despite the transformative potential of GenAI based RE, several barriers hinder practical adoption.
  arXiv  Detail & Related papers  (2024-09-10T02:44:39Z)
• A Comparative Study of Android Performance Issues in Real-world Applications and Literature [16.259447959343067]
  We conduct a large-scale comparative study of Android performance issues in real-world applications and literature.<n>Our results show a substantial divergence exists in the primary performance concerns of researchers, developers, and users.<n>It is crucial for our community to intensify efforts to bridge these gaps and achieve comprehensive detection and resolution of performance issues.
  arXiv  Detail & Related papers  (2024-07-06T14:43:40Z)
• ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints [31.90180597239974]
  Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems.<n>We introduce a new diagnostic benchmark, ActionReasoningBench, which encompasses 8 domains and includes questions for up to 19 action sequences.<n>This benchmark rigorously evaluates Large Language Models (LLMs) across six key RAC dimensions.
  arXiv  Detail & Related papers  (2024-06-06T13:15:37Z)
• Tool-Augmented Reward Modeling [58.381678612409]
  We propose a tool-augmented preference modeling approach, named Themis, to address limitations by empowering RMs with access to external environments. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources. In human evaluations, RLHF trained with Themis attains an average win rate of 32% when compared to baselines.
  arXiv  Detail & Related papers  (2023-10-02T09:47:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.