Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge

• URL: http://arxiv.org/abs/2605.08518v1
• Date: Fri, 08 May 2026 22:00:58 GMT
• Title: Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
• Abstract summary: We revisit the CODS 2025 assetopslive challenge, a privacy-aware Codabench competition on industrial multi-agent orchestration.<n>We combine final rank sheets, a 300- server log, 149-team registrations, best-submission exports, the organizer winners report, the companion assetopslive system paper, and verified planning-track source trees.<n> hidden evaluation changes the story: public and private scores correlate moderately in planning but negatively in execution.
• Score: 3.5112866130906926
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design patterns were rewarded. We revisit the CODS 2025 \assetopslive{} challenge, a privacy-aware Codabench competition on industrial multi-agent orchestration built on \assetops{}. We combine final rank sheets, a 300-submission server log, 149-team registrations, best-submission exports, the organizer winners report, the companion \assetopslive{} system paper, and verified planning-track source trees. Five results stand out. First, the public planning leaderboard saturates at 72.73\%, and richer prompts do not improve that peak. Second, hidden evaluation changes the story: public and private scores correlate moderately in planning ($r{=}0.69$) but negatively in execution ($r{=}{-}0.13$), with several 45.45\% public execution systems reaching 63.64\% on the hidden set. Third, the \tmatch{} term is numerically almost inert in the official composite -- combined on a 0--1 scale with 0--100 percentage scores, it contributes at most 0.05 points per track, and rescaling would swap the top two teams. Fourth, the competition is operationally account-based but substantively team-based: 149 registered teams reduce to 24 with non-zero public scores and 11 fully ranked, while 52.3\% of deduplicated registrations list multiple usernames. Fifth, successful execution methods mostly improve guardrails -- response selection, contamination cleanup, fallback, and context control -- rather than novel agent architectures. These findings identify which behaviors the evaluation rewarded, and motivate scale-aware composites, skill-level diagnostics, and versioned artifact release.

Related papers

• Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead [9.25138311822007]
  Small differences on coding-agent leaderboards are often read as an ordering of systems.<n>We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models.<n>We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution.
  arXiv  Detail & Related papers  (2026-09-15T16:29:12Z)
• Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports [5.580052437190499]
  We reimplement the stated matching rule for only five of twelve inspected systems.<n>Ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings.<n>On an external, human-adjudicated set, no mechanical matcher---lexical, embedding, or entailment---agrees with the reviewers more than seven times in ten.
  arXiv  Detail & Related papers  (2026-09-01T04:45:44Z)
• FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training [0.0]
  Open-ended language-model evaluation often substitutes another model or a small preference panel for a missing answer key.<n>We introduce FlavourBench, which compiles dense answer maps from a versioned culinary environment.<n>We evaluate 27 frontier endpoints on the same 534 substitution, pairing, and constraint tasks, yielding 14,418 complete model-task observations.
  arXiv  Detail & Related papers  (2026-08-20T21:14:52Z)
• Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short [51.667769734342635]
  Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models.<n>We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them.<n>We show that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T11:57:17Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation [4.781417767575192]
  This report presents TAP-JEPA, our runner-up submission to the EPIC-KITCHENS-100 (EK-100) Action Anticipation Challenge at EgoVis 2026.<n>The task is to anticipate the next verb, noun, and verb-noun action from an egocentric clip that ends before the target action begins.<n>Instead of fine-tuning a large video backbone, TAP-JEPA builds a compact anticipation model on frozen V-JEPA 2.1 features.
  arXiv  Detail & Related papers  (2026-05-30T10:23:12Z)
• TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories [61.51984029109709]
  We introduce TraceGraph, a graph-based framework that turns released multi-model agent trajectories into shared decision landscapes.<n>For each task, TraceGraph builds a graph over observable action-observation states from pooled rollouts before model identity is introduced.<n>It then overlays outcome-informed productive cores and trap regions, and summarizes each rollout with three events: Access, Trap exposure, and Repair.
  arXiv  Detail & Related papers  (2026-05-29T13:40:31Z)
• Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference [1.2299000423193074]
  TokenArena is a continuous benchmark that measures inference at granularity endpoint along five core axes.<n>We release the framework, schema, probe and eval harness, and a v1.0 leaderboard snapshot under CC BY 4.0.
  arXiv  Detail & Related papers  (2026-05-01T00:05:54Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• Second Competition on Presentation Attack Detection on ID Card [12.609487726543774]
  20 teams were registered, and 74 submitted models were evaluated.<n>For Track 1, the "Dragons" team reached first place with an Average Ranking and Equal Error rate (EER) of AV-Rank of 40.48% and 11.44% EER, respectively.<n>For the more challenging approach in Track 2, the "Incode" team reached the best results with an AV-Rank of 14.76% and 6.36% EER, improving on the results of the first edition of 74.30% and 21.87% EER, respectively.
  arXiv  Detail & Related papers  (2025-07-27T20:18:52Z)
• The Leaderboard Illusion [61.27964089648608]
  Arena has emerged as the go-to leaderboard for ranking the most capable AI systems.<n>We identify systematic issues that have resulted in a distorted playing field.
  arXiv  Detail & Related papers  (2025-04-29T15:48:49Z)
• The Fourth Monocular Depth Estimation Challenge [100.38910331027051]
  This paper presents the results of the fourth edition of the Monocular Depth Estimation Challenge (MDEC)<n>It focuses on zero-shot generalization to the SYNS-Patches benchmark, a dataset featuring challenging environments in both natural and indoor settings.<n>The challenge received a total of 24 submissions that outperformed the baselines on the test set.<n>The challenge winners improved the 3D F-Score over the previous edition's best result, raising it from 22.58% to 23.05%.
  arXiv  Detail & Related papers  (2025-04-24T17:59:52Z)
• Soft Condorcet Optimization for Ranking of General Agents [44.90789674063613]
  We describe a novel ranking scheme inspired by social choice frameworks, called Soft Condorcet Optimization (SCO)<n>SCO rankings are on average 0 to 0.043 away from the optimal ranking in normalized Kendall-tau distance across 865 preference profiles from the PrefLib open ranking archive.<n>SCO ranking provides the best approximation to the optimal ranking, measured on held-out test sets, in a problem containing 52,958 human players across 31,049 games of the classic seven-player game of Diplomacy.
  arXiv  Detail & Related papers  (2024-10-31T18:17:39Z)
• The 7th AI City Challenge [87.23137854688389]
  The AI City Challenge's seventh edition emphasizes two domains at the intersection of computer vision and artificial intelligence. The 2023 challenge had five tracks, which drew a record-breaking number of participation requests from 508 teams across 46 countries. The participating teams' top performances established strong baselines and even outperformed the state-of-the-art in the proposed challenge tracks.
  arXiv  Detail & Related papers  (2023-04-15T08:02:16Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)
• Context-aware Retail Product Recommendation with Regularized Gradient Boosting [4.524282268368634]
  In the FARFETCH Fashion Recommendation challenge, the participants needed to predict the order in which various products would be shown to a user. The dataset comprises over 5,000,000 recommendation events, 450,000 products and 230,000 unique users. We have designed a unique context-aware system that takes the similarity of a product to the user context into account to rank products more effectively.
  arXiv  Detail & Related papers  (2021-09-17T14:08:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.