Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge¶

🕒 Published (v1): 2026-05-08 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Retrospective analysis of the CODS 2025 AssetOpsBench competition, which evaluated multi-agent AI systems on long-horizon Industry 4.0 tasks (predictive maintenance, fault diagnosis, root-cause analysis) under privacy-preserving, hidden-scenario conditions. The central finding is that public and hidden execution scores are statistically uncorrelated (\(\rho = -0.13\), \(n = 13\), \(p = 0.71\)), so leaderboard position does not predict hidden robustness. The analysis surfaces concrete scoring formula errors, ranking instability, and a dichotomy between planning (diffuse, commoditized) and execution (structured, guardrail-dominated) strategies.

Problem¶

Static benchmarks misrepresent real-world agent capability because they reward narrowly specified, easily prompt-optimized tasks while leaving multi-step orchestration, tool-use robustness, and privacy-preserving execution unmeasured. Competition-based evaluation offers a principled alternative, but only if the evaluation infrastructure—scoring formulas, public/hidden split design, metric composition—is itself sound. This paper addresses what a large-scale agentic competition actually measured, what it failed to measure, and how its incentive structure shaped submitted systems.

Method¶

The authors analyze six artifact classes produced by the competition: registration forms (149 teams, 349 declared member slots), a 300-row server submission log, per-track score exports (public and hidden), scoring traces, organizer award reports, and 331 accessible source-code artifacts. Analysis proceeds across five dimensions:

Participation funnel: attrition rates from registration → valid submission → cross-track ranking.
Ranking robustness: Spearman \(\rho\) between public and hidden scores per track; sensitivity sweep over 80 \((\alpha, s)\) configurations varying execution weight \(\alpha\) and t-match rescaling factor \(s\).
Score-composition sensitivity: the composite formula is \(C_t = 0.6\,S_t^{\text{pub}} + 0.3\,S_t^{\text{priv}} + 0.1\,\tau_t\), \(F = 0.4\,C_{\text{plan}} + 0.6\,C_{\text{exec}}\). The t-match semantic term \(\tau_t \in [0,1]\) is on a different scale than \(S \in [0,100]\), contributing at most 0.05 composite points.
Agent cost fingerprint: five axes per domain agent—tokens sent, API call depth, wall-clock duration, phase stability, and strategy variance (CV)—extracted from execution trajectory logs.
Strategy attribution: 331 source artifacts embedded with a sentence-transformer, clustered via HDBSCAN separately per track; class TF-IDF extracts archetype-defining tokens.

The two competition tracks isolate orthogonal capabilities: Track 1 fixes the executor and varies the planning prompt/DAG construction (track1_planning.py); Track 2 fixes the plan and varies workflow execution logic (track2_execution.py, replacing SequentialWorkflow with a DynamicWorkflow supporting parallel paths, fallback, and context aggregation). All submissions use LLaMA-3-70B on 11 public scenarios (Phase 1) and 11 novel hidden scenarios (Phase 2).

Key Contributions¶

First competition-track benchmark combining agentic evaluation, physical-asset industrial domain (chillers, air-handling units), and privacy-preserving hidden-scenario execution.
Empirical demonstration that public execution leaderboard is a zero-signal predictor of hidden performance (\(\rho = -0.13\)).
Identification of a score-composition error: the t-match semantic term is numerically inert (≤0.05 composite points) due to mismatched scales; correcting it changes the top-two ordering.
Five-axis agent cost fingerprint revealing a key inversion: cheapest domain (TSFM, 35K tokens) has highest prompt sensitivity (CV = 1.68); most expensive domain (WO, 244K tokens) is most robust (CV = 0.76).
HDBSCAN-based archetype taxonomy: Track 2 (execution) collapses into 3 tight clusters (1.6% noise) vs. Track 1 (planning) scattering into 7 diffuse clusters (32% noise)—execution strategy is structured, planning is commoditized.
Release of competition scenarios, scoring traces, and a portable diagnostics checklist for future agentic benchmark designers.

Results¶

Planning saturation: public leaderboard saturates at 72.73%; only 8 distinct scores across 20 teams.
Execution correlation: \(\rho_{\text{exec}}(\text{pub}, \text{priv}) = -0.13\) (\(n=13\), \(p=0.71\))—indistinguishable from zero. \(\rho_{\text{plan}}(\text{pub}, \text{priv}) = 0.69\) with mean drop of \(-11.30\) points.
Ranking instability: official top-1 holds in only 44% of 80 scoring configurations; mean Kendall's \(\bar{\tau} = 0.61\) (\(\sigma = 0.19\)) between official and alternative rankings.
t-match scale error: rescaling \(\tau_t \times 100\) swaps top-two teams and exchanges 3rd/4th; equal track weights also flips top-two.
Hallucination coupling: \(r \approx -0.93\) (Pearson) between hallucination rate and overall task quality across all 22 scenarios—hallucination is a leading failure indicator, not a standalone metric.
Work-order hardness: the three hardest scenarios in both development and evaluation phases are WO tasks, each with hallucination rates above 0.73.
Strategy pattern: execution top performers are "guardrail engineers"—they improve response cleanup, fallback logic, and context control, not agent architectures. The dominant planning archetype (P1, 32.4%) is KB-grounded prompting.
Funnel attrition: 149 registered → 24 with valid scored submission → 11 ranked across both tracks; 17.7% of 300 submissions failed conformance checks.

Limitations¶

Small hold-out sample (\(n = 13\) for execution correlation) limits statistical power; \(p = 0.71\) cannot distinguish zero signal from modest positive signal.
Source-code analysis covers only 331 accessible artifacts; strategy taxonomy may not generalize to withheld submissions.
All submissions use a single base model (LLaMA-3-70B), preventing conclusions about model-agnostic harness design.
Hidden evaluation uses only 11 novel scenarios; scenario-level variance dominates any capability signal.
Multi-username team accounts (52.3% of teams) create reconciliation overhead and potential attribution errors invisible to per-account export analysis.
Guardrail-rewarding scoring structure systematically underweights novel architectural contributions, skewing the strategy distribution toward deployment engineering.

Relevance to Harnesses / Meta-Harnesses¶

The competition framework is itself a meta-harness: it orchestrates four domain-specific agents (IoT, FMSR, TSFM, WO) through a Sensing→Reasoning→Actuation pipeline, with separate tracks surgically isolating planning (DAG construction) from execution (workflow orchestration), making it a direct case study in harness decomposition. The key finding—that public and hidden execution scores do not correlate—is a structural warning for anyone designing evaluation harnesses with a development/holdout split: harness scoring formulas must be validated for scale consistency (the t-match bug) and the public signal must be tested for predictive validity before the hidden phase. The cost fingerprint methodology (tokens, API calls, wall-clock, phase stability, CV per agent) is a portable instrumentation pattern directly applicable to any multi-agent harness seeking to distinguish task-intrinsic cost from prompt-sensitivity artifacts. The archetype taxonomy result—execution strategy converges while planning stays diffuse—suggests that execution harnesses should be designed with tight interface contracts to reduce the strategy space and enable meaningful comparison.