NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems

🕒 Published (v1): 2026-06-25 16:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NOVA is a level-aware agent harness that automates architecture evolution for industrial recommender systems by coupling an SGD-inspired "architecture gradient" (aggregating prior modifications, verification diagnostics, metric feedback, and trajectory memory) with a progressive verification cascade that intercepts silent failures before expensive training. Deployed at Tencent on a >1B-user advertising system, it achieves 60.0% effective pass rate on literature-to-production transfer tasks and cuts human-attended cycle time by >13×.

Problem

Existing automation (AutoML, generic LLM coding agents) cannot reliably evolve production recommender architectures: AutoML only tunes local hyper-parameters and misses cross-module topology changes, while coding agents optimize for software-level signals (compilation, unit tests) that do not detect semantic failures—runnable candidates that degrade AUC, calibration, or GMV by silently violating recommender-specific invariants (e.g., wrong attention masking direction, degenerated self-attention, corrupted logit-fusion path).

Method

NOVA represents each candidate architecture as \(A_t = (G_t, \phi_t, F_t)\) (model graph, structural hyper-parameters, feature configuration) and searches over a production-constrained space \(\mathcal{A}_\Omega\).

Architecture gradient. At iteration \(t\), NOVA computes

\[g_t = \mathrm{Grad}(e_{t-1},\, V_t,\, \Delta J_t,\, H_t),\]

an SGD-inspired (non-differentiable) update signal that aggregates the previous modification \(e_{t-1}\), verification diagnostics \(V_t\), offline metric change \(\Delta J_t\), and trajectory memory \(H_t\). The gradient encodes three outputs: weak components (bottleneck identification), modification directions (next candidates to explore), and forbidden directions (failed/negative patterns to avoid). The next modification is selected as

\[e_t^* \in \arg\max_{e \in \mathcal{E}} \mathrm{Score}(e;\, g_t, H_t) \quad \text{s.t.} \quad \mathrm{Apply}(A_t, e) \in \mathcal{A}_\Omega.\]

Verification cascade. Before incurring GPU training cost, each candidate passes four sequential gates: (1) structure-semantic gate—checks shape/dtype, feature-to-token mapping, attention masking direction, and logit-fusion logic against skill specifications; (2) local testing gate—single-machine executability; (3) offline inner loop—AUC-based ranking over verified survivors; (4) online outer loop—GMV/Bias validation in A/B test. Failed candidates at any gate are recorded in \(H_t\) as forbidden directions, feeding back into the next \(g_{t+1}\) rather than acting only as post-hoc filters.

Level-aware control. Tasks are classified L1–L4 by complexity (atomic tuning → open-ended innovation). Tasks within covered skill specifications run in AutoRun; uncovered or high-risk decisions route to Copilot (human-in-the-loop confirmation).

Multi-agent implementation. A Main Agent coordinates seven specialized stage agents (initialization, solution design, code modification, semantic verification, local test, offline evaluation, online experiment) and maintains the architecture gradient across rounds.

Key Contributions

• Architecture gradient: an SGD-analogue, non-differentiable update signal for discrete constrained architecture search that unifies prior modifications, verification diagnostics, metric feedback, and trajectory memory into a single directed search signal.
• Silent-failure-aware verification cascade: a four-stage (semantic → local → offline → online) progressive filter that converts failure diagnostics into reusable forbidden directions, making verification part of the search signal rather than a reject gate.
• Level-aware harness (L1–L4) with AutoRun/Copilot routing: task complexity classification that matches automation depth to risk, enabling autonomous operation on covered modifications while escalating to human oversight for novel or high-risk changes.
• Industrial deployment: production integration at Tencent serving >1B users, evaluated end-to-end from offline AUC to live GMV/Bias in A/B tests.

Results

• L2 ScaleUp (effective pass rate, EPR): NOVA 54.5% vs. coding-agent baseline (exact figure not shown in excerpt but described as highest).
• L3 Literature-to-Production (EPR): NOVA 60.0%; literature pass rate (LPR) 86.7%—"substantially outperforming coding-agent baselines and more than doubling the EPR of the human expert loop."
• Cycle time: >13× reduction in human-attended time for one literature-to-production cycle.
• Online A/B (L3 candidate, three pCVR objectives): GMV +1.25%, +1.70%, +2.02%; pCVR bias reduced by 58.8%, 66.7%, 37.3%.
• Silent failure reduction: fewer runnable-but-negative candidates compared to coding-agent baselines (quantified results truncated in excerpt).

Limitations

• Skill specifications for the semantic gate are manually curated from historical seed rules and LLM-assisted checks; the gate does not generalize to architectures outside its covered specification set—uncovered cases fall through to expensive offline/online evaluation.
• The architecture gradient is an LLM-prompt-based analogue, not a true gradient; its quality depends on LLM instruction-following and may degrade for highly novel (L4) modifications where no historical evidence exists.
• Offline AUC is used as the proxy for online GMV/Bias, introducing a potential offline-online gap that verification cannot close.
• L3/L4 tasks still require Copilot (human confirmation) for uncovered modifications, limiting full automation for the most complex cases.
• Evaluation is limited to two task types (L2 ScaleUp, L3 Literature-to-Production) in a single industrial system; generalizability to other recommendation architectures or domains is unverified.

Relevance to Harnesses / Meta-Harnesses

NOVA is a direct instantiation of a domain-specific meta-harness: it wraps multiple specialized agents (code-generation, semantic-verification, offline-evaluation, online-experiment) inside a coordinating Main Agent that maintains global state (trajectory memory \(H_t\), architecture gradient \(g_t\)) and enforces a structured feedback loop across iterations—exactly the pattern of a meta-harness that orchestrates sub-agents rather than calling tools directly. The level-aware routing (L1–L4, AutoRun/Copilot) demonstrates how harnesses can embed risk-aware escalation policies to gate automation depth. The "verification cascade as gradient denoising" design is a novel harness primitive: failed sub-agent outputs are not discarded but recycled as forbidden directions, showing how meta-harnesses can turn negative feedback from inner agents into search signal for future iterations. For researchers tracking this topic, NOVA illustrates how production-grade harnesses add correctness-verification layers absent from general-purpose agent scaffolds (e.g., SWE-agent), and provides a concrete blueprint for domain-constrained agentic search with human-in-the-loop fallback.