🆕 Today's Digest — Harnesses / Meta-Harnesses¶
New papers/posts collected 2026-07-24.
9 new today
📋 In brief¶
This batch is unified by a single meta-question: how do you build and honestly evaluate the scaffolding around LLMs, not the models themselves? Several papers make the explicit empirical claim that harness design dominates model scale — DocOps finds harness selection is the primary performance driver, and PRO-LONG demonstrates that a minimal log-and-search harness beats specialized memory systems at a fraction of the token cost. A second strong thread is adversarial stress-testing of harnesses: IssueTrojanBench and OpenSkillRisk both find that even state-of-the-art agent frameworks (including Claude Code) fail to reliably block malicious inputs, exposing a gap between benchmark performance and real-world robustness. Multi-agent orchestration — decomposing tasks across a Manager plus specialized sub-agents — appears as the dominant structural pattern in GS-Agent, EmoAgent-R1, EvoDRC, and AutoGlue, suggesting the field is converging on hierarchical delegation as a default architecture. Cutting across all of this is a growing preoccupation with evaluation integrity: WorkBuddy Bench uses CVE reverse-engineering to resist contamination, DocOps uses artifact-level verifiers, and several papers restrict metrics to domain-specific instruments rather than aggregate scores, signaling that the community is tightening what "a passing harness" actually means.
📝 Today's papers¶
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction¶
Tencent WorkBuddy Bench is a 260-task, four-domain coding-agent benchmark (Code, Web, Office, Security) built around a shared, open, Harbor-style task-directory harness and run reproducibly on two agent backends. Contamination resistance comes from reverse-engineering real commits and CVEs into colloquial, role-played prompts whose text is not web-searchable, backed by dataset versioning rather than secrecy. No suite-wide average is reported because the four subsets use incompatible scoring instruments.
GS-Agent: Creating 4D Physical Worlds With Generative Simulation¶
GS-Agent is a multi-agent framework that converts natural language prompts into physically plausible 4D worlds (dynamic 3D scenes over time) by placing a physics engine in the loop alongside three specialized agents: a Manager, an Entity Agent, and a Render Agent. Unlike text-to-video diffusion models that generate plausible pixels, GS-Agent writes executable simulation code that produces ground-truth physics states, enabling direct kinematic extraction and autonomous error recovery. It outperforms Sora2 and Wan2.2 on physical plausibility and instruction fidelity while enabling fine-grained user control via natural language.
EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization¶
EmoAgent-R1 trains a single MLLM to dynamically select and instantiate specialized emotion-reasoning agents at inference time, replacing static monolithic prompting. A two-phase pipeline uses answer-conditioned CoT cold-start training followed by Progressive Group-Relative Policy Optimization (P-GRPO), which adds PMI-inspired token-level modulation to GRPO's uniform credit-assignment scheme. Experiments on MER-UniBench show gains over static-prompt and standard GRPO baselines on sentiment analysis, basic emotion recognition, and fine-grained emotion detection.
IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests¶
IssueTrojanBench is a fully automated benchmark that evaluates AI coding agents (Cursor, Claude Code, Codex Desktop) against indirect prompt injection delivered through malicious GitHub issues. Across 4,176 experimental runs spanning 696 adversarial artifacts, 66.5% of attacks penetrate all guardrails; agent-framework-level defenses provide almost no additional protection beyond the underlying LLM.
OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills¶
OpenSkillRisk is a benchmark of 263 real-world risky third-party skills drawn from public skill marketplaces, designed to evaluate whether CLI agent systems can detect and avoid latent execution-time safety risks. It evaluates 3 CLI agent harnesses and 13 LLMs using both execution-level and cognition-level metrics, finding that even the best system (Claude Code + Claude Sonnet 4.6) still triggers unsafe execution in 17.87% of cases.
PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning¶
PRO-LONG is a minimal agent harness that replaces heuristic memory systems with a single design choice: append every observation, action, and outcome to a plain log file, then let a coding agent search that log programmatically. On ARC-AGI-3, it matches or beats specialized harnesses (up to 76.1% pass@1) while consuming 4.2–5.8× fewer tokens, and reaches 97.4% best@2 with Fable 5 at 2,000 actions for $1,750.
EvoDRC: A Self-Evolving Agentic Framework for Automated DRC Violation Repair¶
EvoDRC is a skill-evolution framework that deploys LLM repair agents on block-level chip layouts to automate post-route DRC violation repair. It transfers repair knowledge from a reference design, decomposes large layouts into bounded crops, and continuously evolves layer-specific skill files using a traceable knowledge database. On seven block-level designs from the DAC26-DRC-Benchmark, EvoDRC achieves a 73.5% reduction in design rule violations (DRVs).
DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations¶
DocOps is a deterministically verifiable benchmark for evaluating LLM agents on end-to-end document manipulation across DOCX, XLSX, PPTX, and PDF formats, using a two-axis taxonomy (3 operation families × 4 difficulty tiers) and artifact-level verifiers that check both requested edits and structural preservation invariants. The central empirical finding is that harness selection—not model scale—is the primary performance driver, with even the best configuration (GPT-5.5 + Codex + skills) achieving only a 0.671 pass rate. Three pervasive failure modes are identified: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata.
Bridging Behavior and Implementation: Automated Java Glue Code Generation for Behavior-Driven Development¶
AutoGlue is a hierarchical multi-agent framework that automatically generates Java BDD glue code from natural-language Cucumber/Gherkin steps. It decomposes the task into behavior interpretation, context retrieval, and code generation, achieving 58.7% API F1 improvement over few-shot prompting and producing directly usable code for 46.1% of evaluated steps.