Skip to content

🗓️ 2025 in Review — Harnesses / Meta-Harnesses

Compact conclusion of 400 digested 2025 papers.

The year in one paragraph

The 2025 literature reveals a decisive shift from hand-crafted single-agent pipelines toward composable, self-constructing multi-agent harnesses — meta-level frameworks that orchestrate agent populations rather than individual models. The dominant pattern is decomposition: tasks are split across specialized sub-agents (planners, coders, critics, verifiers) coordinated by a controller that manages routing, context, and feedback. Three concurrent pressures are driving this: the context-window ceiling on single-agent depth, the reliability gap in monolithic LLM calls, and the explosion of domain-specific deployment demands (medical, financial, scientific). By late 2025, the frontier has moved again toward automated harness construction — systems that design, compile, and self-optimize their own agent topologies from task descriptions alone — and toward RL-trained meta-controllers that replace human-designed orchestration logic.

🕒 Timeline

  • 2025-07: Emergence of 30+ agent scientific systems (cmbagent) and hierarchical theatrical orchestration (HAMLET, HIMA); FSM-based auto-construction demonstrated (MetaAgent)
  • 2025-08: Survey consolidation of OS-agent field; cost-efficiency realism sets in — observation masking beats LLM summarization; VistaWise shows low-parameter embodied agents can match API-scale competitors
  • 2025-09: Domain-specialization wave: medical (RadAgents, MedLA, AMANDA), financial (FinDebate, CodeGym), scientific (Re4, MOSAIC) harnesses proliferate; red-teaming meta-frameworks emerge (SafeSearch)
  • 2025-10: Meta-harness evaluation infrastructure matures — self-evolving safety benchmarks (AgenticEval), no-human annotation at Walmart scale (ScalingEval), plug-and-play scientific knowledge (xKG, PLAGUE)
  • 2025-11: Automated workflow generation breaks through — A²Flow eliminates hand-crafted operators via MCTS over auto-derived abstractions; Agint compiles natural-language specs to typed DAGs; enterprise production deployment (MAFA at JPMorgan, 1M-utterance backlog cleared)
  • 2025-12: RL-augmented harnesses set new performance ceilings (AgentMath-30B: 90.6% AIME24); empirical workflow comparison shows agentic MCP flows beat rigid expert pipelines at small model scales (Workflows vs Agents); context management hardened to O(1) (MemSearcher)

📌 Key papers

  • EnCompass (Dec 2025, NeurIPS 2025) — Formalizes probabilistic angelic nondeterminism to decouple agent workflow logic from search strategy; a single @encompass.compile annotation enables best-of-N, beam, or MCTS without rewriting agent code.
  • A²Flow (Nov 2025, AAAI) — Eliminates the hand-crafted operator bottleneck in automated workflow generation via MCTS over LLM-derived reusable abstractions; +19.3% on embodied/game tasks versus prior MCTS baselines.
  • AgentMath (Dec 2025, ICLR 2026) — Three-stage automated data synthesis + agentic RL with 4–5× asynchronous distributed training speedup; 90.6% on AIME24, surpassing o3-mini and Claude-Opus-4.0-Thinking.
  • Agint (Nov 2025, NeurIPS 2025) — Compiles natural-language specifications into typed, effect-aware DAGs through a six-tier type hierarchy with a hybrid LLM/function JIT runtime and Unix-style CLI toolchain.
  • AgenticEval (Sep 2025, ACL 2026) — Self-evolving LLM safety benchmark that iteratively hardens test cases from evaluation failures; demonstrates GPT-5 EU AI Act compliance collapsing from 72.5% to 36.4% across three iterations.
  • InfiAgent (Sep 2025, ICLR 2026) — DAG-based multi-agent framework with Git-style self-evolution of topology; outperforms ADAS by 9.9% average accuracy across five benchmarks.
  • MetaAgent (Jul 2025, ICML 2025) — Automatically constructs multi-agent FSM topologies from task descriptions by LLM-generating agents/states/transitions and merging redundant states; matches or exceeds human-designed baselines.
  • MemSearcher (Nov 2025, ACL 2026) — Replaces ReAct's ever-growing history with a 1,024-token O(1) natural-language memory trained via multi-context GRPO; outperforms ReAct-style baselines on seven QA benchmarks.
  • CodeGym (Sep 2025, ICLR 2026) — Scalable pipeline converting static coding problems into verifiable multi-turn tool-use RL environments; GRPO-trained Qwen2.5-32B gains +14.0 points on ALFWorld OOD.
  • SafeSearch (Sep 2025, ICML 2026) — Automated red-teaming framework showing LLM-based search agents are broadly vulnerable (90.5% peak ASR on GPT-4.1-mini) to single-injected-page adversarial attacks; baseline defenses fail.
  • The Complexity Trap (Aug 2025, NeurIPS 2025) — Simple observation masking matches or beats LLM-based trajectory summarization on SWE-bench Verified at O(1) overhead; a calibration anchor for harness complexity debates.
  • MAFA (Oct 2025, AAAI) — Production multi-agent annotation system at JPMorgan Chase, clearing a 1M-utterance backlog and saving 5,000+ annotation hours annually; concrete enterprise evidence that configurable multi-agent harnesses scale.
  • Rethinking Agentic Workflows (Oct 2025, COLM 2025) — Empirically validates that Divide-and-Conquer with 3-shot demonstrations is the most reliable agentic strategy for Text2SQL; base model choice matters at least as much as workflow design.
  • Workflows vs Agents for Code Translation (Dec 2025, NeurIPS 2025) — Direct comparison showing MCP-based agentic flows beat rigid expert workflows at 8B and 30B scales; how tools are invoked outweighs which tools are available.
  • PLAGUE (Oct 2025, ICLR 2026) — Plug-and-play meta-framework decomposing jailbreak attacks into swappable Planner/Primer/Finisher modules with a lifelong-learning memory bank; 81.4% SRE on o3, 67.3% on Claude Opus 4.1.

➡️ Read the full trend analysis