Skip to content

AutoRAS: Learning Robust Agentic Systems with Primitive Representations

🕒 Published (v1): 2026-06-19 14:05 UTC · Source: Arxiv · link

Why this paper was selected

Automated agentic system design with primitive representations; meta-harness self-improvement

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoRAS automates the design of robust multi-agent LLM systems by representing system designs as sequences of symbolic primitives (encoding both topology and behavior) and optimizing those sequences via GFlowNet trajectory balance training with execution-derived safety signals. Unlike prior automated design approaches that treat robustness as a post-hoc concern, AutoRAS bakes robustness into the generation-and-optimization loop. It achieves the best average vanilla accuracy (74.27%) and smallest adversarial performance drop (2.13%) across four benchmarks against 11 baselines.

Problem

Existing automated multi-agent system designers (AFlow, MaAS, G-Designer, GPTSwarm) optimize for task performance but treat robustness as an afterthought. Defenses are reactive and failure-mode-specific. Three fundamental difficulties make robust design hard: (i) Entanglement — topology and behavior cannot be co-designed independently; (ii) Unpredictability — adversarial and internal failure modes are open-ended and evolving; (iii) Equifinality — many structurally distinct systems achieve similar outcomes, creating non-unique optimization landscapes that destabilize gradient-based search.

Method

AutoRAS decomposes the design problem into three coupled stages:

  1. Primitive Sequence Generation. A system \(S = (V, E, B, G(\cdot), K)\) is represented as a sequence \(X = (x_1, \ldots, x_L)\) over a primitive alphabet \(\Phi = \Phi_{\text{struct}} \cup \Phi_{\text{act}}\). Structural primitives encode topology (sequential, parallel, fork/merge patterns via CTRL_SEQ, CTRL_PAR_k, CTRL_FORK_k); action primitives encode behaviors (AGT_COT, AGT_REFINE, AGT_ENS) and safeguards (SAFE_Filter, SAFE_Hygiene, SAFE_CrossAgent). A lightweight encoder–decoder (MiniLM embeddings + cross-attention) parameterizes the forward policy \(P_\theta(X \mid q, s)\), conditioned on query \(q\) and textual robustness signals \(s\).

  2. Robustness-Aware Execution. A stack-based compiler deterministically maps \(X\) to an executable workflow, enforcing legality via an RPN-style stack machine that emits a mask \(m_t \in \{-\infty, 0\}^{|\Phi|}\) at each step. Execution is logged into a structured trace \(T\); a monitor extracts correctness \(u\), cost \(c\), external robustness \(r_{\text{ext}} = (1-p)^j\), and internal robustness \(r_{\text{int}} = (1-p)^k\) (where \(j\), \(k\) count safeguard violations and internal failures), plus natural-language diagnostic signals \(s\).

  3. Flow-Based Optimization. The reward $\(R(X) = \alpha u + \rho\, r_{\text{ext}} + \eta\, r_{\text{int}} - \beta c > 0\)$ folds robustness directly into the design objective. GFlowNet trajectory balance (TB) trains the policy so \(P_\theta(X) \propto R(X)\), naturally handling equifinality by distributing probability mass across behaviorally equivalent designs. A textual gradient \(\nabla_t(\nu(X))\)—LLM-distilled rationales from execution traces—additionally refines the natural-language prompt components of action primitives without retraining backbone LLMs. The joint update is \(\nabla L = \nabla_\theta L_{\text{TB}} + \nabla_t(\nu(X))\).

Key Contributions

  • Agentic Primitive vocabulary: a unified symbolic representation over \(\Phi_{\text{struct}} \cup \Phi_{\text{act}}\) that co-encodes topology and behavior, reducing system design to legal sequence generation.
  • AutoRAS framework: closed loop of primitive generation → stack-based compilation → robustness-aware execution → flow-based optimization, with robustness as a first-class design objective.
  • GFlowNet TB + textual gradient: trajectory-level credit assignment handles equifinality; LLM-distilled textual gradients refine prompt-space components that numeric gradients cannot reach.
  • Stack-based legality enforcement: compiler-derived mask \(m_t\) prunes infeasible primitives online, ensuring every generated sequence is a valid executable system.
  • Comprehensive evaluation: 11 baselines, 4 benchmarks (MMLU, MSMARCO, MATH, ProgramDev), both vanilla and adversarial (brain/memory/tool/agent-to-agent attack) settings.

Results

  • Vanilla accuracy: AutoRAS achieves highest average of 74.27% across MMLU, MSMARCO, MATH, ProgramDev; best or runner-up on all four datasets individually.
  • Adversarial robustness: smallest average performance drop of 2.13% under attack, vs. substantially larger degradation for AFlow, MaAS, and other automated designers.
  • Cross-domain stability: lower variance across tasks than strong baselines (e.g., AFlow excels on ProgramDev but degrades on MATH).
  • Primitive sensitivity: performance degrades gracefully when primitives are removed and saturates when added, indicating robustness to primitive-set choices.
  • Comparison against production-grade multi-agent systems (Appendix F.4) shows AutoRAS remains competitive with strong robustness.

Limitations

  • Evaluation uses single-round interaction (\(K = 1\)); multi-round settings are not characterized.
  • Backbone LLMs are accessed via API with fixed temperature = 1; fine-tuning backbone models is outside scope.
  • Robustness measures \(r_{\text{ext}}\), \(r_{\text{int}}\) depend on LLM-based auditing (Cemri et al., 2025) for failure detection, inheriting its false-positive/negative rates.
  • Primitive vocabulary is manually designed; the method is shown to be stable across vocabulary variations but the initial vocabulary design remains a human choice.
  • Maximum sequence length \(L = 16\) caps system complexity; very large or deeply nested topologies cannot be represented.
  • Textual gradient refinement modifies prompt components but does not retrain backbone LLMs, limiting adaptation depth.

Relevance to Harnesses / Meta-Harnesses

AutoRAS is a meta-harness for multi-agent system construction: it does not execute tasks directly but instead generates, compiles, executes, and optimizes the harnesses (multi-agent workflows) that do. The primitive-sequence abstraction is directly analogous to the DSL/configuration layer that meta-harnesses use to specify pipeline topology, with the stack-based compiler playing the role of a harness instantiator. The closed-loop design (generate → compile → execute → monitor → optimize) mirrors the self-improving meta-harness pattern where the orchestrator uses execution feedback to refine its own configuration. The key advancement over prior automated design work is that robustness is embedded as a first-class objective in the meta-level optimization loop rather than applied post-hoc, offering a blueprint for meta-harnesses that need to maintain reliability under adversarial or failure-prone conditions.