Skip to content

Automating the Design of Embodied AgentArchitectures

🕒 Published (v1): 2026-06-29 10:45 UTC · Source: Arxiv · link

Why this paper was selected

Automated embodied agent architecture design; changes how we build agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces AGENT CANVAS, a typed-graph runtime for representing embodied agents as editable node-and-wire programs, and KDLoop, a knowledge-distill loop optimizer that cycles through THINK–CRITIC–EXPERIMENT–DISTILL phases to automate embodied agent architecture design. Three AAS variants (ADAS, AFlow, KDLoop) are evaluated across four published embodied executors (MapGPT, SmartWay, ExploreEQA, VoxPoser) spanning VLN, EQA, and manipulation, producing a 3×4 outcome matrix. Results confirm that architecture-level search yields deployable success-rate gains in several cells, while also exposing three constraints specific to embodied AAS: evaluation noise, local edit basins, and partial episode-level credit assignment.

Problem

Agent Architecture Search (AAS) automates LLM-agent workflow design in text-only domains, but has not been systematically applied to perceptual embodied agents. Embodied agents interact with stateful simulators, produce noisy multi-episode evaluations, involve long execution traces (observations, actions, tool calls, planner outputs), and lack a small palette of composable task-agnostic primitives analogous to chain-of-thought or self-refinement. Existing embodied systems still rely on hand-designed module choices per benchmark, even as the design space grows with the proliferation of foundation models and embodied tools.

Method

Executor substrate — AgentCanvas: Each embodied agent is represented as a typed JSON node-and-wire graph backed by Python node modules. Nodes expose typed input–output ports; structural edits (add edges, swap nodes, replace subgraphs) are applied as structured patches with type-compatibility checks before rollouts. Every rollout is instrumented: node firings record inputs/outputs, and nodes may write internal breadcrumbs (planner decisions, prompts, tool calls, self-reports). Simulator-dependent nodes are replicated across workers; shared foundation-model nodes remain singleton batching services.

Optimizer substrate — Coding-agent harness: A shared outer orchestrator runs proposer → implementer → evaluator. The implementer uses an off-the-shelf coding-agent session (Claude Code) with file, shell, and tool access to apply multi-file patches to the graph workspace. Only the proposer and persistent memory differ across AAS variants.

KDLoop (novel optimizer): Four-phase cycle per iteration: 1. THINK — reads typed distilled memory, proposes up to three experiments tagged by intervention axis (prompt content, topology, observation pipeline, state-memory, model configuration, action-space). 2. CRITIC — checks proposals against prior failed patches and constraints. 3. EXPERIMENT — applies and evaluates the selected edit via simulator rollouts. 4. DISTILL — writes confirmed findings, refuted edits, open conjectures, and coverage maps to typed persistent files. A triggered REFLECT phase audits stall patterns and can mark parts of the search space as exhausted.

ADAS and AFlow ports: ADAS preserves Reflexion-style proposals and a flat append-only archive; each LLM proposal call is an independent tool-augmented sub-agent. AFlow preserves score-softmax parent selection and anti-replay memory, but searches free-form structural changes over typed Executor graphs instead of a curated operator library.

Search setting: Method-seeded from four published agents (MapGPT, SmartWay, ExploreEQA, VoxPoser); backbone fixed to GPT-5-MINI (except VoxPoser). Orchestrator is Claude Opus 4.7 with a 1M-token context window. Fitness \(f(c)\) is success rate (SR) over multi-episode simulator rollouts. Best candidate returned: \(\arg\max_{t \le B} f(c_t)\).

Key Contributions

  • AgentCanvas: first typed-graph runtime coupling editable agent graphs to stateful simulator execution and episode-level logging.
  • KDLoop: a knowledge-distill loop with axis-tagged memory designed to reduce blind local resampling and improve coverage of the graph edit space.
  • Shared harness: ADAS and AFlow ported to the same coding-agent substrate, isolating search-policy differences from tool/log access differences.
  • 3×4 outcome matrix: first systematic evaluation of AAS-style search on perceptual embodied agents validated via simulator rollouts (not text-only benchmarks).
  • Characterization of three embodied-AAS failure modes: rollout noise masking edit effects (\(\sigma\) up to 5.9 SR), search trapped in local edit basins, and episode-level credit assignment only partially emerging.
  • Leak detection methodology: SmartWay–AFlow excluded after identifying evaluator-side information access; VoxPoser–KDLoop surfaces a substrate-level logging fault before SR mutation.

Results

  • MapGPT (VLN, R2R/MP3D): baseline 46.9±3.1%; ADAS +2.2 pp, AFlow +7.6 pp (54.5±3.1%), KDLoop +7.1 pp (54.0±2.3%); AFlow and KDLoop reach similar SR via different edit axes (stopping rules, heading-band action gates, elevation deadband).
  • ExploreEQA (active EQA, HM-EQA): baseline 43.0±1.7%; ADAS no improvement (no graph-level lever found); AFlow +4.7 pp (47.7±2.1%); KDLoop +3.0 pp (46.0±1.0%); gains driven by answer-formatting and sentinel-handling changes.
  • SmartWay (continuous-action VLN, VLN-CE): baseline 29.7±2.1%; ADAS +4.0 pp (33.7±2.5%); AFlow +9.0 pp excluded as leak-bearing (progress-tracker wired to ground-truth evaluator); KDLoop +1.3 pp (31.0±4.6%).
  • VoxPoser (zero-shot VLA, LIBERO): baseline 9.0±0.0%; ADAS +3.7 pp, AFlow +3.9 pp; KDLoop detects injected logging fault (dynamically dispatched LMP sub-calls missing from self-report channel), diagnoses fault as below the editable graph layer, makes no SR mutation.
  • No optimizer dominates: ADAS concentrates on topology/model-config; AFlow on observation pipeline; KDLoop spreads edits across observation, prompt, control, and state-memory axes.
  • Noise characterization: SmartWay-AFlow reruns at 43/32/41% (38.7±5.9%); KDLoop-SmartWay reruns at 32/35/26% (31.0±4.6%); lower-variance gains (MapGPT, ExploreEQA) remain separated from baselines after three post-selection reruns.

Limitations

  • Evaluation noise: single-pass fitness during search can overestimate a candidate; rollout variance (σ up to 5.9 SR) means selection ≠ verification; three-pass rerun floor is used but adds evaluation cost.
  • Local edit basins: ADAS and AFlow lack axis-level memory and can revisit the same neighborhood repeatedly (ADAS rediscovers SR=0.4769 on MapGPT across six iterations); both run to the iteration cap in all cells.
  • Partial credit assignment: detailed episode logs are available to all optimizers but not systematically used; scalar SR misses the evaluator-side leak in SmartWay–AFlow until post-hoc inspection.
  • KDLoop coverage–exploitation trade-off: typed coverage tracking can redirect search away from productive basins before full exploitation; KDLoop is not uniformly superior to ADAS/AFlow.
  • Method-seeded only: search explores graph-level modifications around published methods; no evidence that AAS can assemble an embodied agent from scratch.
  • Task-specific architectures: navigation, QA, and manipulation systems contain task-specific perception and action interfaces; search does not generalize across task families.
  • No no-architecture baseline hardening: the paper does not compare against tuned single-agent baselines (per the concern raised for text-domain AAS), so it is unclear whether gains come from architecture vs. incidental prompt tuning.

Relevance to Vision-Language Models

The four seed executors (MapGPT, SmartWay, ExploreEQA, VoxPoser) all integrate frozen VLMs or LLMs as core reasoning modules, making this paper directly relevant to researchers studying how VLMs are composed into agentic pipelines for grounded tasks. The AAS framework treats the graph-level architecture wrapping a frozen VLM as the object of optimization, offering a principled alternative to manual module engineering in VLN and EQA systems. The finding that graph-level edits (action gates, answer-format constraints, stop rules) can yield +7 pp gains without fine-tuning the underlying VLM suggests that architecture design is an underexplored leverage point for VLM-based embodied systems. The leak detection and noise characterization results also serve as methodological warnings for VLM-agent benchmarking, where evaluation noise and data contamination can silently invalidate comparative claims.