Skip to content

Multi-Paradigm Agent Interaction in Practice:A Systematic Analysis of Generator-Evaluator, ReAct Loop,and Adversarial Evaluation in the buddyMe Framework

🕒 Published (v1): 2026-05-16 05:35 UTC · Source: Arxiv · link

Why this paper was selected

Systematic analysis of three agent interaction paradigms within unified production framework

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

buddyMe is an open-source Python multi-agent framework that unifies Generator-Evaluator pre-execution review (the "Sprint Contract"), ReAct-style tool-use loops, and a three-layer memory system under a single orchestrator. Its central novelty is an Evaluator-Defender adversarial discussion mechanism in which an independent EvalAgent scores outputs across six dimensions—with real code execution as evidence—while a Defender agent challenges the verdict until consensus is reached. Empirical evaluation on five task scenarios yields a weighted quality score of 0.82/1.00 on a real HTML-generation task.

Problem

Existing multi-agent frameworks (CrewAI, AutoGen, LangGraph) treat evaluation as an external concern, decoupled from the execution pipeline. This creates two gaps: (1) agents execute tasks without systematic self-assessment tied to the original requirements, and (2) evaluation results are never fed back to correct execution-time decisions. Additionally, single-pass LLM evaluation is susceptible to context pollution (shared client bias of +0.12) and the categorical-vs-numerical inconsistency problem (score=0.8 with approved=False).

Method

buddyMe follows a five-phase pipeline: requirement pre-review → task decomposition with checkpoint replanning → subtask execution via ReAct loops → real execution verification → adversarial evaluation discussion.

Three-client isolation. Three independent LLM clients are instantiated—main executor (_client), sub-client for the Generator/Defender role, and eval-client for the independent Evaluator—ensuring zero context sharing between executor and evaluator.

Sprint Contract (pre-execution review). Before execution, a Generator produces a structured requirement document containing deliverables, quantifiable success criteria, risk areas, and a tagged step plan ([SEARCH]/[CREATE]/[EDIT]/[VERIFY]). An independent Evaluator audits completeness, verifiability, plan coverage, and risk identification, outputting a numerical score. Approval condition: score >= 0.8 (not a boolean gate). Max 3 review rounds; the latest Generator plan is forcibly adopted if consensus is not reached.

ReAct execution. TaskPipeline constructs per-subtask system prompts embedding the Sprint Contract, prior subtask results, and tool schemas. Eight tools are available; tool access is phase-restricted (verification steps get only read/edit tools). Per-subtask step bound: max_steps=11. An embedding-matched Skill library enables automatic capability discovery via invoke_skill.

Checkpoint-based dynamic replanning. At designated checkpoints the remaining plan is re-submitted to the LLM alongside completed results; the plan is rebuilt if needed.

Evaluator-Defender adversarial discussion. Post-execution, the EvalAgent performs six-dimensional scoring with real execution verification (Python via asyncio.create_subprocess_shell() with 30 s timeout; HTML structural validation; shell scripts). The weighted score \(S = \sum_{i} w_i s_i\) is computed server-side, not by the LLM. The Defender then issues factual rebuttals (≤200 chars each); the Evaluator revises scores if warranted. Consensus: Defender acceptance or \(|S_{t} - S_{t-1}| \leq 0.05\). Ceiling: 20 rounds.

Three-layer memory. Working memory (MessageHistory, in-session), episodic memory (SubtasksManager, inter-subtask via subtask_results.json), and long-term memory (MemoryStore with score-decay-consolidate lifecycle across sessions), all injected into the system prompt at invocation time.

Key Contributions

  • Sprint Contract mechanism: structured pre-execution requirement document with quantifiable success criteria serving as both planning baseline and evaluation reference; catches omissions in 20% of complex tasks.
  • Six-dimensional evaluation schema (Task Completion 0.30, Truthfulness 0.20, Tool Accuracy 0.15, Output Quality 0.15, Error Recovery 0.10, Efficiency 0.10) with server-side score computation and five-layer JSON extraction robustness.
  • Evaluator-Defender adversarial discussion: independent adversarial scrutiny converging in 2–3 rounds for 95% of cases, eliminating the +0.12 single-client evaluation bias.
  • Three-client isolation architecture: empirically validated to eliminate systematic evaluator inflation.
  • Embedding-matched Skill system: automatic tool/capability discovery without manual specification.
  • Engineering lessons: numerical thresholds > boolean gates; context isolation prevents bias; data collection ≠ data consumption.

Results

  • Weighted quality score 0.82/1.00 (Grade B) on a real-world Beijing travel guide HTML generation task (26 min, 26 tool calls, 1,041-line output).
  • Dimension breakdown: Task Completion 0.9, Truthfulness 0.9, Output Quality 0.9, Error Recovery 0.8, Tool Accuracy 0.7, Efficiency 0.5.
  • Sprint Contract pre-review: 80% of tasks pass on round 1; Scenario 3 (blog) improved from 0.70 → 0.88 after identifying a missing visualization step; Scenario 5 (Vue dashboard) required forced adoption after 3 rounds.
  • Adversarial discussion: avg. 2–3 rounds; Defender acceptance rate ~70%; score convergence rate ~25%; max-round ceiling hit ~5%; score adjustments typically within ±0.05.
  • Three-client isolation eliminated a measured +0.12 evaluation bias versus shared-client baseline.

Limitations

  • Efficiency dimension consistently underperforms (0.5 in case study) due to absent file-read cache and search deduplication; same file read 8–9Ă— across subtasks.
  • Real execution verification limited to Python, HTML, and shell; projects requiring external services (APIs, databases) cannot be offline-verified.
  • Fixed evaluation weights (\(w_{\text{completion}}=0.30\), etc.) are not calibrated per task type; no data-driven weight optimization.
  • Single-user, synchronous architecture; multi-user concurrent scaling requires significant re-architecture.
  • Evaluation is performed on only five task scenarios with a single model pair (DeepSeek-v4-pro + GLM-5.1); generalizability is not established.
  • No ablation study isolating the contribution of individual mechanisms.

Relevance to Harnesses / Meta-Harnesses

buddyMe is a concrete instantiation of a meta-harness pattern: a fixed outer control loop (Sprint Contract → ReAct execution → checkpoint replan → adversarial evaluation) that governs multiple inner agents and enforces quality contracts at stage boundaries. The Sprint Contract mechanism directly parallels the "pre-flight contract" design common in meta-harnesses—where the harness validates intent before spawning workers—and the Evaluator-Defender loop mirrors the adversarial-verify pattern used in multi-agent review harnesses to prevent single-point evaluator failure. The three-client isolation finding (quantifying +0.12 bias from shared context) is an empirical data point directly relevant to harness designers deciding whether to reuse or isolate LLM clients across pipeline stages. The paper also contributes two actionable harness-engineering heuristics: prefer numerical gate conditions over booleans for automated LLM decisions, and explicitly wire data sources to consumers rather than assuming implicit propagation.