Skip to content

PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework

🕒 Published (v1): 2025-11-04 21:48 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PublicAgent is a sequential multi-agent pipeline that decomposes end-to-end open data analysis (query → dataset discovery → statistical analysis → report) into four specialized agents coordinated by an orchestrator. The authors run systematic ablations across five LLMs and 50 queries to extract five empirical design principles for multi-agent LLM system construction. The core claim is that specialization provides value orthogonal to model strength.

Problem

Single-LLM end-to-end analytical workflows over heterogeneous open data repositories suffer from three compounding failure modes: (1) attention dilution across growing contexts, (2) task interference when distinct reasoning patterns (semantic search vs. statistical computation) co-occupy a single context, and (3) unchecked error propagation across pipeline stages. No prior system integrates query clarification, heterogeneous dataset discovery, validated code generation, and report synthesis into a single framework for unstructured public data.

Method

PublicAgent defines an orchestration function \(f_o\) that coordinates four specialized agents sequentially: - Intent Clarifying Agent (\(f_q\)): detects up to three ambiguities in \(Q_u\) and iteratively resolves them into an enhanced query \(Q_e\). - Data Discovery Agent (\(f_d\)): performs semantic search over repositories (e.g., data.gov), selects dataset \(D_i\), and synthesizes structured metadata \(M\) (schema, statistics, sample rows, provenance). - Data Analysis Agent (\(f_x\)): decomposes \(Q_e\) into discrete experiments, generates isolated Python programs on a preloaded dataframe, executes them, and validates outputs for anomalies (e.g., zero-value filtering errors). - Report Generation Agent (\(f_g\)): synthesizes \(\{Q_u, Q_e, D_i, E\}\) into a natural-language report.

The orchestrator validates each stage output and initiates targeted retries or upstream re-evaluation on failure. Report quality is scored as \(Q(R) = \frac{1}{4}(F(R) + C(R) + V(R) + H(R))\) on a 1–10 rubric across factual consistency, completeness, relevance, and coherence, judged by an LLM evaluator.

Key Contributions

  • Five empirically derived design principles for multi-agent LLM systems (see Results).
  • PublicAgent framework: orchestrator + four specialized agents over public data repositories with natural language interfaces at every stage.
  • Ablation study isolating individual agent contributions across five LLMs and 50 queries.
  • Taxonomy of agent types: universal (discovery, analysis—consistently effective) vs. conditional (report, intent—effectiveness varies by model).

Results

  • Specialization vs. monolith: full pipeline achieves 97.5% agent win rate over single-model baseline even for the strongest tested model, confirming benefit independent of model scale.
  • Universal vs. conditional agents: universal agents (discovery, analysis) show low cross-model variance (std dev 12.4%); conditional agents (report, intent) show high variance (std dev 20.5%).
  • Failure mode severity: removing discovery or analysis agents causes catastrophic failure (243–280 failure instances); removing report or intent agents causes quality degradation only.
  • Complexity invariance: win rates remain stable across task complexity tiers (86–92% for analysis, 84–94% for discovery), suggesting the benefit is workflow management, not reasoning augmentation.
  • Model sensitivity: analysis agent effectiveness ranges from 42–96% across the five models, requiring model-aware architecture decisions.

No absolute quality score numbers (e.g., mean Q(R)) are reported for the full pipeline in the provided text.

Limitations

  • Evaluation is limited to 50 queries across five models; generalizability to broader query distributions or additional repositories is unverified.
  • The win-rate metric relies on LLM-as-judge rubric scoring, which introduces its own bias and may not align with end-user satisfaction.
  • The pipeline is strictly sequential; parallel or dynamic routing among agents is not explored.
  • Dataset coverage is anchored to data.gov; behavior on repositories with different metadata conventions is not assessed.
  • No latency or cost analysis comparing the multi-agent pipeline against single-model baselines.

Relevance to Harnesses / Meta-Harnesses

PublicAgent is a concrete instantiation of a pipeline harness: a fixed orchestration layer that sequences specialized sub-agents, validates inter-stage outputs, and performs targeted retries—exactly the coordination logic that meta-harnesses must implement. The five design principles it derives (universal vs. conditional agent categories, failure mode taxonomy, model-aware architecture) provide empirical grounding for harness designers deciding how to decompose tasks and where to place validation gates. The distinction between catastrophic-failure agents (discovery, analysis) and quality-degradation agents (report, intent) directly informs which harness stages require hard-stop error handling vs. best-effort fallback.