Skip to content

AutoData: A Multi-Agent System for Open Web Data Collection

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoData is a multi-agent system (NeurIPS 2025) that converts a single natural-language instruction into a collected dataset by orchestrating eight specialized agents across research and development squads. Its core novelty is OHCache, an oriented hypergraph cache system that replaces broadcast communication with targeted message routing, reducing token cost while improving accuracy. It also introduces Instruct2DS, the first benchmark for evaluating live, open-web data collection.

Problem

Existing web data collection approaches are either brittle wrapper-based crawlers that require per-site manual engineering, or LLM-based scrapers that incur high computational/financial cost due to broadcast-style communication and artifact-polluted message channels. No benchmark existed for evaluating agents on live, dynamic web sources (prior benchmarks used static, archived pages).

Method

AutoData coordinates eight agents via a central manager (MGR) in two squads: - Research squad (plan, web, tool, blueprint agents): decomposes the instruction, browses live web sources, and produces a development blueprint. - Development squad (engineering, test, validation agents): converts the blueprint into executable code, debugs it, runs it, and validates the output.

Communication is managed by OHCache, comprising three components: 1. Oriented Message Hypergraph: models each message as a directed hyperedge from one source agent to a specific subset of target agents, eliminating broadcast overhead. Each agent's decision context includes only messages addressed to it. 2. Oriented Hyperedge Formatter: transforms free-form agent outputs into structured, machine-interpretable messages with agent-specific schemas before hypergraph insertion. 3. Local Cache System: a special node in the hypergraph that stores bulky artifacts (e.g., HTML files) locally, broadcasting only a cache identifier, preventing artifact-embedded message pollution.

Agents follow the ReAct paradigm (reason before acting). The system supports both web crawling and REST API calls.

Key Contributions

  • Eight-agent MAS with a two-squad architecture (research + development) coordinated by a manager agent, requiring only a natural-language instruction as input.
  • OHCache: oriented hypergraph communication replacing broadcast messaging with targeted hyperedge routing, plus structured formatting and artifact caching.
  • Instruct2DS: first open-web data collection benchmark spanning academic (conference papers), finance (stock), and sports domains, with symbolic instruction templates and ground-truth datasets.
  • Comprehensive evaluation across Instruct2DS, SWDE, Extended SWDE, and HumanEval.

Results

  • Instruct2DS (Academic F1): AutoData 91.85 vs. Manus 69.27, OpenManus 67.39, AutoAgent 65.79, AutoAgents 63.42; also outperforms Cline (83.50) and Cursor (84.37) while being faster.
  • Instruct2DS cost: AutoData $0.57 vs. Manus $2.49, AutoAgent $1.28, OpenManus $1.08.
  • Instruct2DS time-to-execution: AutoData 5.58 min vs. Manus 15.37 min.
  • SWDE F1: AutoData 89.25 vs. Manus 89.22, AutoScraper 88.69, Reflexion 82.40.
  • Extended SWDE F1: AutoData 77.44 vs. Manus 75.64, AutoScraper 76.21.
  • HumanEval Pass@1: AutoData+GPT-4 86.9 (vs. MetaGPT 85.9, GPT-4 67.0); AutoData+GPT-4o 92.5 (vs. GPT-4o 90.2).
  • Picture book case study: AutoData Acc 89.58 / Comp 98.13 / Cost $0.91 vs. Manus 63.93 / 79.76 / $1.86.
  • Survey BibTeX extraction: AutoData F1 91.16 / Cost $1.40 vs. Manus 74.70 / $2.55.
  • Ablation confirms all three OHCache components contribute; removing entire OHCache degrades performance and raises cost most sharply.

Limitations

  • Evaluation is constrained to three domains (academic, finance, sports); generalization to arbitrary web domains is assumed but not systematically tested.
  • HumanEval performance, while competitive, shows that the system is not specifically optimized for code generation and may lag behind dedicated coding agents on harder tasks.
  • The hypergraph topology (which agents receive which messages) is hand-designed per agent role; dynamic topology learning is not explored.
  • Cost and latency figures depend on the underlying LLM backbone (GPT-4/4o), so results may not transfer to open-weight models.
  • The paper does not evaluate robustness to website layout changes or anti-scraping measures over time.

Relevance to Agentic AI / LLM Agents

AutoData is a concrete instantiation of task-decomposed multi-agent orchestration applied to a high-value real-world task (dataset curation), directly relevant to researchers studying how agents coordinate on long-horizon, tool-heavy workflows. OHCache's oriented hypergraph addresses a fundamental scalability problem in MAS—broadcast communication causing token bloat and hallucination—offering a principled, structured alternative to naive full-history sharing. The two-squad (research → development) division of labor mirrors patterns seen in software-engineering agents (ChatDev, MetaGPT) but specializes them for data acquisition pipelines. Instruct2DS fills a benchmark gap for evaluating agents on live web interaction, which is increasingly central to agentic system evaluation.