Skip to content

MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation

🕒 Published (v1): 2025-04-17 01:25 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MetaSynth is a meta-prompting framework in which a central orchestrator LLM (the meta-LM) dynamically composes and dispatches instructions to specialist "expert" agents to generate diverse synthetic text, maintaining a stateful memory of all prior outputs to enforce distinctness via Conditional Instance Generation. Using only 25M tokens of MetaSynth-generated data, Mistral-7B-v0.3 is adapted to Finance (+4.08% avg) and Biomedicine (+13.75% avg) without any real data, outperforming template-prompted baselines that actually degrade the model.

Problem

LLM-generated synthetic data suffers from low diversity due to two root causes: (a) static template prompts with variation limited to slot-filling, causing repetitive syntactic structures and lexical patterns, and (b) uninformative seed initialization. This low diversity limits downstream utility for domain adaptation via continual pre-training (CPT). Prior work (Self-Prompting, Attrprompt, CLINGEN, Explore-Instruct) all exhibit this failure mode. The paper also targets the emerging challenge of public human-text scarcity projected to be exhausted by 2028.

Method

MetaSynth combines two ideas: Meta-Prompting and Conditional Instance Generation.

Meta-Prompting: A central meta-LM (Claude 3 Sonnet) acts as orchestrator in a centralized multi-agent system. It dynamically selects agent types and composes their instructions based on its full history. Agents see only partial context ("fresh eyes"), preventing the meta-LM's errors from propagating. Required agents are pinned by meta-prompt constraints to prevent degenerative loops; beyond those, the meta-LM is free to invoke any agent type.

Conditional Instance Generation (Algorithm 1): At iteration \(i\), a new synthetic instance \(I_i\) is selected as: $\(I_i = \arg\max_{I} \; p\!\left(I \;\middle|\; \{I_0,\ldots,I_{i-1}\}, S_i;\, \theta\right) \times \mathbb{E}\!\left[\operatorname{div}\!\left(\{I_0,\ldots,I_{i-1}\}, I\right)\right]\)$ subject to \(I\) conforming to the expanded seed set \(S_i = \operatorname{ExpandSeeds}(S_{i-1}, \{I_0,\ldots,I_i\})\). The expanding seed set (keywords or topic-labeled documents) and the growing instance classification table in the meta-LM's history together enforce distinctness.

Workflow agents include: Seed Keyword Extraction Expert, Domain Expert (instantiated with varied personas), Summarizer Expert, Content Analyst Expert, Seed Keyword Expansion Expert, and Writing/Linguistic Expert. Summaries stand in for full documents in the growing history, mitigating context window limits.

Topic-aware seed selection: A dynamic k-NN over embedding space (jina-embeddings-v2-base-en) retrieves new seeds after every \(M\) synthetic documents, selecting neighbors whose LLM-assigned topic label differs from recently synthesized documents.

MetaSynth-Instruct extends the pipeline to instruction synthesis and evolution using predetermined agents (Document Transformation Expert, Persona Suggestion Expert, Complexity Expert, Question Editor Expert), producing instruction–response pairs for CPT and encoder fine-tuning.

Exit criteria: meta-LM emits <end> when the target count is reached; on error, the error is appended to history for retry; after \(N\) failures the iteration is discarded.

Key Contributions

  • MetaSynth meta-prompting harness for diverse synthetic document generation using Conditional Instance Generation with expanding seed sets and a stateful classification table.
  • MetaSynth-Instruct for instruction synthesis and open-ended evolution driven entirely by the meta-LM's own prompt composition, without human-written text.
  • Demonstrated that diverse synthetic data from MetaSynth obviates the need to mix real data: 25M pure synthetic tokens outperform 25M real tokens in both Finance and Biomedicine.
  • Finding that template-prompted data (even conditioned on prior outputs and varied ICL exemplars) degrades CPT performance relative to the base model; MetaSynth avoids model collapse.
  • A seven-metric diversity evaluation suite including Task2Vec diversity coefficient, Gzip compression ratio, N-gram diversity (1-GD, 4-GD), Remote Clique, Chamfer Distance, and a newly proposed Mean Inverse Frequency (MIF) score.

Results

  • Finance CPT (Mistral-7B-v0.3, 25M MetaSynth-Docs-Instructions-Responses tokens, no real data): +4.08% avg over base across 5 benchmarks (ConvFinQA, NER, FPB, Headline, FiQA-SA).
  • Biomedicine CPT (same setup): +13.75% avg over base across 5 benchmarks (PubMedQA, USMLE, MQP, RCT, ChemProt).
  • 1:1 real + MetaSynth docs outperforms 1:1 real + template-prompted docs by +3.08% (Finance) and +8.85% (Biomedicine).
  • Real + template-prompted CPT falls below the no-CPT base in Biomedicine (50.48% vs 52.94%).
  • Diversity (Finance, seed-document setting vs. template): compression ratio −14.12%, Task2Vec +13.45%, Chamfer Distance +54.18%, 1-GD +96.97%, 4-GD +46.01%.
  • MetaSynth diversity approaches Common Crawl on Task2Vec, Chamfer, and 1-GD/4-GD metrics.
  • BERT fine-tuned on MetaSynth-Instruct outperforms BERT fine-tuned on template-prompted data on FiQA-SA, FPB, and Headlines (Finance encoder tasks).
  • General benchmark degradation after MetaSynth-only CPT is minimal (e.g., MMLU drops <3pp).

Limitations

  • Meta-prompting orchestration is substantially more expensive per token than template prompting; cost at scale (beyond 25M tokens) is not quantified.
  • Degeneration (repetitive agent exchange loops) can still occur despite required-agent constraints, due to noisy message passing.
  • Failed iterations are silently discarded after \(N\) retries; impact on final corpus coverage and diversity is not analyzed.
  • Only Claude 3 Sonnet used as meta-LM; generalization to weaker or open-weight orchestrators is untested.
  • Evaluation limited to two domains (Finance, Biomedicine) and one base model (Mistral-7B-v0.3).
  • No human evaluation of diversity; all diversity metrics are automated proxies.

Relevance to Harnesses / Meta-Harnesses

MetaSynth is a textbook meta-harness: a stateful orchestrator LM that dynamically assembles a pipeline of specialist agents, routes subtasks, aggregates results, and enforces structural constraints (required agent invocations, error-retry loops, exit criteria) to prevent runaway execution — exactly the control-flow responsibilities a meta-harness must handle. The Conditional Instance Generation algorithm formalizes the harness's inner loop with an explicit diversity objective, demonstrating that harness-level memory (the instance classification table) is architecturally necessary to prevent output degeneration. The paper's central finding — that meta-harness orchestration yields qualitatively superior outputs over flat template prompting, sufficient to reverse model collapse — provides empirical grounding for investing in harness complexity over simpler prompt engineering.