SCALE: Towards Collaborative Content Analysis in Social Science with Large Language Model Agents and Human Intervention

🕒 Published (v1): 2025-02-16 00:19 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SCALE is a multi-agent LLM framework that simulates the full social-science content analysis pipeline — independent text coding, structured inter-agent discussion, and iterative codebook evolution — using \(N\) LLM agents each embodying distinct social-scientist personas. Human experts can intervene at configurable scopes (targeted vs. extensive) and roles (collaborative vs. directive). On five real-world datasets the system reaches 0.701 average accuracy automatically and 0.872 with expert intervention.

Problem

Social-science content analysis requires multiple rounds of manual annotation, expert discussion, and codebook refinement — a process that is labor-intensive, subjective, and hard to scale as digital corpora grow. Existing LLM agent systems lack dynamic rule evolution, structured inter-agent deliberation, and grounding in social-science methodology.

Method

SCALE operates in a repeating four-phase loop:

1. Coder Simulation — configure \(N\) LLM agents \(\mathcal{A} = \{a_i\}_{i=1}^N\) with distinct real-world social-scientist personas \(\mathcal{P} = \{p_i\}_{i=1}^N\) and an initial codebook \(\mathcal{C}\) (either expert-seeded or empty).
2. Bot Annotation — each agent independently annotates a batch of \(B\) text entries into discrete categories following codebook rules.
3. Agent Discussion — when any agent's coding diverges from consensus, agents engage in up to \(K\) structured discussion rounds, updating annotations and explanations until unanimous agreement or round limit is reached.
4. Codebook Evolution — agents collaboratively propose, debate, and finalize an updated codebook (adding examples, modifying or removing rules), which feeds the next task cycle.

Human intervention is layered onto steps 3–4 via four combinable modes: targeted (discussion only) vs. extensive (discussion + codebook evaluation), crossed with collaborative (agents may accept or reject expert feedback) vs. directive (agents must comply). Each mode is implemented through carefully engineered prompts.

Key Contributions

• First system to simulate rigorous quantitative content analysis (coding + discussion + codebook refinement) for social science via LLMs.
• Praxis-informed design co-developed with domain social scientists, faithfully mirroring the manual workflow.
• Configurable human intervention portal with four combinable modes derived from social influence and HCI theory.
• Dynamic codebook evolution loop enabling self-improving annotation rules without human authorship.
• Empirical validation across 7 tasks spanning multi-class and multi-label classification on real expert-annotated datasets.

Results

• Automatic mode (no intervention): average accuracy 0.701 across 7 tasks (GPT-4O backbone).
• Prompting ablation: TOT over vanilla: +6.51%; self-consistency: +2.31%; COT occasionally harmful on ambiguous tasks (e.g., CN-NP).
• Model gap: GPT-4O outperforms GPT-4O-mini by 10.89% on average.
• Discussion impact: inter-agent discussion raises average agreement rate by 41.1% and annotation accuracy by 15.4%; removing discussion entirely costs −14.3% accuracy.
• Human intervention: average accuracy rises to 0.872 (+12.6% over automatic).
• Directive > collaborative: +13.1% accuracy gain; extensive > targeted: +15% gain.
• Task variability: CES (domain-specific, subjective) gains 20% from intervention; PIS (sentiment) gains 5.4%.
• Discussion limits: in FWPE and PIS, agreement gain from discussion is <3% when agents are entrenched in conflicting stances.

Limitations

• Agent entrenchment: LLM agents may not converge even after \(K\) discussion rounds, leaving unresolved disagreements that bound accuracy.
• Evaluated only on GPT-4O and GPT-4O-mini; no cross-provider comparison (Gemini, Claude, open-source models).
• Directive/extensive intervention is more accurate but increases cost and reduces automation throughput.
• Datasets are small (40–200 text entries); scalability to large corpora is claimed but not benchmarked.
• Persona diversity requires tuning: very low or very high inter-agent persona distance reduces discussion productivity; optimal level is task-dependent and not automatically determined.

Relevance to Harnesses / Meta-Harnesses

SCALE is effectively a domain-specific meta-harness: it orchestrates a structured pipeline of heterogeneous agent roles (coders, discussants, codebook editors) in an iterative loop where each phase's output feeds the next cycle's configuration — a canonical harness control pattern. The codebook evolution step is a self-improving rule-refinement mechanism directly analogous to dynamic prompt/schema updates in general-purpose meta-harnesses. The tiered human intervention design (scope × role) provides a reusable architectural template for inserting authoritative control signals at configurable points in any multi-agent pipeline without restructuring the underlying agent logic. For practitioners building annotation or quality-control harnesses, SCALE's empirical analysis of batch size \(B\), discussion rounds \(K\), and agent count \(N\) offers concrete guidance on loop hyperparameters.