Skip to content

Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning

🕒 Published (v1): 2025-06-30 16:02 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Auto-TA is a fully automated multi-agent LLM pipeline for end-to-end thematic analysis (TA) of unstructured clinical narratives, eliminating human-in-the-loop transcript review. It uses role-conditioned coder agents, theme-generation agents, and an autonomous feedback agent with iterative refinement, optionally extended by RLHF via PPO. Applied to CHD focus-group transcripts, it completes analysis in under 10 minutes per 10k-word document.

Problem

Manual TA of clinical narratives requires 40–60 hours per 10–15 interviews and limits scalability. Prior LLM-assisted approaches retain human-in-the-loop review of full transcripts, which defeats the scalability benefit. No prior fully automated system performs the complete TA pipeline—coding, theme generation, evaluation, and iterative refinement—without human intervention.

Method

Auto-TA operates in three interacting agent layers:

  1. Generation Agents: \(k=4\) role-conditioned GPT-4o coder agents (Physician, Surgeon, Researcher, Layperson), each receiving the full transcript (or contiguous chunks \(x \in \mathbb{R}^{\leq 1500}\) words), independently emit code sets \(C_r = \{c_{r,1}, \ldots, c_{r,n_r}\}\). A second tier of theme-generation agents cluster the merged codes into preliminary themes \(\Theta^{(0)}\).

  2. Feedback Agent: An autonomous critic scores themes on Credibility/Confirmability (\(C\)), Dependability (\(D\)), and Transferability (\(T\)), then proposes edits (ADD, SPLIT, COMBINE, DELETE). Heuristic rules apply edits (e.g., \(C < 0.7 \Rightarrow \text{ADD}\)) until \(\|s^{(t+1)} - s^{(t)}\|_1 < 0.05\) or \(t_{\max} = 5\).

  3. RLHF (optional): Human raters assign binary rewards per theme set; a reward model \(R_\phi\) is trained via MSE loss; the theme-generation policy is optimized with PPO: $\(\theta \leftarrow \theta + \alpha \nabla_\theta \mathbb{E}_{\pi_\theta}\left[R_\phi(\Theta^{(t)}) - \beta\,\text{KL}(\pi_\theta \| \pi_{\text{SFT}})\right]\)$ The feedback agent's scores are concatenated to the reward model input for reward shaping.

Key Contributions

  • End-to-end automated TA pipeline requiring no manual coding or full transcript review.
  • Role-conditioned multi-agent coder ensemble with 4 domain identities; identity conditioning improves credibility by +11.54–+16.28 points over baseline.
  • Autonomous feedback agent with heuristic edit rules and convergence-based stopping criterion.
  • Optional RLHF path (PPO) for adapter fine-tuning to researcher preferences without repeat transcript reads.
  • Novel evaluation framework mapping TA trustworthiness criteria (Lincoln & Guba) to computable metrics (\(C\), \(D\), \(T\)) alongside cosine, Levenshtein, and BLEU alignment to human themes.

Results

  • Credibility (C): Baseline \(82.13 \pm 18.96\); best (Cardiac Surgeon) \(98.41 \pm 4.76\) (+16.28).
  • Dependability (D): Approximately \(0.395\)–\(0.400\) across identities; identity augmentation has negligible or slightly negative effect (−0.005 to −0.041).
  • Transferability (T): Baseline \(0.308 \pm 0.018\); Medical Doctor \(0.334 \pm 0.007\) (+0.026), Psychologist \(0.325 \pm 0.015\) (+0.027), computed over all 36 train/validation splits.
  • Semantic alignment (Cbi): Baseline \(0.132 \pm 0.027\); identity-augmented agents show slightly lower cosine similarity (0.107–0.121), interpreted as identity agents expanding the thematic space beyond human annotations rather than regressing.
  • Throughput: Under 10 minutes per 10k-word transcript vs. 40–60 hours manually.
  • RLHF component reported as in-progress; no quantitative RLHF results included.

Limitations

  • RLHF integration is incomplete at time of publication; no empirical RLHF results reported.
  • Tested on a single CHD clinical corpus (9 transcripts, 42 participants); domain generalizability unvalidated.
  • Agents do not communicate with each other (no inter-agent debate or negotiation), limiting collaborative reasoning.
  • High sensitivity to prompt wording; minor prompt variations produce substantially different themes, raising reproducibility concerns.
  • Coarse 5-point integer scoring by LLM evaluators lacks resolution to detect micro-scale refinement improvements.
  • Human-theme alignment as a gold standard is questionable: multiple valid theme sets can exist for a single transcript.

Relevance to Harnesses / Meta-Harnesses

Auto-TA is a domain-specific multi-agent harness that orchestrates a fixed pipeline of specialist agents (coder → theme-generator → feedback critic → optional RL trainer) with convergence-based looping—a structural pattern directly analogous to meta-harnesses that wrap LLM calls in evaluation-and-refinement loops. The RLHF extension shows how a harness can incorporate a trainable reward model to steer downstream agents, a pattern relevant to meta-harnesses that self-optimize their own orchestration policies. The heuristic feedback loop (threshold-gated edit proposals iterated to convergence) is an explicit example of a harness controlling agent behavior programmatically rather than leaving control to a single model. For researchers building general meta-harnesses, the evaluation framework (mapping qualitative trustworthiness criteria to computable convergence signals) illustrates how domain-specific stopping criteria can be engineered into the harness control plane.