Skip to content

CLAIM: An Intent-Driven Multi-Agent Framework for Analyzing Manipulation in Courtroom Dialogues

🕒 Published (v1): 2025-06-04 16:22 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLAIM is a two-stage multi-agent framework for detecting manipulation in courtroom dialogues, combining Intent-Driven Chain-of-Thought (CoT) prompting with a four-agent pipeline of fine-tuned Mistral-7B models. It is paired with LegalCon, a new 1,063-conversation annotated dataset covering manipulation detection, manipulator identification, and technique classification. CLAIM outperforms zero-shot and few-shot baselines across all three tasks on this dataset.

Problem

Existing NLP manipulation-detection work targets social media and mental health dialogues; no dataset or specialized framework exists for the legal domain. Long-form courtroom transcripts pose particular difficulty: manipulation is embedded in legal jargon, power dynamics, and rhetorical strategy, and SoTA models (e.g., ChatGPT-4o) fail to identify the primary manipulator or technique in context-dependent cases.

Method

Stage 1 — Intent-Driven CoT Prompting: Each speaker's communicative intent is extracted from the full dialogue using a structured CoT prompt, producing intent summaries that serve as intermediate reasoning scaffolds.

Stage 2 — Multi-Agent Pipeline: Four specialized agents, each a Mistral-7B fine-tuned on LegalCon via QLoRA (lr \(= 10^{-4}\)), operate sequentially: 1. Detector Agent — binary manipulation presence judgment. 2. Analyzer Agent — identifies primary manipulator and classifies techniques (11 categories). 3. Evidence Agent — extracts textual evidence supporting the labels. 4. Meta Agent — aggregates all agent outputs into final labels.

All agents receive both the raw dialogue and the Stage 1 intent summaries as input. Fine-tuning used a single RTX 3060 with QLoRA to stay within memory constraints.

Key Contributions

  • LegalCon dataset: 1,063 annotated courtroom conversations (663 manipulative / 400 non-manipulative) sourced from OYEZ Supreme Court transcripts and legal TV proceedings; average ~1,000 words per dialogue; annotated for three tasks with inter-annotator agreement \(\kappa = 0.68\) (detection), \(\kappa = 0.59\) (manipulator), Krippendorff's \(\alpha = 0.41\) (techniques).
  • 11 manipulation technique taxonomy developed with psychology expert consultation.
  • CLAIM framework: two-stage intent-grounded multi-agent system that separates intent inference, detection, evidence gathering, and aggregation into distinct specialized roles.
  • First computational framework targeting manipulation analysis specifically in courtroom discourse.

Results

Manipulation Detection (Table 3, Mistral-7B): - CLAIM: Accuracy 0.731, F1 0.727 - Few-shot baseline: Accuracy 0.667, F1 0.664 - Zero-shot baseline: Accuracy 0.609, F1 0.518

Primary Manipulator Identification (Table 4, Mistral-7B): - CLAIM: Accuracy 0.609, F1 0.602 - Few-shot baseline: Accuracy 0.481, F1 0.489 - Zero-shot baseline: Accuracy 0.340, F1 0.340

Manipulation Technique Classification (Table 5, Mistral-7B): - CLAIM: Jaccard \(J = 0.3618\), Accuracy 0.2564 - Few-shot baseline: \(J = 0.3145\), Accuracy 0.2179 - Zero-shot baseline: \(J = 0.1265\), Accuracy 0.0385 - Llama-3.1-8B baselines consistently underperform Mistral-7B equivalents across all tasks.

Limitations

  • Manipulation annotation is inherently subjective; moderate inter-annotator agreement (\(\kappa < 0.70\)) reflects this irreducible ambiguity.
  • LegalCon covers only US English courtroom settings with a limited set of case types; generalizability to other jurisdictions or languages is untested.
  • CLAIM was optimized for long-form courtroom dialogues; performance on short or informal dialogues is expected to degrade.
  • Computational constraints (single RTX 3060) limited batch size and training epochs, preventing experimentation with larger models.
  • Technique classification performance remains low (Jaccard ~0.36), indicating the multi-label taxonomy is difficult to resolve even with the full framework.

Relevance to Harnesses / Meta-Harnesses

CLAIM instantiates a canonical multi-agent harness pattern: a sequential pipeline where specialized sub-agents each own a distinct subtask (detect → analyze → retrieve evidence → aggregate), coordinated by a Meta Agent that synthesizes upstream outputs into a final verdict. The two-stage design — an intent-extraction pre-pass feeding downstream agents — is a concrete example of using intermediate reasoning artifacts as inter-agent context, directly relevant to harness architectures that chain CoT scaffolds across agent boundaries. For researchers building meta-harnesses, the explicit decomposition of a complex NLP judgment into four agent roles with defined input/output contracts, plus the use of domain-fine-tuned models per role, demonstrates both the productivity gains and the engineering overhead of role-specialized pipelines. The Meta Agent aggregation step is also a microcosm of the synthesis/judge-panel pattern common in review harnesses.