Structured Reasoning for Fairness: A Multi-Agent Approach to Bias Detection in Textual Data¶

🕒 Published (v1): 2025-03-01 05:27 UTC · Source: Arxiv · Venue: AAAI 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a three-agent pipeline (checker → validation → justification) for detecting bias in text by first classifying statements as fact or opinion, then scoring bias intensity, and finally generating an interpretable explanation. Evaluated on 1,500 WikiNPOV samples using GPT-4o, it achieves 84.9% accuracy versus 71.9% for a zero-shot baseline—a 13.0 percentage point gain significant at \(p < 10^{-6}\).

Problem¶

Existing bias detection methods rely on static lexicons or single-pass LLM prompts that (a) miss context-dependent and nuanced biases, (b) conflate factual and opinionated statements, and (c) produce no interpretable justification for why a statement is flagged.

Method¶

A sequential multi-agent pipeline built on GPT-4o (also tested with Claude 2.1 and Gemini 1.5 Flash):

Checker agent: Classifies each statement \(S\) as FACT or OPINION via \(\text{Decision}(S) \in \{\text{FACT}, \text{OPINION}\}\).
Validation agent: For OPINION statements (and FACT statements flagged as potentially biased), computes \(\text{Validate}_{\text{bias}}(S) = f_{\text{LLM}}(S)\) and assigns a binary bias level (HIGH → biased, LOW → unbiased).
Justification agent: Generates a concise, token-level explanation referencing specific words or framing devices that caused the classification.

Factual statements with no bias signal are short-circuited directly to justification, conserving compute. All agents are called asynchronously; output is structured JSON containing statement type, bias score, and justification.

Key Contributions¶

A staged fact/opinion decomposition prior to bias scoring, demonstrating that disentangling these dimensions improves downstream accuracy.
A validation agent with HIGH/LOW bias intensity scoring for opinionated and escalated factual statements.
An interpretability layer that produces token-level, auditable justifications alongside every binary classification.
Cross-LLM portability: pipeline accuracy stays near 80% when swapping GPT-4o for Claude 2.1 or Gemini 1.5 Flash on a 100-statement subset.

Results¶

Pipeline accuracy: 84.9% vs. Baseline 71.9% on 1,500 WikiNPOV statements (+13.0 pp); \(z > 8.0\), \(p < 10^{-6}\).
Pipeline F1: 0.505 vs. Baseline 0.472; Pipeline improves precision (0.494 vs. 0.328) while sacrificing some recall (0.518 vs. 0.839).
Cross-LLM (100-statement subset):
GPT-4o: 83.0% accuracy, F1 0.564
Claude 2.1: 81.0% accuracy, F1 0.296 (notably low recall: 0.235)
Gemini 1.5 Flash: 78.0% accuracy, F1 0.736

Limitations¶

Evaluated on a single dataset (WikiNPOV); generalization to medicine, law, or journalism is unvalidated.
Binary fact/opinion classification cannot handle hybrid statements that blend verifiable claims with subjective framing.
Bias intensity is binary (HIGH/LOW), losing gradation; continuous scoring would better represent severity.
Claude 2.1 shows sharply degraded recall (0.235), suggesting significant model-sensitivity not fully explained.
No multilingual evaluation; no confidence calibration reported.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a direct instance of a task-specific multi-agent harness: a fixed, sequential pipeline of specialized LLM agents (checker → validator → justifier) where each stage gates or routes inputs to the next. It demonstrates the core harness design pattern—agent decomposition, conditional branching (fact escalation path), and structured JSON output for downstream auditability—applied to a classification task. For harness researchers, it is a concrete case study of how a staged agent topology outperforms a single-prompt baseline and how output schemas enable cross-agent state passing. The asynchronous, modular design also illustrates how harnesses can be made LLM-agnostic by isolating model calls behind a uniform prompt interface.