Skip to content

BAID: A Benchmark for Bias Assessment of AI Detectors

🕒 Published (v1): 2025-12-12 12:01 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BAID is a fairness benchmark of 208,166 document pairs evaluating four open-source AI-generated text detectors across seven sociolinguistic bias dimensions. It reveals systematic recall disparities that disproportionately disadvantage underrepresented writing styles—particularly dialectal (AAVE, Singlish) and informal (GenZ) text—exposing how aggregate detector accuracy masks subgroup-level inequity.

Problem

Existing AI-text detector evaluations focus on detection accuracy under neutral conditions, not fairness. Prior work identified bias against English Language Learners (ELLs) in isolation, but no systematic, multi-dimensional audit existed covering demographics, dialect, age, grade level, formality, political ideology, and topic simultaneously.

Method

The authors construct paired datasets: for each human-written document, GPT-4.1 or Claude Sonnet 3.7 generates a semantically equivalent AI-authored counterpart using subgroup-specific zero-shot prompts (e.g., AAVE-authentic syntax for dialect samples, GenZ slang conventions for formality samples). Semantic alignment is verified with cosine similarity ≥ 0.85 between pairs. Four detectors—Desklib (DeBERTa-v3-large fine-tuned), E5-small (LoRA-tuned encoder), Radar (adversarially trained), and ZipPy (compression-ratio heuristic)—are evaluated at default thresholds on human-written texts only, with precision, recall, and F1 reported per subgroup.

Key Contributions

  • BAID dataset: 208,166 human/AI document pairs spanning 41 subgroups across 7 bias dimensions, sourced from ASAP 2.0, Blog Authorship Corpus, AAVE/Singlish/SAE corpora, GenZ dataset, and political news datasets.
  • Structured prompt design preserving semantic content while inducing subgroup-specific writing style in AI counterparts.
  • Systematic disaggregated evaluation of four detectors, surfacing per-subgroup precision/recall/F1 rather than aggregate metrics.
  • Empirical evidence that recall—not precision—is the primary axis of bias, with underrepresented groups suffering the highest misclassification rates.

Results

  • Desklib achieves highest overall consistency (F1 0.89–0.96 on demographics/grade level) but degrades severely on Singlish (F1 0.33), AAVE (0.27), and GenZ (0.14).
  • E5 maintains high precision on demographics (0.95–0.99) but extremely low recall (0.03–0.45), yielding F1 of 0.30–0.45; near-total failure on GenZ (F1 0.04).
  • Radar is moderate and stable (F1 0.60–0.75) but drops on Singlish (0.21) and GenZ (0.02).
  • ZipPy has near-zero F1 on demographics and most grade levels (0.03–0.27) due to length sensitivity, but achieves high recall (0.95–0.99) on age, dialect, and topic categories with low precision (0.49–0.54), yielding F1 ~0.65–0.67.
  • Political ideology dimension: Desklib F1 0.93–0.97; E5 F1 0.06–0.14; ZipPy F1 0.58–0.59.
  • Recall on AI-generated samples is uniformly high across all detectors, confirming detectors recognize machine-text statistical fingerprints but not human writing from underrepresented groups.

Limitations

  • Only four detectors evaluated; commercial systems (GPTZero, Turnitin) and multimodal/cross-lingual models excluded.
  • Benchmark is English-only; no multilingual extension.
  • ZipPy's compression-based architecture is highly sensitive to input length, complicating direct comparison with neural models.
  • AI-generated subgroup samples reflect prompt conditioning, not authentic demographic diversity, limiting their use for bias measurement.
  • Default decision thresholds used throughout; threshold calibration could substantially alter results.

Relevance to Agentic AI / LLM Agents

As LLM agents are increasingly deployed to generate, edit, or moderate text at scale—in tutoring systems, content moderation pipelines, and hiring platforms—AI-text detectors serve as downstream gatekeepers whose biases propagate into agentic workflows. BAID provides the first multi-dimensional audit framework to quantify where these detectors systematically fail, which is directly relevant to anyone building agents that interact with or are audited by such detectors. The benchmark's structured prompt methodology for generating subgroup-conditioned synthetic text also demonstrates a reusable evaluation pattern applicable to fairness auditing of agentic outputs more broadly.