Skip to content

A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems

🕒 Published (v1): 2025-06-03 15:32 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A two-agent LLM framework — a Dialect Agent that translates non-SAE queries into Standard American English and a Privacy Policy Agent that answers them — mitigates dialect-based performance disparities in privacy policy QA without any model retraining. On PrivacyQA, GPT-4o-mini zero-shot accuracy jumps from 0.394 to 0.601; maximum cross-dialect F1 disparity drops by up to 82%. The framework is model-agnostic and requires only minimal dialect metadata supplied as a prompt.

Problem

Privacy policy QA systems perform significantly worse on non-standard English dialects (AAVE, Jamaican, Aboriginal, Welsh, Southwest England) than on Standard American English, disproportionately harming marginalized communities. Prior mitigation approaches require dialect-specific fine-tuning or large dialectal training corpora, which are impractical to collect in sensitive domains like privacy.

Method

A prompt-based two-agent pipeline operating at inference time:

  1. Dialect Agent (Step 1): Given a concise linguistic description of the user's dialect, the agent translates the dialectal query \(q_d\) into Standard American English (SAE) while preserving intent.
  2. Privacy Policy Agent (Step 2a): Receives the SAE-translated query plus a privacy policy snippet and produces an initial answer with rationale.
  3. Dialect Agent validation (Steps 2b/2c): The original dialectal question, the policy segment, and the Privacy Agent's answer are fed back to the Dialect Agent, which judges whether the answer captures the user's dialectal intent. On disagreement, the Privacy Agent is prompted to revise; this loop runs at most 2 iterations until agreement or the iteration cap is hit.

The framework is formally defined around minimizing the disparity metric \(\Delta(f) = \max_{d_i, d_j \in D} |\Phi_{d_i}(f) - \Phi_{d_j}(f)|\). Dialects are synthesized from SAE benchmarks using Multi-VALUE's 189 rule-based linguistic perturbations.

Key Contributions

  • Exhaustive benchmark of dialect biases across state-of-the-art LLMs (GPT-4o-mini, Llama 3.1 8B, DeepSeek-R1-Distill-Qwen-14B) on PrivacyQA and PolicyQA.
  • A novel training-free multi-agent framework requiring only brief dialect metadata as a prompt, enabling deployment across models without fine-tuning.
  • Structured ablation and error analysis of the agent collaboration loop, including the effect of initial vs. final answers and few-shot example inclusion.

Results

PrivacyQA: - GPT-4o-mini: zero-shot F1 0.394 → multi-agent-zero 0.601; Max Diff reduced from 0.093 (zero) to 0.025 (multi-agent-zero). - Llama 3.1: AVG F1 0.368 (zero) → 0.528 (multi-agent-zero); Max Diff 0.144 → 0.029. - DeepSeek-R1: Max Diff 0.037 (zero) → 0.017 (multi-agent-zero), lowest across all models. - Multi-agent-few (GPT-4o-mini) achieves highest average F1 (0.598), surpassing few-shot baseline (0.565).

PolicyQA: - GPT-4o-mini: zero-shot 0.352 → multi-agent-zero 0.464; Max Diff 0.029 → 0.020. - Llama 3.1: AVG Diff reduced from 0.023 (zero) to 0.006 (multi-agent-zero). - Cross-dialect disparity reductions of up to 82% measured by Max Diff.

Limitations

  • Dialect descriptions are short hand-crafted summaries; coverage quality depends on the accuracy and completeness of those summaries.
  • Multi-VALUE generates synthetic dialect data via rule-based perturbation, not natural speaker production; results may not fully generalize to authentic dialectal speech.
  • The refinement loop is capped at 2 iterations, which may be insufficient for highly ambiguous dialectal queries.
  • Evaluated on only 5 of the 50 Multi-VALUE dialects in the main text (lowest-performing subset); full-dialect results are in the appendix only.
  • Increased inference cost: each query requires multiple LLM calls (at minimum 2, up to ~4 with iterations).
  • The footnoted disclaimer acknowledges this is prompt-based coordination, not autonomous multi-agent systems in the classical sense.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a prompt-based multi-agent harness: two specialized LLM roles (Dialect Agent, Privacy Policy Agent) are orchestrated through a fixed control loop with structured inter-agent feedback and an iteration cap — the core pattern of a meta-harness that manages sub-agent lifecycles and refinement cycles. The iterative disagreement-and-revise loop mirrors the critic/reviser patterns seen in agentic harness design, and the paper explicitly notes this is structured prompt coordination rather than autonomous agency, making it a clean reference case for harness taxonomy. The training-free, model-agnostic design also demonstrates how harness-level orchestration can substitute for dataset-level intervention, a transferable architectural principle.