Skip to content

A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

πŸ•’ Published (v1): 2026-07-08 23:24 UTC Β· Source: Arxiv Β· link

Why this paper was selected

Empirical reliability of audio LLM judges for full-duplex voice agents β€” evaluation methodology for conversational agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

πŸ’¬ Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically validates Gemini audio-language models (LALMs) as judges of full-duplex voice-agent conversations, scoring raw stereo waveforms on 8 production dimensions against a 3-rater human reference standard. On 5–6 of 8 dimensions, Gemini 2.5 Flash performs comparably to a fourth human rater, at roughly two orders of magnitude lower cost. The study is the first to validate a LALM judge over multi-rater, full-duplex conversational audio rather than isolated TTS utterances.

Problem

Prior LALM-as-judge work benchmarks models on isolated text-to-speech utterances with single-rater or system-level MOS correlations. No study had validated a LALM judge against a calibrated multi-rater human standard on raw stereo full-duplex agent–client conversations, where turn-taking, channel asymmetry, and conversational disfluencies all affect rating distributions. The study fills this gap for production-grade evaluation pipelines.

Method

Corpus: 209 stereo sessions from a production customer-support voice agent β€” 152 natural full-duplex conversations across 13 accent-and-condition strata (6 clean Γ— 6 accents, plus 7 codec-degraded strata), plus 57 adversarial defect-injected clips (6 DSP defects: hard clipping, dead air, additive noise at βˆ’2 to βˆ’8 dB SNR, truncation, phoneme overdub, 8 kHz resample).

Judges: Two production LALM judges (AgentSpeechFidelity and ConversationalAudioQuality, both Gemini 2.5 Flash via Vertex AI) receive raw stereo WAV bytes with no transcription; each returns structured JSON scores on 8 Likert-scale (1–5) dimensions.

Human baseline: 3 Salesforce-contracted annotators scored all 209 sessions Γ— 8 dimensions with anchored rubrics and a pre-batch calibration meeting (5,016 observations total).

Agreement statistics: (i) pairwise Spearman \(\rho\) (1,000-iteration bootstrap 95% CIs) for rank agreement; (ii) proportion of sessions within 1 point of the 3-rater human mean; (iii) ordinal Krippendorff \(\alpha\) for chance-corrected reliability; (iv) Newcombe-Wilson CIs on LALM-minus-human recall difference for defect detection.

Cross-model replication: All 209 sessions re-scored with Gemini 3.5 Flash and Gemini 3.1 Pro Preview, same prompts, same judges.

Key Contributions

  • First multi-rater validation of a LALM judge on raw full-duplex stereo agent-client audio (vs. prior work on isolated utterances).
  • Empirical evidence that Gemini 2.5 Flash achieves near-human rank agreement (\(\rho\) gap \(\leq 0.07\)) on 5/8 dimensions and \(\geq\)60% within-1-point simple agreement on 6/8 dimensions.
  • Identification of a "reliability paradox": Krippendorff \(\alpha \approx 0\) even at >90% within-1 agreement on ceiling-saturated dimensions, motivating per-regime reporting.
  • Adversarial defect analysis showing LALM and human coverage are complementary, not nested (e.g., LALM catches clipping as prosody degradation; humans catch it as audio_clarity degradation).
  • Cross-model finding that rank-ordering transfers across Gemini versions, but absolute calibration does not β€” a model swap requires re-validating simple agreement, not just rank correlation.
  • Directional cost estimate: ~\(2\) orders of magnitude cost reduction vs. three-rater human coverage of a 100-session weekly batch.

Results

  • Rank agreement: LALM-human \(\rho\) departs from human-human \(\rho\) by \(\leq 0.07\) on 5/8 dimensions; 95% bootstrap CIs overlap on 7/8 dimensions for Gemini 2.5 Flash.
  • Simple agreement: \(\geq\)60% within-1-point of human mean on 6/8 dimensions; three dimensions reach \(\geq\)89% (accent_dialect_handling 92.3%, speech_clarity 90.9%, audio_clarity 89.0%).
  • Failing dimensions: speaking_rate_adaptation (48.8%) and overall_fidelity (43.5%) fall below 50%; humans themselves show low rank agreement there (ρ = 0.17 and 0.11).
  • Adversarial sensitivity: LALM is significantly more sensitive than humans in 4/48 (defect Γ— dimension) cells; humans are significantly more sensitive in 3/48 cells; 41/48 show no significant difference (underpowered at \(n \leq 8\)).
  • Critical blind spot: clip Γ— audio_clarity β€” humans detect clipping 8/8; LALM 0/8. Robust under paired McNemar test.
  • Gemini 3.5 Flash: Improves simple agreement to 8/8 dimensions (\(\geq\)60%), rank-correlation gap within 0.07 on 4/8.
  • Gemini 3.1 Pro Preview: Rank correlation comparable (5/8 within 0.07), but calibration offset causes audio_clarity mean score to be βˆ’1.52 points below human mean; within-1 agreement drops to 34.9% on audio_clarity and 37.3% on speaking_rate_adaptation.
  • Natural conversations alone (n=152): Headline counts unchanged; audio_clarity human-human \(\rho\) collapses from 0.47 to 0.07 once adversarial clips are excluded, revealing pooled correlation was measuring shared defect sensitivity, not natural-speech discrimination.

Limitations

  • Primary evidence base is a single LALM (Gemini 2.5 Flash) on a single production system; generalizability to other voice agents or LALM providers is untested.
  • Adversarial arm is underpowered: \(n \leq 8\) per (defect, dimension) cell β€” 41/48 non-significant cells are absence of power, not demonstrated equivalence.
  • Human raters are a single three-person panel from one annotation vendor; inter-rater variance is high (rater means differ by >1 Likert point on speech_clarity and audio_clarity), limiting the robustness of the human ground truth.
  • Cost estimates are directional internal figures, not reproducible unit-cost models.
  • overall_fidelity and speaking_rate_adaptation show poor reliability for both humans and LALM; the paper attributes this to rubric construct invalidity but does not fix it.
  • The study covers only the Gemini model family; cross-provider generalization is unexamined.
  • Paper text is truncated before Section 7 is complete, so additional stated limitations may exist.

Relevance to Agentic AI / LLM Agents

Automated evaluation of voice agents is a direct bottleneck for scaling agentic AI quality assurance: human rating is the current ceiling on how frequently production agent conversations can be audited. This work provides a methodology and empirical threshold (within-1 simple agreement \(\geq\)60%, \(\rho\) gap \(\leq 0.07\)) for deciding when an LLM-as-judge can substitute for human raters in production agent pipelines, directly applicable to any agentic system producing auditable outputs. The cross-model calibration finding β€” that rank-ordering transfers but absolute scoring does not across model versions β€” is a practical warning for teams maintaining automated judge pipelines for agents as underlying models are upgraded. The complementary-coverage pattern (LALM and humans catch different defects) argues for hybrid human-LALM evaluation architectures rather than full replacement, a design principle relevant to any agent monitoring system.