Spoken Conversational Agents with Large Language Models¶

🕒 Published (v1): 2025-12-02 10:02 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This is a tutorial paper (ACL/EMNLP-style proposal) covering spoken conversational agents built on LLMs, tracing the arc from probabilistic n-gram language models through end-to-end multimodal speech-language models. It surveys cascaded vs. end-to-end architectures, cross-modal adaptation theory, and open challenges including accent/dialect fairness. No novel method or experimental results are presented.

Problem¶

Despite strong performance of closed-source voice-LLMs (GPT-4o, Gemini-1.5-pro) on classical speech tasks, there is no comprehensive treatment of the design principles and mechanisms for integrating speech modalities into LLMs, nor of multi-agent/self-play extensions for spoken dialogue.

Method¶

Three-hour structured tutorial organized into thematic 35-minute segments: (1) history of probabilistic LMs for speech; (2) LLM adaptation for audio/speech signals, including cross-modal pre-training, population-risk-based transferability estimation, and post-ASR LLM correction; (3) spoken dialogue systems, from Siri-era intent/slot models through RL-based dialogue management to LLM-powered multi-turn TOD/ODD systems. Covers both cascaded (ASR → LLM) and end-to-end (joint speech-text tokenization, autoregressive generation) paradigms.

Key Contributions¶

Unified historical narrative from n-gram/Bayesian LMs → contextual end-to-end models → voice-interface LLMs.
Survey of cross-modal adaptation theory (population risk, transferability estimation) motivating different spoken-agent pipeline designs.
Overview of fairness/diversity gaps: accent, dialect, sociolect variability; argues for fairness-driven evaluation metrics beyond WER.
Discussion of future directions: situated dialogue grounded in web pages, tables, forms, vision, knowledge graphs, and embodied expression (gestures, facial movement).
Ethics coverage: speech privacy, speaker identification risks, equitable data collection.

Results¶

No empirical results reported. This is a tutorial survey paper; all cited numbers and performance claims belong to referenced works (e.g., Radford et al. 2023 / Whisper, Reid et al. 2024 / Gemini 1.5, Zhang et al. 2023 / SpeechGPT).

Limitations¶

Presents no original experiments, benchmarks, or novel technical contributions.
Coverage is inherently selective (7–8 key papers examined in depth out of dozens cited).
Claims about model performance disparities across accents/dialects are qualitative; no quantitative fairness analysis is provided.
Tutorial format means depth on any single topic is constrained by a 35-minute segment structure.

Relevance to Agentic AI / LLM Agents¶

The paper explicitly frames spoken dialogue systems as a form of multi-agent architecture, citing self-play and layered cognitive agents as a key motivation—directly relevant to agent orchestration research. The progression from RL-based dialogue management to LLM-powered TOD/ODD systems mirrors the broader shift from programmatic to neural agents. Future directions on situated dialogue (grounding agents in structured documents, tables, vision, and knowledge) map closely to tool-augmented and multimodal LLM-agent paradigms. The fairness and diversity discussion raises underexplored robustness challenges for deployed conversational agents.