PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue¶

🕒 Published (v1): 2026-06-11 04:59 UTC · Source: Arxiv · Venue: INTERSPEECH 2026 · link

Why this paper was selected

INTERSPEECH 2026; multi-agent prosody+semantic reasoning pipeline framework

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PRISM is a four-agent framework (Perceiver, Manager, Responder, Vocalizer) for empathetic spoken dialogue that avoids prosody loss from cascade ASR pipelines by translating acoustic/paralinguistic signals into natural-language descriptions before LLM reasoning. It decouples speech perception, dialogue management, response generation, and TTS synthesis into specialized agents coordinated via structured intermediate representations. PRISM outperforms all baselines on the AvaMERG multimodal empathetic dialogue benchmark across automatic, human, and LLM-based evaluations.

Problem¶

Cascade ASR→LLM→TTS pipelines discard prosodic/emotional cues at the transcription step, yielding semantically appropriate but empathetically hollow responses. End-to-end speech models preserve acoustic richness but treat emotion as implicit, resist knowledge injection without retraining, and accumulate errors across strictly sequential stages. No prior work jointly models prosodic perception, emotional reasoning, on-demand knowledge augmentation, and prosody-controlled synthesis in a single interpretable pipeline.

Method¶

PRISM decomposes empathetic spoken dialogue into four agents:

Perceiver: Extracts structured paralinguistic state \(s = \{T, a\}\) from raw speech using Whisper (ASR), emotion2vec (emotion classification), WebRTC VAD (pause ratio \(p = t_{\text{silence}}/t_{\text{total}}\)), frame-level RMS energy \((\mu_E, \sigma_E)\), filler rate \(f = N_f / N_{\text{tok}}\), and a heuristic certainty score \(c = \text{clip}(0.55\tilde{r} + 0.25(1-\tilde{p}) + 0.20(1-\tilde{f}),\ 0, 1)\).
Manager: Translates numeric/categorical prosodic attributes into a natural-language description \(D\) via rule-based thresholding followed by few-shot LLM prompting (GPT-3.5-Turbo). Also performs lightweight post-hoc verification of Responder outputs for emotion/strategy alignment.
Responder: A fine-tuned LLM (Qwen2.5-7B or Llama-3.1-8B, trained on TOOL-ED) that takes \((T, D, H)\) as input and jointly decides whether to invoke external commonsense tools (COMET-BART) and generates response text \(R\) plus target emotion \(e\) and intensity \(\lambda\).
Vocalizer: Controls StyleTTS2 via two-stage parameter computation: base parameters \((\alpha, \beta, d, \kappa)\) initialized from \((e, \lambda)\), then adaptively refined using user paralinguistic attributes \(a\). Applies text-side prosody shaping (pause insertion, punctuation adjustment) before synthesis.

Key Contributions¶

A multi-agent architecture with feedback-driven coordination that propagates emotional intent from perception through synthesis without irreversible information loss.
A prosody-to-language translation mechanism converting acoustic numerical cues into interpretable natural-language descriptions for stable LLM reasoning.
On-demand tool invocation for external commonsense knowledge (COMET-BART), enabling plug-and-play knowledge updates without retraining any model component.
A two-stage prosody parameter controller in Vocalizer that conditions TTS on both target empathetic intent and the user's observed prosodic state.

Results¶

PRISM (Qwen) ROUGE-1/2/L: 0.2254/0.0745/0.1872 vs. best baseline Qwen2.5-Omni-7B: 0.1880/0.0542/0.1555.
PRISM (Llama) BERTScore: 0.8801, highest among all models; PRISM (Qwen): 0.8792.
BLEU-4: PRISM (Qwen) 0.0571, PRISM (Llama) 0.0555, vs. best baseline (Qwen2.5-Omni-7B) 0.0352.
Dist-2 (lexical diversity): PRISM (Llama) 0.2574, PRISM (Qwen) 0.2519, vs. best baseline (Qwen2.5-Omni-7B) 0.2380.
Human evaluation (5-point Likert): PRISM comparable or superior to LLaMA-Omni2 and OpenS2S across Empathy, Fluency, Informativity, Consistency, Prosodic Appropriateness, Audio-Text Alignment; ICC = 0.81.
LLM-based A/B (GPT-4o): PRISM wins majority over both OpenS2S and LLaMA-Omni2 on empathy, fluency, consistency.
Ablation: removing prosody description or replacing adaptive knowledge invocation with always-on or always-off knowledge degrades all metrics substantially.

Limitations¶

Manager relies on GPT-3.5-Turbo (closed API), introducing latency, cost, and reproducibility concerns.
Evaluated on a single dataset (AvaMERG/TOOL-ED); generalization to other languages or domains is untested.
Prosody-to-language translation uses heuristic thresholding rules; rule quality is not systematically ablated.
No latency or real-time performance figures reported; suitability for live dialogue is unclear.
Certainty score \(c\) is a hand-crafted heuristic with fixed weights (0.55/0.25/0.20), not learned from data.

Relevance to Harnesses / Meta-Harnesses¶

PRISM is a concrete domain-specific multi-agent harness that coordinates specialized sub-components (Perceiver, Manager, Responder, Vocalizer) via structured intermediate representations—a paradigm directly relevant to harness/meta-harness design. The Manager's dual role as both a prosody-to-language translator and a post-hoc alignment verifier exemplifies how a harness layer can orchestrate heterogeneous sub-agents while enforcing consistency constraints without retraining. The on-demand tool invocation pattern (Responder decides at runtime whether to call COMET-BART) demonstrates how harnesses can route to external capabilities dynamically, a key design primitive in meta-harness architectures. For researchers building general-purpose orchestration harnesses, PRISM illustrates how decoupling perception, reasoning, and generation into separately tunable agents enables modular updates and interpretable error attribution across the pipeline.