Dialogue Summarization with Emotion Dynamics Using Topic- and Participant-Centric Decomposition

🕒 Published (v1): 2026-07-16 09:53 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a hierarchical Chain-of-Agents (CoA) framework for dialogue summarization that jointly captures semantic content and emotion dynamics by decomposing dialogues from both topic-centric and participant-centric perspectives. It introduces new emotion trajectory metrics (Levenshtein, n-gram overlap, Jaccard, cosine over compressed emotion sequences) to evaluate how well summaries preserve emotional flow. Experiments on AMI and IEMOCAP using small LLaMA models show that combining both decomposition views and providing explicit emotion labels with trajectory-preservation instructions yields the best performance.

Problem

Existing dialogue summarization treats conversations as static transcripts focused on semantic content, ignoring: (1) the emotional trajectories of participants across a dialogue, (2) multimodal affective cues (audio, video), and (3) the complementary structural perspectives of topic progression versus participant-specific viewpoints. No prior work has systematically integrated multimodal emotion perception into summarization.

Method

The framework is a three-layer pipeline composed of five specialized LLM/MLLM agents:

1. Multimodal Data Understanding Layer: A Topic Segmentation Agent \(f_T\) partitions dialogue \(D = \{U_j\}_{j=1}^L\) into \(n\) topical segments \(\{T_i: (a_i, b_i)\}_{i=1}^n\); an Emotion Recognition Agent \(f_E\) assigns utterance-level emotion labels \(e_{U_j} \in E\) (7-class: neutral, happiness, sadness, surprise, excited, frustration, anger) from text, audio \(A_j\), and video \(V_j\).

2. Perspective Modeling Layer: A Topic Summarization Agent \(f_{S}^{\text{topic}}\) generates per-segment summaries \(S_{T_i}\) from utterance-emotion pairs within each topic. Concurrently, a Participant-Specific Summarization Agent \(f_{S}^{\text{participant}}\) generates per-speaker summaries \(S_{P_k}\) from that speaker's utterance-emotion pairs.

3. Summary Aggregation Layer: A Summary Aggregation Agent \(f_{S}^{\text{dialogue}}\) combines \(\{S_{T_i}\}\) and \(\{S_{P_k}\}\) into a global summary \(S_D\), formalized as \(f_{S}^{\text{dialogue}} \circ f_{S}^{\text{participant}} \circ f_{S}^{\text{topic}} \circ f_E \circ f_T(D) = S_D\).

Emotion trajectory evaluation compresses raw sequences \(e^{\text{gt}}, e^{\text{pred}}\) by removing consecutive duplicates into \(\hat{e}^{\text{gt}}, \hat{e}^{\text{pred}}\), then computes normalized Levenshtein similarity, n-gram overlap (n ∈ {1,2,3}), Jaccard over transition pairs \(T_2\), and cosine similarity over transition frequency vectors. Content quality is measured by reference-free BLANC-help: \(\text{BLANC}_\text{help} = \text{Acc}_\text{help} - \text{Acc}_\text{base}\).

Summarization agents use LLaMA 3.2 3B and LLaMA 3.1 8B; emotion recognition uses Emotion-LLaMA (multimodal); topic segmentation uses LLaMA 3.1 8B. Datasets: AMI (26 meetings, 4-person) and IEMOCAP (150 dyadic dialogues).

Key Contributions

• Hierarchical CoA framework for multimodal dialogue summarization integrating semantic and emotion dynamics from both topic and participant perspectives.
• Novel emotion trajectory evaluation suite: normalized Levenshtein, n-gram overlap, Jaccard, and cosine similarity computed on compressed emotion sequences.
• Empirical analysis showing that combined topic+participant decomposition outperforms either alone at the Summary Aggregation Layer, particularly on emotion metrics.
• Ablation over a 2×2 design (emotion labels present/absent × trajectory instruction present/absent) quantifying the separate and joint contributions of emotion annotations and prompting strategy.

Results

Perspective Modeling Layer (Table 1): Topic-based summaries consistently outperform participant-based across both models and datasets. Example (LLaMA 3.1 8B, AMI): Topic LEV=0.356 vs. Participant LEV=0.167; Topic NGR=0.507 vs. Participant NGR=0.503.

Summary Aggregation Layer (Table 2): Combined (topic+participant) setup outperforms topic-only and participant-only across most metrics. Example (LLaMA 3.1 8B, AMI): Combined NGR=0.540, COS=0.295 vs. Topic-only NGR=0.507, COS=0.278. BLANC varies little across setups (range ~0.013–0.081), while emotion metrics show larger differences.

RQ2 ablation (Table 4, LLaMA 3.1 8B, AMI, Combined setup): - Baseline E−/T− (no emotion labels, no trajectory instruction): NGR=0.325, COS=0.155 (3B); higher for 8B. - E+/T+ achieves best emotion metrics: NGR=0.503, COS=0.222 (3B, AMI). - Even E−/T+ (no labels, but trajectory instruction) improves over E−/T−, showing prompting alone helps. - BLANC scores remain largely insensitive to emotion configuration (~0.058–0.065 range for 3B AMI).

No external baselines (e.g., state-of-the-art summarization systems) are included; comparisons are internal ablations only.

Limitations

• No ground-truth summaries exist for this task; BLANC-help is used as a reference-free proxy, limiting content evaluation rigor.
• AMI retains only 26 dialogues after multimodal filtering; IEMOCAP lacks topic boundary annotations, requiring off-the-shelf models that may introduce label noise upstream.
• Emotion trajectory metrics are extracted from generated summaries by prompting LLaMA 3.1 8B — introducing a second model's errors into the evaluation loop.
• Only LLaMA 3.2 3B and LLaMA 3.1 8B are benchmarked; generalization to larger or proprietary models is unverified.
• Both datasets are scripted or meeting-style English dialogues; domain coverage is narrow.
• The fully automated pipeline (off-the-shelf topic and emotion models) is relegated to an appendix, leaving the end-to-end practical performance undercharacterized in the main results.

Relevance to Agentic AI / LLM Agents

This paper is a direct application of multi-agent LLM decomposition (Chain-of-Agents) to a complex NLP task, demonstrating how specialized agents handling constrained subtasks (segmentation, emotion recognition, perspective summarization, aggregation) outperform a monolithic model formulation. For researchers tracking agentic AI, it provides a concrete case study of hierarchical agent pipelines with parallelizable stages and structured information passing between agents. The framework also raises practical questions relevant to agentic systems deployed in real collaborative settings — specifically, how agents should handle privacy-sensitive emotion signals and operate gracefully when certain modality inputs are unavailable. The results reinforce that agent specialization pays dividends primarily in capturing nuanced, dynamic properties (emotion trajectories) rather than coarse content overlap, a finding with broader implications for agentic task decomposition.