Skip to content

Agent-Based Modular Learning for Multimodal Emotion Recognition in Human-Agent Systems

🕒 Published (v1): 2025-12-02 21:47 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a multi-agent framework for multimodal emotion recognition where each modality encoder (facial, speech, text) operates as an autonomous agent coordinated by a central supervisor-classifier orchestrator. The key design goal is modularity: agents can be independently trained, replaced, or extended without retraining the entire pipeline. The implementation is a proof-of-concept supporting vision (YOLOv8-Face + ResNet-50), audio (emotion2vec), and text (Whisper + FRIDA) modalities evaluated on CMU-MOSEI.

Problem

Monolithic multimodal emotion recognition models couple all modality encoders into a single network, making them expensive to retrain when a modality encoder is updated, fragile to modality addition/removal, and opaque in terms of per-modality contribution. There is no prior work that systematically evaluates how modular agent-based architectures affect accuracy, retraining efficiency, and interpretability in multimodal emotion recognition.

Method

Four specialized agents process video input in parallel: FED (Facial Emotion Detection: YOLOv8-Face → ResNet-50 → 512-dim embeddings), SER (Speech Emotion Recognition: emotion2vec → 256-dim embeddings), TED (Text Emotion Detection: Whisper Large V3 Turbo → FRIDA → 768-dim embeddings), and AED (Audio Event Detection: CNN-14 on AudioSet → 527-dim tags used only as gating metadata, not fused).

Each agent's variable-length temporal sequence is pooled (mean/median/max) and zero-padded/truncated to a uniform 1024-dim vector. The three primary modality vectors are concatenated into \(f_{\text{concat}} \in \mathbb{R}^{3072}\).

A Ridge regression adapter then aligns \(f_{\text{concat}}\) to the normalized MOSEI embedding space (original 35+74+300 features padded to 1024 per modality), allowing pre-trained classifiers to be reused without retraining when encoders change: $\(f_{\text{scaled}} = \frac{f_{\text{concat}} - \mu_{\text{adapter}}}{\sigma_{\text{adapter}}}, \quad f_{\text{adapted}} = W_{\text{adapter}} f_{\text{scaled}} + b_{\text{adapter}}\)$

The fusion classifier (orchestrator) supports CatBoost, MLP, or logistic regression on \(f_{\text{processed}} \in \mathbb{R}^{3072}\) and outputs 5-class sentiment (Very Negative → Very Positive).

Training is staged: (1) each agent trained independently on modality-specific data; (2) adapter trained on paired (current pipeline, normalized MOSEI) embeddings; (3) supervisor classifier trained on MOSEI.

Key Contributions

  • Multi-agent supervisor architecture for multimodal emotion recognition with clean agent/tool separation (deterministic preprocessing = tools; reasoning/embedding = agents).
  • Ridge regression adapter that decouples the fusion classifier from encoder architecture, enabling encoder hot-swap without full retraining.
  • AED as auxiliary gating metadata (speech-presence flag) rather than a fused modality, preventing uninformative channels from polluting the fusion space.
  • Open-source implementation integrating state-of-the-art encoders (emotion2vec, YOLOv8-Face, Whisper Large V3 Turbo, FRIDA, CNN-14).

Results

The paper text provided does not report quantitative benchmark numbers (accuracy, F1, WAF, etc. on MOSEI or any other dataset). The work is described explicitly as a proof-of-concept; no ablation tables or comparison baselines appear in the supplied text.

Limitations

  • No quantitative evaluation is reported in the available text; the claim of maintained accuracy is unverified.
  • The Ridge regression adapter assumes a linear alignment between the current pipeline's embedding space and MOSEI's original feature space — a strong assumption when encoders differ substantially in architecture.
  • Zero-padding to 1024-dim per modality for embeddings smaller than 1024 introduces structural sparsity that may degrade downstream classifiers.
  • Temporal pooling (mean/median/max) to produce fixed-length vectors discards temporal dynamics, which are informative for emotion inference.
  • The paper is a single-institution proof-of-concept; no comparison to monolithic MER baselines (e.g., MulT, MISA, UniMSE) is provided.

Relevance to Harnesses / Meta-Harnesses

The supervisor-orchestrator pattern here is a direct instantiation of the meta-harness concept: independent specialized sub-agents expose a standardized embedding interface, and a central coordinator assembles their outputs into a final decision. The Ridge regression adapter acts as an interface contract layer — analogous to an adapter harness that lets sub-agents be hot-swapped without disrupting the orchestrator's expectations. The staged training pipeline (per-agent → adapter → supervisor) mirrors harness initialization sequences where components are validated independently before integration. For harness engineers, the paper's agent/tool distinction (deterministic preprocessing = tool, learned reasoning = agent) is a useful design principle that clarifies where modular boundaries should sit.