Skip to content

Dialogue to Detection: A Multimodal Hybrid NLP Pipeline for Insurance Fraud Detection

🕒 Published (v1): 2026-06-26 11:59 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A synthetic multimodal pipeline for insurance fraud detection at the First Notification of Loss (FNOL) stage, generating GPT-2 dialogues and xTTS two-speaker audio, then fusing NER, BERT-RAG semantic retrieval, and Resemblyzer speaker embeddings into an interpretable rule-based risk score. The dataset and pipeline are fully open-source, addressing the lack of public multimodal benchmarks in insurance fraud.

Problem

Insurance fraud costs $300B+/year (US) and £1.1B+/year (UK), but FNOL fraud detection is held back by the absence of publicly available multimodal datasets pairing conversational call audio with aligned transcripts — real data is locked behind privacy regulations (GDPR) and industry contracts, and existing public corpora are text-only or from non-insurance domains. Speech carries paralinguistic deception cues (tone, hesitation), but without shareable benchmarks, progress on cross-modal fusion (voice identity + narrative similarity) is stalled.

Method

An end-to-end open-source pipeline with six stages:

  1. Synthetic Dialogue Generation — GPT-2 (GPT2LMHeadModel) generates balanced fraudulent/legitimate FNOL transcripts from user-provided templates and structured variables (name, age, postcode, policy number). Stochastic decoding (temperature + nucleus/top-k) injects conversational disfluencies ("um", "uh").
  2. Multi-Speaker Audio Synthesis — xTTS converts transcripts into two-speaker agent–customer call recordings. 38 distinct synthetic speakers across the full dataset.
  3. ASR + Diarisation — WhisperX transcribes and diarises the audio, producing time-stamped, speaker-separated transcripts with realistic ASR errors. Customer speech is isolated for downstream analysis.
  4. Entity Extraction — Hybrid NER (RoBERTa-base) + regex extracts structured attributes (name, postcode, age, product, policy number). Regex guarantees precision for formalised identifiers; NER covers unseen contexts.
  5. Hybrid NLP Classification + RAG — Fine-tuned BERT (lr=2e-5, batch=4, weight decay=0.01) for binary fraud classification. A FAISS-indexed sentence-transformer (all-MiniLM-L6-v2) retrieves semantically similar historical claims via LangChain for RAG, flagging narrative reuse across customers at >90% similarity + >0.9 confidence.
  6. Speaker Clustering — Resemblyzer extracts fixed-dimensional speaker embeddings; cosine similarity >0.75 with cluster size >4 signals cross-case voice reuse.
  7. Fraud Risk Scoring — Weighted fusion \(R = w_f S_{\text{features}} + w_s S_{\text{similarity}} + w_v S_{\text{voice}}\) with \((w_f=0.4, w_t=0.3, w_v=0.3)\) mapping to low/medium/high risk bands. Structured feature checks include early claims (<30d), high-cost (>1.2× avg), payment irregularities (>5 missed), and household profile inflation (>4 profiles).

Key Contributions

  • First reproducible synthetic multimodal FNOL dataset (balanced fraudulent/legitimate, realistic entity distributions and audio properties)
  • First end-to-end multimodal pipeline for insurance fraud integrating dialogue generation, two-speaker audio synthesis, ASR+diarisation, NER+regex, BERT-RAG, and speaker embedding clustering
  • Interpretable rule-based risk-scoring framework fusing structured, textual, and vocal signals — transparent audit trail per flag

Results

Synthetic dataset (1009 samples total): 500 customer-only transcripts, 250 agent–customer transcripts + 250 recordings (12 speakers), 9 feature-only texts + 9 gTTS recordings, 250 feature-rich agent–customer transcripts + 250 xTTS recordings (38 speakers).

Transcription & Feature Extraction (from Table 1): - STT WER: 11.9% (Common Voice baseline), 15.9% (xTTS transcribed→diarised→features), 20.4% (xTTS gTTS-based) - Feature extraction F1 on xTTS transcribed-diarised: Name 85.4%, Postcode 47.5%, Age 85.9%, Policy No 35.9%, Product 92.1% - Feature extraction F1 on direct text: 100% across all fields

Binary BERT classifier (Table 2): - Accuracy 53–67% on TTS/xTTS diarised/undiarised transcripts - Recall 43–96% (highest on xTTS diarised: 95%) - Average confidence 87–93% - On gTTS single-speaker (simplest setting): accuracy 89%, precision 96%, recall 95% - Test set WER 0.0085 with 100% accuracy/precision/recall/F1/AUC — caveated as optimistic due to small, controlled synthetic data

Voice clustering: Full pipeline runs with 46.62% diarisation accuracy on common voice dataset, 93.24% on xTTS-based recordings.

Limitations

  • Synthetic dataset is limited in size and controlled in variability, yielding optimistic classifier performance estimates (authors explicitly caution this)
  • Diarisation accuracy remains modest (46.62% on Common Voice baseline), which propagates errors through entity extraction and speaker embedding stages
  • Feature extraction F1 is weak for postcode (47.5%) and policy number (35.9%) on transcribed+diarised audio — ASR noise significantly degrades structured attribute capture
  • Assumes deterministic heuristics (agent speaks first for speaker labelling; household size ≤4) that may fail in real call-centre conditions
  • No real-world evaluation — entirely synthetic, so transfer to operational FNOL environments is unvalidated
  • Rule-based scoring weights (0.4/0.3/0.3) are manually set, not learned from data

Relevance to Agentic AI / LLM Agents

This work is a pipeline orchestration system, not a single agent, but several components are agent-adjacent: the GPT-2 dialogue generator acts as a data-generation agent, the BERT-RAG module retrieves and conditions on historical context (a retrieval-augmented decision agent), and the rule-based fusion layer aggregates heterogeneous signals into an interpretable judgement. For the agentic AI audience, the architecture demonstrates how to wire together open-source NLP/Speech components into a multi-stage reasoning pipeline for a high-stakes domain — a pattern directly transferable to fraud detection, compliance, and other regulated decision-making settings where interpretability is non-negotiable.