Skip to content

RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

🕒 Published (v1): 2026-03-17 11:48 UTC · Source: Arxiv · Venue: INTERSPEECH 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RECOVER is a post-ASR agentic framework that corrects rare and domain-specific entity phrases by generating multiple ASR hypotheses via temperature sampling, retrieving top-K entity candidates via hybrid scoring, and orchestrating three specialized tools (fuse, propose, verify) with a constrained LLM editor. Across five diverse domains it achieves 8–46% relative entity-WER reduction with up to +22 pp entity recall gain, while leaving overall WER essentially unchanged.

Problem

End-to-end ASR models (e.g., Whisper) degrade on rare, domain-specific entities. Post-ASR LLM correction fails when an entity is deleted or heavily corrupted in the single best hypothesis—the LLM has no evidence signal to recover the missing term. Existing contextual biasing approaches require access to decoder internals unavailable in black-box production ASR.

Method

RECOVER operates as a tool-using agent (implemented in Agno) with three sequential tools over a shared context of \(N=5\) hypotheses (Whisper-small at \(T \in \{0.0, 0.2, 0.4, 0.6, 0.8\}\)) and top-\(K=200\) entity candidates.

Dynamic Entity Candidate Retrieval scores each candidate phrase \(b\) against all hypotheses using: $\(\text{score}(b) = n_{\text{exact}} \cdot w_e + f_{\text{best}} \cdot w_f + p_{\text{hit}} \cdot w_p\)$ with \(w_e=1.0\), \(w_f=1.2\), \(w_p=0.6\) combining exact token hits, normalized Levenshtein similarity (fuzzy), and phonetic prefix matching.

Tool 1 (Fuse Hypotheses) selects or merges hypotheses via one of four strategies: - 1-Best: passes greedy output unchanged. - Entity-Aware Select: picks the hypothesis with the most exact entity-candidate substring hits. - ROVER Ensemble: entity-aware pivot selection followed by Needleman–Wunsch global alignment and majority-vote token merge (insertions accepted only if supported by ≥3/5 hypotheses). - LLM-Select: delegates selection and initial entity fixes to the LLM given all hypotheses and top-K candidates.

Tool 2 (Propose Corrections) prompts the LLM (GPT-4o) to output strictly entity-list-bound find/replace edits as JSON with character offsets, confidence, and a reason.

Tool 3 (Verify & Apply) applies deterministic guardrails: replacement must be in the entity list; case-only changes are dropped; character offsets are re-computed if misaligned; normalized Levenshtein similarity between original and replacement must exceed a threshold; edits are applied left-to-right with overlap rejection.

Key Contributions

  • Multi-hypothesis diversity via temperature sampling from a single ASR model, demonstrating complementary errors across temperatures are exploitable for entity recovery.
  • Three-tool agentic pipeline cleanly separating hypothesis fusion, LLM-based proposal, and deterministic verification.
  • Four fusion strategies sharing the same downstream pipeline, enabling modular ablation.
  • Constrained LLM editing that restricts replacements to an entity phrase list, preventing hallucination and overcorrection.
  • Evaluation across five heterogeneous domains (finance, ATC, medical, general speech, dialogue) showing domain-dependent strategy advantage.

Results

  • 1-Best (single hypothesis + constrained LLM): 33.2% RWERR on Earnings-21, 41.2% on Common Voice, 20.4% on Eka-Medical, 12.7% on ContextASR-Bench.
  • LLM-Select (GPT-4o): best or near-best on 4/5 datasets; 33.4% RWERR (Earnings-21), 23.1% (Medical), 45.7% (Common Voice), 17.8% (ContextASR-Bench), 14.5% (ATCO2).
  • Entity recall gains with LLM-Select: +10.6 pp (Earnings-21), +8.2 pp (ATCO2), +6.2 pp (Medical), +21.5 pp (Common Voice), +0.9 pp (ContextASR-Bench).
  • Overall WER preserved or improved by LLM-Select: Medical 17.23%→13.93%; ContextASR 9.40%→9.30%; Earnings-21 13.59%→13.55%.
  • ROVER Ensemble on ContextASR-Bench achieves the top single RWERR (20.8%) but raises ATCO2 WER by +15.8 pp due to insertion noise.
  • GPT-4o-mini LLM-Select ablation shows 2.2–15.7% RWERR—smaller but consistent gains, confirming stronger LLM reasoning materially helps.
  • Substitution reduction accounts for 63–93% of total error reduction; insertions remain low or decrease in all settings.

Limitations

  • Requires a pre-built entity phrase list; not applicable when candidate entities are unknown at inference time.
  • ROVER Ensemble and Entity-Aware Select degrade overall WER on noisy domains (e.g., ATCO2 WER 48.63%→64.43%) due to insertion noise in non-entity regions.
  • Evaluated only with Whisper-small; generalization to other ASR architectures or languages is not demonstrated.
  • All five temperature hypotheses come from a single model; true ensemble diversity (multiple ASR systems) is not explored.
  • No adaptive per-segment strategy selection; the best strategy must be chosen globally per domain.
  • Dependency on GPT-4o for best performance; GPT-4o-mini shows substantially lower gains.

Relevance to Harnesses / Meta-Harnesses

RECOVER is a textbook example of a domain-specific agent harness: it decomposes a multi-step pipeline into discrete tool functions (Fuse → Propose → Verify), orchestrated by a controlling agent, with each tool having a clean interface and deterministic guardrails preventing downstream corruption. The four fusion strategies implemented as swappable modules within a shared pipeline mirror the strategy-routing pattern found in meta-harnesses, where a controller selects sub-strategies at runtime. The constrained LLM editing pattern—where the LLM proposes and a deterministic layer verifies—is a reusable harness primitive directly applicable to any tool-use pipeline where LLM hallucination must be suppressed. Future work explicitly mentions multi-agent architectures and adaptive per-segment strategy selection, signaling movement toward a fuller meta-harness with dynamic orchestration.