Skip to content

MAC: A Multi-Agent Framework for Interactive User Clarification in Multi-turn Conversations

🕒 Published (v1): 2025-12-15 10:02 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAC is a multi-agent clarification framework for task-oriented dialogue that assigns ambiguity resolution responsibilities hierarchically: a supervisor agent handles domain-agnostic ambiguities while domain expert agents resolve slot-level underspecification. On MultiWOZ 2.4, this coordinated approach raises task success 7.8% over a no-clarification baseline while simultaneously reducing average dialogue turns from 6.53 to 4.86.

Problem

Multi-agent conversational systems (manager + domain experts) lack principled mechanisms for deciding when to ask clarification, which agent should ask, and what to ask. Existing clarification work targets single-agent settings; in multi-agent pipelines, naive clarification either overloads the supervisor with domain knowledge it lacks or forces users to repeat context to multiple experts.

Method

MAC instantiates a two-tier clarification architecture over a supervisor + N domain-expert setup. A taxonomy partitions ambiguity types by tier: the supervisor handles 7 domain-agnostic categories (domain ambiguity, intent ambiguity, vague goal specification, etc.) using only commonsense reasoning with no database access; each expert handles 5 domain-specific categories (parameter underspecification, value vagueness, constraint conflict, entity disambiguation, inference confirmation). At each dialogue turn, the supervisor first evaluates is_ambiguous(u); if positive it emits <clarify>q</clarify>, otherwise it routes with <route>domain</route>. The routed expert then independently evaluates ambiguity and either issues a domain-targeted clarification or executes an API call. Only one clarification is issued per turn. The entire control flow is prompt-driven with structured XML output tags; no fine-tuning is required.

Key Contributions

  • First multi-agent LLM framework explicitly designed for coordinated user clarification in task-oriented dialogue.
  • Formal two-tier ambiguity taxonomy (7 supervisor-level + 5 expert-level categories) that delineates clarification responsibility by agent role.
  • Demonstration that both levels must clarify: enabling only one tier yields intermediate gains; only the combined setup achieves the full +7.8% success / −1.67 turn improvement on MultiWOZ 2.4.
  • Model-agnostic results across GPT-4o, GPT-4o-mini, and Qwen3-235B-A22B; open-source models gain more (+7.28 pts) from structured clarification than proprietary ones (+4.68 pts).
  • 11.5% absolute improvement over prior state-of-the-art TOD system (AutoTOD) on MultiWOZ 2.4.

Results

  • MAC (both tiers) vs. no-clarification baseline: 62.3% vs. 54.5% task success (Max@5); 58.40 ± 2.10 vs. 53.72 ± 0.92 (Avg@5); 4.86 vs. 6.53 avg turns.
  • Supervisor-only clarification: 57.1% Max@5, 5.11 avg turns — better than expert-only but worse than combined.
  • Expert-only clarification: 55.6% Max@5, 5.53 avg turns — marginal gain over no-clarification.
  • vs. prior TOD systems (same evaluation protocol): MAC 58.40% vs. AutoTOD 46.90%, FNCTOD 44.40%, SimpleTOD 22.00%.
  • Multi-agent vs. single-agent with clarification: 58.4% vs. 52.4% (GPT-4o-2024-11-20).
  • Ablation: removing supervisor's ambiguity/vagueness handling causes −6.20 pts; removing expert slot/parameter blocking causes −2.18 pts.
  • Open-source: Qwen3-235B-A22B gains +7.28 pts from clarification vs. +4.68 for GPT-4o.

Limitations

  • Requires large-parameter models; the clarification capabilities have not been distilled to smaller LLMs.
  • Evaluation relies on LLM-as-a-Judge, which can hallucinate positive scores; no hybrid deterministic metric is used.
  • ~1% of conversations enter clarification loops where agent and user simulator cycle without progress, indicating brittleness in the user simulation and the termination logic.
  • Results are on a single benchmark (MultiWOZ 2.4) using a GPT-4o-based user simulator, limiting ecological validity.

Relevance to Harnesses / Meta-Harnesses

MAC directly instantiates a meta-harness pattern: a supervisor agent that orchestrates routing and first-pass ambiguity resolution, combined with specialist sub-agents that each execute a focused subtask. The structured XML control-flow tags (<clarify>, <route>, <response>) and the algorithm (Algorithm 1) constitute a lightweight harness protocol that coordinates multi-agent execution without a separate orchestration layer. The taxonomy design — pre-specifying which ambiguity types belong to which agent tier — is analogous to how meta-harnesses pre-specify which subtasks belong to which sub-agent, making the paper directly relevant to principled decomposition and delegation in agentic pipelines. The finding that open-source models benefit more from structured harness design than proprietary models is a practical result for harness builders choosing backbone LLMs.