Skip to content

Adaptive Multi-Agent Response Refinement in Conversational Systems

🕒 Published (v1): 2025-11-11 14:48 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MARA (Multi-Agent Refinement with Adaptive agent selection) is a multi-agent framework that refines LLM conversational responses across three orthogonal quality dimensions—factuality, persona alignment, and coherence—using specialized agents coordinated by a learned planner. The planner dynamically generates both the subset and ordering of refining agents per query rather than applying a fixed pipeline. MARA consistently outperforms single-agent refinement baselines (Self-Refine, SPP) and multi-agent baselines (LLMvLLM, MADR, MultiDebate) across three challenging conversational benchmarks.

Problem

LLMs in multi-turn conversational settings produce responses that can simultaneously fail on factual accuracy, user-persona alignment, and turn-level coherence. Single-agent refinement (e.g., Self-Refine) suffers from confirmation bias—the agent over-commits to its initial output—and cannot holistically handle all three aspects. Existing multi-agent refinement approaches either target only factual errors (LLMvLLM, MADR) or lack explicitly scoped agent roles (MultiDebate), making them ineffective when persona and coherence issues co-occur with factual errors.

Method

MARA decomposes refinement into four components:

  1. Three specialized refining agents, each instantiated via a distinct prompt template:
  2. Fact-refining agent: corrects factual errors against provided knowledge (\(r_{\text{fact-refine}} = \text{LLM}(P_{\text{fact-refine}}(r))\)).
  3. Persona-refining agent: aligns responses with the user profile.

  4. Coherence-refining agent: ensures logical continuity with prior turns.

  5. A planner agent: given query \(q\) and initial response \(r\), produces an ordered sequence \(s_{\text{planner}} = \text{LLM}(P_{\text{planner}}(q, r))\) specifying which subset of agents to invoke and in what order, plus justifications for each decision.

  6. Sequential chained execution: agents refine one after another in the planner-determined order, each receiving the previous agent's output and the planner's justification as context.

The design contrasts with two ablated communication strategies: simultaneous (all agents run in parallel, merged by a finalizer) and fixed-order sequential (always applies all three agents in a fixed sequence). All agents are unsupervised LLMs with no additional training; the primary backbone is Claude Sonnet 3, with Claude Sonnet 3.5 specifically assigned to the fact-refining agent for improved grounding.

Key Contributions

  • Three-role specialization of refining agents (fact, persona, coherence) explicitly tailored to the failure modes of multi-turn conversational LLMs.
  • Planner agent that dynamically selects both the set and sequence of refiners per query, producing justifications that guide downstream agents.
  • Empirical demonstration that adaptive agent selection outperforms random sequencing, while an oracle (ideal) planner sets a performance ceiling that motivates further planner improvement.
  • Evaluation across five datasets (PersonaChat, INSCIT, FoCus, PRODIGy, Ubuntu Dialogue Corpus) using G-Eval and human evaluation with Spearman's \(\rho\) correlation.

Results

  • FoCus (persona + factual): MARA overall 74.51 vs. best baseline SPP 60.47; groundedness 0.65 vs. 0.51 (No Refine 0.49).
  • INSCIT (factual only): MARA overall 72.61 vs. best baseline MultiDebate 61.70.
  • PersonaChat (persona only): MARA overall 62.00 vs. Self-Refine 58.41.
  • PRODIGy (role-play, GPT-4o-mini backbone): MARA overall 63.00 vs. best baseline No Refine 52.50.
  • Planner vs. random agent assignment (FoCus subset): MARA 2.54/0.58/2.07/2.76 (Coh/Grd/Nat/Eng) vs. random 2.42/0.47/1.95/2.64; ideal (oracle) planner reaches 2.88/0.80/2.58/2.93.
  • Human evaluation on FoCus: MARA overall 82.88 vs. No Refine 65.50 vs. Self-Refine 44.38; Spearman's \(\rho\) with G-Eval ranges 0.35–0.58 across metrics.

Limitations

  • Planner agent falls well short of the oracle ideal planner (e.g., groundedness 0.58 vs. 0.80 on FoCus subset), indicating significant room for improvement in planning quality.
  • Brute-force ideal-planner evaluation was only feasible on 20 conversations due to computational cost, limiting statistical confidence.
  • Framework uses unsupervised LLMs with no task-specific training; performance is bounded by the quality of prompt engineering.
  • The fact-refining agent relies on externally provided knowledge; no retrieval mechanism is built in, so it cannot handle queries where relevant facts are absent from context.
  • All experiments sample only 100 conversations per dataset (~500–670 queries); scale may affect generalizability.

Relevance to Harnesses / Meta-Harnesses

MARA is a concrete instance of a meta-harness pattern: a planner agent that orchestrates a pool of specialized sub-agents, dynamically deciding which sub-agents to invoke and in what order based on per-query analysis. This is functionally analogous to a meta-harness that routes tasks through a configurable agent pipeline rather than a fixed workflow. The planner-as-coordinator abstraction—where the planner emits a structured execution plan (ordered agent sequence + justifications) consumed by downstream agents—is directly transferable to general-purpose harness design. The gap between the current planner and the oracle upper bound quantifies the value of better meta-level planning, motivating learned or search-based orchestration strategies relevant to any harness that dynamically composes sub-agents.