Skip to content

R-Debater: Retrieval-Augmented Debate Generation through Argumentative Memory

🕒 Published (v1): 2025-12-31 07:33 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

R-Debater is a retrieval-augmented, multi-agent framework for multi-turn debate generation that operationalizes "argumentative memory" — retrieving and recontextualizing prior debate moves and evidence to maintain stance fidelity across turns. It combines a structured debate knowledge base (annotated with Walton's argumentation schemes), a logic-analysis agent, and an iterative generation-verification loop. On the ORCHID Chinese debate dataset, it outperforms LLM and RAG baselines on both single-turn and multi-turn metrics, with 76.32% human preference rate.

Problem

LLMs generate fluent but stance-shallow debate outputs: they hallucinate evidence, drift from their assigned position across turns, and fail to exploit prior argumentative moves. Existing RAG systems optimize for short factual QA and cannot reconcile retrieved material with the adversarial, multi-turn structure of competitive debate. Agentic frameworks (e.g., Agent4Debate) lack explicit evidence grounding and argumentative memory across turns.

Method

R-Debater operates via three sequential pipelines:

  1. Database Construction — 1,134 ORCHID debate transcripts are parsed into utterance-level units; a pre-trained encoder produces dense embeddings; a Scheme Annotation Agent assigns seven Walton argumentation schemes (example-based, expert opinion, causal, analogical, etc.) per utterance using few-shot LLM prompting; a Scoring Agent rates each scheme {poor, general, good, excellent}, yielding enriched records (u_i, e_i, S_i, Score_i).

  2. Debate Data Retrieval — Two parallel agents operate given the partial debate history: (a) a Logic & Summarization Agent that converts opponent utterances to pseudo-first-order predicates via prompted LLM (f_pred), chains them into a reasoning summary (f_infer), then flags logical flaws as natural-language control signals L (f_logic, inspired by SymbCoT); (b) a Keyword Extraction Agent that performs coarse keyword-based retrieval then re-ranks by cosine similarity to obtain top-k exemplars, filtering to only schemes scoring above 2.0 average.

  3. Debate Generation — An initial utterance is generated conditioned on history, logical flaw signals L, and retrieved exemplars. A Debate Summarization Agent produces a structured debate overview; a Judgement Agent scores stance faithfulness, argumentative relevance, and scheme compliance, emitting a binary pass/fail and textual feedback. A Text Modification Agent revises on failure; this loop repeats until the utterance passes all criteria.

Key Contributions

  • First systematic study of multi-turn debate statement generation with LLMs under realistic adversarial settings.
  • R-Debater framework integrating Walton argumentation schemes, symbolic-style opponent logic analysis, RAG over an annotated debate knowledge base, and iterative multi-agent verification.
  • Evaluation on ORCHID with 1,000-item retrieval corpus and 32-debate held-out set spanning 7 domains, using both InspireScore (utterance-level) and Debatrix (debate-level) metrics.
  • Expert-alignment study showing Scheme Annotation Agent achieves Cohen's κ ≈ 0.72 and Scoring Agent achieves Pearson r = 0.64 vs. human experts.

Results

  • Single-turn (InspireScore, Table 1): R-Debater (GPT-4o) scores 0.822 vs. Agent4Debate 0.783, Naive RAG 0.770, LLM 0.641; Logic dimension approaches near-perfect (≈0.997 across all backbone models).
  • Multi-turn Debatrix (Table 2): R-Debater vs. Agent4Debate on GPT-4o: Overall 1.23 vs. 0.77; Source 1.13 vs. 0.87; Argument 1.26 vs. 0.74. DeepSeek-V3: Overall 1.25 vs. 0.75.
  • Human evaluation (Table 6): R-Debater preferred by 76.32% of 20 expert debaters vs. Agent4Debate 15.79%, LLM 7.89%, Naive RAG 0%.
  • Ablation (Table 5, gpt-4o-mini): Removing Argumentation Scheme causes the largest drop (0.831→0.528 InspireScore); removing Logic & Summarization Agent: 0.831→0.761; removing Optimization Module: 0.831→0.776.
  • Scheme Annotation Agent: Jaccard 0.737, Precision 0.823, Hamming Loss 0.129, Cohen's κ micro 0.723, Krippendorff's α 0.723.

Limitations

  • Dependent on pretrained LLMs; retrieval only partially mitigates hallucination when retrieved evidence conflicts with model priors.
  • Multi-agent architecture increases latency and token cost; not suitable for real-time or large-scale deployment.
  • Knowledge base coverage and recency directly bound performance; stale or domain-narrow corpora degrade argument quality.
  • Evaluated exclusively on ORCHID (Chinese formal debate); generalization to multilingual, informal, or cross-domain debate is unvalidated.
  • Evaluation metrics (InspireScore, Debatrix) retain subjective components; human evaluation is small-scale (20 annotators, 32 debates).

Relevance to Harnesses / Meta-Harnesses

R-Debater is a concrete example of a domain-specific multi-agent harness where the orchestration logic — retrieval, symbolic analysis, generation, iterative verification — is hard-coded into a pipeline with well-defined agent roles and inter-agent communication contracts, rather than being learned end-to-end. The iterative judge→revise loop (Judgement Agent + Text Modification Agent) is a canonical self-repair sub-harness pattern directly applicable to quality-control loops in general meta-harnesses. The separation of data construction, retrieval, and generation into composable pipelines illustrates how domain knowledge (argumentation schemes, stance labels) can be encoded into harness infrastructure rather than left to model priors. The expert-alignment evaluation of internal agents is notable methodology for validating the trustworthiness of harness sub-components independently of end-task metrics.