MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

🕒 Published (v1): 2026-07-20 14:38 UTC · Source: Arxiv · Venue: TMLR · link

Why this paper was selected

TMLR; Tresp group; multi-agent debate + RL trains compact models to reason efficiently

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MADA-RL is a post-training framework that specializes compact LLMs (≤4B parameters) into generator and critic roles via LoRA adapters, training critics with a counterfactual advantage that rewards them for correcting generator errors rather than reproducing correct answers. Applied to DeepSeek-R1-Distill-Qwen-1.5B, it achieves +2.0 accuracy points over the base model (\(p < 0.001\)) using 16× fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy–trainable-parameter Pareto front across five math benchmarks.

Problem

Compact LLMs (≤4B params) underperform on multi-step reasoning, yet existing RL-based fine-tuning approaches suffer from three compounding problems: they typically require full-model fine-tuning (high memory/compute), they use static mean-reward normalization which produces unstable or imprecise credit assignment, and they leave test-time deliberation (multi-agent debate) structurally disconnected from the training objective.

Method

MADA-RL is a two-stage GRPO-based framework operating over a shared base model with per-agent LoRA adapters (\(18.5 \times 10^6\) trainable parameters each):

Stage 1 — Generator training. \(N\) generators are trained independently on disjoint data subsets with a composite reward: $\(R_G(\hat{y}, y) = 2\,R_{\text{acc}}(\hat{y}, y) + R_{\text{len}}(\hat{y}, y)\)$ where \(R_{\text{acc}} = \mathbf{1}(\hat{y} \equiv y)\) (symbolic equivalence via LaTeX→SymPy) and \(R_{\text{len}}\) penalizes token count in \([-0.5, +0.5]\), capped at zero for incorrect outputs to prevent reward hacking.

Stage 2 — Critic training. A debate-aware dataset \(D'\) is constructed by appending generator outputs to each question and recording per-instance generator accuracy \(\text{acc}_G\). Critics are trained with the counterfactual advantage: $\(A_C(\hat{y}, y, \text{acc}_G) = 2\bigl(R_{\text{acc}}(\hat{y}, y) - \text{acc}_G\bigr) + R_{\text{len}}(\hat{y}, y)\)$ This dynamic, role-conditioned baseline makes the critic's advantage positive only when it outperforms the generator consensus on that instance, directly incentivizing corrective rather than imitative behavior. No value model, replay buffer, or external verifier is required.

Inference. Three generators produce initial answers in parallel (round 1); three critics each receive the concatenated question + all generator answers and revise in subsequent rounds. Final accuracy is computed on critic outputs only.

Key Contributions

• Counterfactual critic advantage: a role-conditioned baseline for multi-agent RL that sharpens credit assignment by conditioning on per-instance generator accuracy
• MADA-RL framework: full two-stage LoRA-based training pipeline for role-specialized generator/critic agents from a shared compact base model
• Ablation isolating mechanism: controlled study demonstrating that gains arise from the critic's learned corrective behavior (highest critic improvement rate: 19.6%), not from extra test-time token budget alone
• Comprehensive benchmark study: evaluation of seven 1.5B-class community fine-tunes under a common multi-agent protocol with rigorous Welch's t-test significance reporting

Results

• MADA achieves 41.9 ± 0.3 average accuracy vs. base DeepSeek-R1-Distill-Qwen-1.5B at 39.9 ± 0.3 (+2.0 points, \(p < 0.001\)), the only parameter-efficient method with non-overlapping error bars above baseline
• Uses 110 × 10^6 total trainable parameters vs. 1.78 × 10^9 for full fine-tuning baselines — 16× fewer — and achieves the largest accuracy-gain-per-trainable-parameter ratio
• DeepScaleR-LoRA (40.5) falls significantly below MADA (\(p = 0.015\)); Still-3-LoRA (41.3) is statistically indistinguishable (\(p = 0.280\))
• MADA does not surpass full fine-tuned Still-3 (43.1, \(p = 0.020\)) or DeepScaleR (44.3, \(p < 0.001\)), which train on substantially larger corpora
• Highest critic improvement rate of any model: 19.6% (fraction of wrong generator answers corrected by critics)
• Per-question inference cost: 33,818 tokens (six agent calls over two rounds, ~10× a single forward pass)

Limitations

• Does not match DeepScaleR or Still-3; the gap is attributed to data scale, not mechanism, but this limits practical ceiling relative to resource-rich competitors
• High inference-time cost (33,818 tokens/question) makes the method unsuitable for latency-sensitive deployment; efficiency claims are strictly about training parameters, not inference compute
• Evaluation restricted to mathematical reasoning benchmarks; generalization to other reasoning domains (code, science, logic) is not demonstrated
• Omits inter-round summarization (to keep compute low), which prior work (Subramaniam et al., 2025) shows yields modest gains; this may understate achievable accuracy with the same training signal
• Role-specialization via disjoint training subsets means each agent sees only 7,500 of 29,925 available problems, potentially limiting individual agent quality

Relevance to Agentic AI / LLM Agents

MADA-RL directly addresses a core challenge in multi-agent LLM systems: how to train structurally distinct roles (generator vs. critic) so that each agent's learned behavior matches its function in a collaborative pipeline, rather than collapsing to imitation. The counterfactual advantage formalizes a credit assignment principle that is broadly applicable wherever agents must improve on each other's outputs, a pattern central to debate, critique, and iterative refinement architectures. The LoRA-based specialization from a shared base model is a practical template for deploying role-differentiated agent fleets without separate model weights. For researchers tracking inference-time scaling and multi-agent debate, the paper provides one of the few controlled ablations confirming that training signal — not token budget alone — determines whether a critic actually corrects rather than echoes.