Skip to content

Collaborative Reasoner: Self-Improving Social Agents with Synthetic Conversations

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Coral is a framework from Meta FAIR for evaluating and training collaborative reasoning in LLMs via multi-turn free-form conversations. The authors show frontier models exhibit pathological agreeableness (>90% agreement rate) that degrades collaborative performance, then fix this with a self-play DPO pipeline using synthetic tree-sampled conversations. Trained collaborative reasoners outperform equivalent single-agent CoT fine-tuning by up to 16.7% absolute and generalize across collaborators and domains.

Problem

LLMs are trained and evaluated almost exclusively in single-turn settings, leaving a gap in social skills required for multi-agent collaboration: assertiveness (maintaining a correct belief under partner pressure) and persuasion (changing a wrong partner's belief). Current RLHF post-training induces excessive agreeableness—models capitulate to incorrect partners, yielding high agreement rate but low agreement correctness. Collaborative multi-turn training data is scarce and expensive to collect at scale.

Method

Coral framework formalizes two-agent, free-form conversational reasoning: agents alternate turns until agreement is reached or 20 turns elapse; correctness is measured as agreement correctness (both agents converge on the gold answer). Belief states are extracted after every turn via an LLM judge to enable scalable, annotation-free social metrics (agreement, assertiveness, persuasiveness).

Self-improvement pipeline (three steps): 1. Tree sampling: for each conversation prefix, sample d responses per turn, randomly continue one branch, and retain sibling nodes as preference candidates. Five independent conversation trees per problem diversify coverage. 2. Belief filtering: label each candidate turn as positive (belief matches gold) or negative (incorrect/unsure); cap preference pairs per problem/turn to prevent easy-example bias. 3. Turn-level DPO: train on (conversation prefix, chosen turn, rejected turn) triples; iterative DPO yielded no additional benefit.

Matrix, a purpose-built serving framework using vLLM/SGLang backends, Ray for autoscaling/load balancing, and gRPC for throughput, enables thousands of parallel conversations at up to 1.87Ă— the speed of best open-source alternatives.

Key Contributions

  • Coral benchmark covering six tasks (MATH, MMLU-Pro, GPQA, ExploreToM, Hi-ToM, MBPP-CR) with agreement correctness and turn-level social metrics
  • Empirical demonstration that frontier LLMs (including O1, GPT-4o, Gemini) are inconsistently or negatively impacted by collaboration compared to single-agent CoT
  • Self-play tree-sampling + belief-filtered DPO pipeline for scalable synthetic conversational training data
  • Matrix: open-source scalable multi-agent data generation infrastructure
  • Demonstration that collaborative training also improves single-agent CoT performance

Results

  • Llama-3.1-8B: Coral+DPO improves collaboration 11.2–29.5% absolute on MBPP-CR, MMLU-Pro, ExploreToM vs. base; outperforms CoT+DPO by up to 14.7%
  • Llama-3.1-70B: Coral+DPO outperforms CoT+DPO by up to 16.7% absolute; reaches 95.0% agreement correctness on ExploreToM vs. 72.9% for base model in Coral setting
  • Qwen-2.5-7B and Ministral-8B: consistent gains of 2.1–7.2% on MATH, 3.1–30.5% on other datasets
  • Cross-collaborator generalization: average 4.9% (MMLU-Pro) to 32.2% (ExploreToM) improvement when paired with GPT-4o, Gemini, Claude, Qwen partners
  • OOD generalization: MMLU-Pro→GPQA: +5.2% (8B), +9.2% (70B); ExploreToM→Hi-ToM: up to +9.9%
  • vs. frontier models: 70B Collaborative Reasoner rivals O1 and Gemini-2.5-Flash on MMLU-Pro (75.2 vs. 82.8/76.5) and ExploreToM (95.0 vs. 86.8/74.1)
  • Human eval: trained model wins on effective disagreement (55 vs. 16 preference) and naturalness (47 vs. 37) but loses on verbosity (22 vs. 60)

Limitations

  • Iterative DPO training found to provide no additional benefit; only one round of self-play training explored
  • Tree expansion does not continue into sibling branches (no MCTS), limiting coverage of long-horizon interaction trajectories
  • Trained models become significantly more verbose, reducing collaboration efficiency
  • Out-of-domain generalization (cross-domain rather than within-domain) not evaluated
  • All evaluations use same-model or controlled cross-model settings; robustness to highly heterogeneous real-world partners unknown
  • Data generation scale capped at hundreds of model instances/thousands QPS; untested beyond this

Relevance to Agentic AI / LLM Agents

This work directly addresses a foundational gap in multi-agent systems: agents that can engage in productive disagreement rather than sycophantically converging on wrong answers. The self-play DPO pipeline is a scalable, annotation-free recipe for injecting social reasoning capabilities into generalist LLMs, making it applicable to any multi-agent pipeline where agents must negotiate, critique, or reach consensus. The finding that collaborative training also improves single-agent CoT is significant—it suggests social/conversational training is a complementary axis to reasoning-focused fine-tuning. Matrix as an open-source infrastructure lowers the barrier to scaling synthetic multi-agent data generation for the research community.