Skip to content

Learning “Partner-Aware” Collaborators in Multi-Party Collaboration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Standard RLHF/DPO-trained LLM agents are provably suboptimal in multi-party settings because they treat all interventions as benign context rather than evaluating their causal utility. This paper introduces Interruptible Collaborative Roleplayer (ICR), a PPO-based training algorithm with a counterfactual KL-divergence regularization term that teaches agents to selectively incorporate helpful interventions while resisting misleading ones. ICR consistently outperforms DPO, IPO, PPO, and behavior cloning baselines on collaborative reasoning tasks.

Problem

LLM agents deployed as collaborators fail when paired with an intervention agent whose suggestions vary in quality. Standard preference alignment (DPO/IPO) and RL (PPO) methods are Bellman-optimal in the underlying MDP but are provably suboptimal in the Modified-Action MDP (MAMDP) setting: they cannot distinguish whether a novel intervention genuinely advances task success or introduces misleading noise. This causes agents to reflexively adopt suggestions, degrading both task accuracy and group common ground (CG) convergence.

Method

ICR extends standard PPO with a counterfactual invariance objective. A counterfactual state s_CF is constructed by prepending a prompt prefix that instructs the collaborator that the current intervention will not improve utility. The training objective adds a KL-divergence penalty between the factual policy π_C(·|s, a^I) and the counterfactual policy π_C^CF(·|s^CF, a^I):

J*(θ_C) = E[Σ γ^t U_task] - λ_H · KL(π_C ∥ π_Ref) - λ_Intent · KL(π_C(·|s,a^I) ∥ π_C^CF(·|s^CF,a^I))

The counterfactual KL is computed token-by-token by reusing the factual rollout tokens and running one extra forward pass with the counterfactual prefix (stop-gradient applied), adding negligible compute over standard PPO. The framework is formalized as a MAMDP M = (S, A^C, A^I, P_S, P_A, R, γ) where the intervention policy π_I is fixed and the collaborator π_C must learn discriminative evaluation of interventions. The reference policy π_Ref is behavior-cloned from GPT-4o expert demonstrations. Models are trained on Meta-Llama-3-8B-Instruct.

Key Contributions

  • Formal proof (Theorem 3.2) that preference-aligned policies (DPO/IPO) and standard RL are suboptimal in MAMDPs even when Bellman-optimal in the underlying MDP.
  • ICR algorithm: counterfactual invariance KL-regularization within PPO that is computationally cheap (one extra forward pass per rollout sample).
  • Theoretical bound connecting counterfactual KL divergence to the MAMDP suboptimality gap.
  • Empirical validation on DeliData Wason Card Selection and Weights Task in both full-press (natural language) and no-press (discrete action) settings.

Results

  • Weights Task, full-press: ICR ACC = 14.06 vs. DPO 9.56 (next best) — 47% improvement.
  • Weights Task, no-press: ICR ACC = 10.87 vs. PPO 7.81 — 39% improvement.
  • DeliData, full-press accuracy: ICR 0.88 vs. DPO 0.82 (+7.3%), BC 0.71 (+24%).
  • DeliData, full-press CG gain: ICR 3.35 vs. PPO 2.94 (+14%); BC yields −0.13 (reduces diversity).
  • DeliData, no-press: ICR ACC = 0.85, CG = 3.18, both best among all baselines.
  • For inequality and ordering propositions in Weights Task, ICR achieves ~300% higher CG than competitors in later turns.
  • ICR (8B) trained model performs comparably to GPT-4o acting as both agents.
  • Optimal regularization: λ_Intent = 0.2; too low (0.01) loses discrimination, too high (1.0) sacrifices task utility.

Limitations

  • All trained models are 8B-scale (Meta-Llama-3-8B-Instruct) due to compute constraints; behavior at larger scales is untested.
  • Counterfactual prompts are hand-crafted and task-specific; generalization of prompt construction to arbitrary domains is unvalidated.
  • The intervention agent is fixed (GPT-4o) at evaluation; robustness to diverse or adversarial intervention agents is not systematically tested.
  • Training data is synthetic (GPT-4o roleplaying both agents), limiting exposure to genuine human collaborative dynamics.
  • Only two task domains evaluated; transfer to real-world, open-ended agentic settings remains future work.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core failure mode in multi-agent LLM systems: agents that blindly incorporate context from other agents without causal evaluation, a form of "sycophancy toward peers." The MAMDP formalization and ICR training algorithm offer a principled mechanism for building safely interruptible agents — a key property for human-AI teaming where the human (or a supervisor agent) may issue corrections of variable reliability. The counterfactual regularization approach is lightweight and composable with standard PPO pipelines, making it practically relevant for any agentic pipeline involving a planner-critic or orchestrator-executor architecture where one agent's outputs condition another's actions. The theoretical result that standard RLHF fails in MAMDP settings is a fundamental negative result for multi-agent alignment.