Skip to content

Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FCGrad is a gradient adjustment method for multi-agent RL in mixed-motive settings that dynamically resolves conflicts between individual and collective policy gradients while enforcing fairness. When objectives conflict (negative inner product), it projects the gradient of the lower-valued objective onto the normal plane of the other, guaranteeing monotonic non-decreasing improvement in both objectives and convergence to equal individual returns. It outperforms reward-restructuring and prior gradient-based baselines on sequential social dilemmas under α-fairness metrics.

Problem

Existing mixed-motive MARL methods (gifting, inequity aversion, social influence) promote cooperation by shaping rewards but ignore fairness with respect to task-defined extrinsic rewards — some agents end up permanently sacrificing individual returns (e.g., always cleaning waste, never collecting apples). Altruistic Gradient Adjustment (AgA), the closest prior gradient-space method, targets stability of the collective objective rather than resolving gradient conflicts, and provides no monotonic improvement or fairness guarantees.

Method

FCGrad operates per-agent on two policy gradients: g_ind = ∇V_ind(θ) and g_col = ∇V_col(θ).

  1. Conflict detection: if ⟨g_ind, g_col⟩ ≥ 0, no conflict — use weighted sum (1−β)g_ind + βg_col.
  2. Conflict resolution: if ⟨g_ind, g_col⟩ < 0, identify which objective has the lower value:
  3. If V_col ≥ V_ind (agent is disadvantaged individually → unfair), project g_ind onto the normal plane of g_col: g̃_ind = g_ind − (⟨g_col, g_ind⟩/‖g_col‖²)g_col.
  4. If V_ind > V_col, project g_col onto the normal plane of g_ind instead.
  5. The projected gradient is a valid ascent direction for its own objective while being orthogonal to (non-interfering with) the other.

In practice, each agent runs independent PPO (IPPO) with two separate value heads (individual and collective) sharing an encoder; FCGrad combines their gradients as the policy update.

Theory: Theorem 3.1 proves monotonic non-decreasing improvement in both V_ind and V_col under L-smoothness. Theorem 3.2 proves |V_ind(θ_t) − V_col(θ_t)| → 0 under Robbins–Monro step sizes and a "conflict recurrence" condition, which implies all individual returns converge to each other (individual fairness).

Key Contributions

  • FCGrad algorithm: lightweight conflict-aware gradient projection with fairness-driven priority (prioritize the disadvantaged objective).
  • Theoretical guarantees of monotonic improvement in both individual and collective objectives and convergence of the individual–collective gap to zero.
  • Demonstration that purely gradient-space methods can achieve fairness without reward redistribution/gifting.
  • Empirical superiority over six baselines (Col, Ind, IA, Weighted, PCGrad, AgA) across α-fairness metrics (Mean, GeoMean, Min) and Gini/Jain's fairness indices in three environments.

Results

  • Unfair Coin Game: FCGrad achieves the best Min (minimum individual return) and lowest Gini (0.010 vs. 0.039 for PCGrad, 0.474 for Col); green and red agents converge to nearly identical returns.
  • Cleanup (N=4, spatially asymmetric): FCGrad achieves the best GeoMean and Min; matches Col on Mean while Col causes Agent 3 to monopolize apple collection and Agent 0 to only clean waste.
  • Harvest (N=4, spatially asymmetric): FCGrad outperforms all baselines across all α values; Agents 0/1 (near apples) moderate harvesting to equalize returns with Agents 2/3.
  • Gini/Jain's index (Table 1): FCGrad ranks in the top 2 for fairness in all three environments; Jain's index 0.999 on Coins vs. 0.526 for Col.
  • Methods relying heavily on individual rewards (Ind, IA) fail to learn in Cleanup/Harvest where intertemporal cooperation is required. AgA consistently underperforms FCGrad.

Limitations

  • The "conflict recurrence" assumption required for the fairness convergence guarantee (Theorem 3.2) may not hold in practice, as it can be disrupted by the non-conflict weighting factor β.
  • Experiments use only small homogeneous-architecture agents (N=2–4); scalability to larger heterogeneous populations is untested.
  • Requires shared reward signal across agents (agents observe collective return), consistent with gifting literature but not fully decentralized.
  • β must be tuned per environment (0.5/0.7/0.8 across the three tasks), with no automated schedule.

Relevance to Agentic AI / LLM Agents

FCGrad addresses a structural challenge in multi-agent systems deployed in real-world mixed-motive settings — the same regime occupied by LLM-agent collectives (e.g., negotiation agents, multi-agent task delegation, resource allocation) where agents must cooperate without full alignment. The gradient-projection technique is policy-agnostic and applicable beyond MARL to any multi-objective optimization underlying agent training, including RLHF pipelines balancing helpfulness vs. safety. The theoretical fairness guarantee (convergence of individual returns) is directly relevant to designing equitable multi-agent systems where role specialization risks creating permanently disadvantaged agents. This work establishes that cooperation and extrinsic-reward fairness can be achieved simultaneously without reward redistribution mechanisms, which is a cleaner model for autonomous agent collectives that lack a "currency" layer.