Skip to content

LOPT: Learning Optimal Pigovian Tax in Sequential Social Dilemmas

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LOPT introduces economic externality theory into multi-agent reinforcement learning (MARL) to quantify social dilemmas numerically, then trains a centralized "tax planner" agent that levies Pigovian-style taxes/allowances on other agents' rewards to internalize externalities and align individual incentives with collective welfare. On Escape Room and Cleanup benchmarks, LOPT converges faster and achieves higher social welfare than LIO, inequity aversion, MOA, and PPO baselines.

Problem

In mixed-motive MARL, self-interested agents rationally defect even when mutual cooperation would maximize collective welfare (social dilemmas). Existing approaches (LIO, inequity aversion, curiosity modules) use hand-crafted or performance-driven reward shaping with no principled economic grounding for quantifying the magnitude of the externality each agent imposes on others—making them fragile and theoretically opaque.

Method

LOPT formalizes externality in MARL (Definition 2): the gap between the optimal joint Q-value and the counterfactual Q-value when agent i deviates from the optimal joint action. The Optimal Pigovian Tax reward shaping is then F_i(s, a_{-i}*, a_i) = Q*(s, a*) − Q(s, a_{-i}*, a_i).

Because Q* is unknown, a centralized tax planner is introduced as a separate RL agent whose observation is the global state + joint action, whose action space is per-agent tax rates θ and allowance rates δ (formulated as multiplicative percentages of each agent's local reward), and whose reward is the sum of all agents' rewards (social welfare). Theorem 2 proves that when the tax planner's policy maximizes social welfare, the resulting percentage-formulated reward shaping qualitatively equals the optimal Pigovian tax shaping. Each base agent is then trained with the shaped reward r̂_i = r_i + F_{θ,δ}^i. The tax planner is optimized via policy gradient with an entropy regularizer that enforces balance between total taxes collected and allowances distributed.

Key Contributions

  • Formal extension of economic externality theory to MARL (Definition 2 + Eq. 6) as a principled numerical measure of social dilemmas.
  • LOPT: a centralized tax planner RL agent that learns per-agent Pigovian tax/allowance rates, with theoretical guarantees (Theorems 1 & 2) that the learned shaping approximates the optimal Pigovian tax.
  • Demonstrated emergence of spontaneous labor specialization (harvesters, cleaners, part-timers) in N=5 Cleanup without explicit role assignment.
  • Scalability to larger, more complex environments (18×25 grid, N=5) where competing baselines degrade.

Results

  • Escape Room N=2: LOPT converges to optimal social welfare 8; PG fails entirely; LIO/LIO-dec achieve near-optimal but with instability and betrayal episodes.
  • Escape Room N=3: LOPT converges to optimal 9; all PG variants fail; LIO/LIO-dec show variance.
  • Escape Room step efficiency: LOPT consistently completes episodes in exactly 1 step (M lever-pulls + door open); all baselines show excess steps.
  • Cleanup N=2, 7×7: LOPT near-optimal; LIO fails to learn; AC-d competitive but doesn't scale.
  • Cleanup N=2, 10×10: LOPT remains near-optimal; IA and AC-c degrade severely; SCM, MOA, PPO partially competitive.
  • Cleanup N=5, 18×25: LOPT reaches ~1400 social welfare at 150M steps; PPO, SCM, MOA significantly lower (exact numbers visible in Figure 6(d) but not tabulated in text).

Limitations

  • The tax planner is fully centralized—requires global state and joint actions at execution time, limiting applicability to decentralized or partially observable settings where such aggregation is infeasible.
  • Theorem 2 assumes other agents' interactive influences are not considered (a simplifying assumption that may not hold in practice).
  • Theorem 1's guarantee requires that taxes occur only on timesteps where r_i ≠ 0, constraining the mechanism's scope.
  • Authors acknowledge future work is needed for a decentralized variant; current complexity scales with agent count and joint action space.
  • Evaluated only on two standard MARL social-dilemma benchmarks; generalizability to real-world mixed-motive settings (e.g., large-scale open-ended environments) is unvalidated.

Relevance to Agentic AI / LLM Agents

LOPT addresses a core challenge for multi-agent systems: how autonomous, self-interested agents can be steered toward cooperative, socially beneficial behavior without requiring full cooperative alignment by design. The tax-planner paradigm—a meta-agent that reshapes the reward landscape for peer agents—is directly analogous to oversight/governance architectures being explored in LLM multi-agent pipelines, where a coordinator or evaluator agent modulates other agents' incentives. The economic grounding (externality internalization) offers a more principled alternative to ad hoc reward shaping commonly used in agentic frameworks. The emergent role specialization in Cleanup (cleaner/harvester/part-timer) is also a concrete demonstration of division of labor arising from well-designed incentive structures, a desiderata for scalable multi-agent LLM systems.