Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning

🕒 Published (v1): 2026-07-21 14:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically extends three parameterized-action actor-critic algorithms — GAC, SAC, and TQC — to shared-experience multi-agent variants (MAGAC, MASAC, MATQC) using a common replay buffer rather than a full CTDE architecture. Evaluated on Platform-v0 and Goal-v0 benchmarks with 3-, 5-, and 10-agent configurations, MAGAC consistently outperforms its single-agent counterpart, while MASAC and MATQC yield only modest gains. Scaling beyond five agents adds computational cost without proportionate performance benefit.

Problem

Single-agent parameterized-action actor-critic methods (PAGAC, PASAC, PATQC) have been validated on hybrid discrete/continuous benchmark tasks, but their multi-agent extensions remain unexplored. Existing MARL work (MADDPG, QMIX, MAPPO) does not address parameterized action spaces, leaving an empirical gap around whether shared-experience multi-agent learning improves coordination and performance in these hybrid environments.

Method

Multiple independent actor-critic agents are instantiated within the same environment. Each agent maintains separate actor and critic networks. All agents write transitions into a shared replay buffer, from which each independently samples and updates — increasing sample diversity without a centralized critic. At each timestep, each agent produces a parameterized action (discrete choice + continuous parameters); these are combined via action fusion into a single executable action accepted by the benchmark environment. Hyperparameters are tuned via Microsoft NNI; each configuration runs 10 independent seeds for 5,000 episodes. Statistical significance is assessed with one-way ANOVA and Tukey HSD post-hoc tests.

Key Contributions

• Extension of PASAC, PAGAC, and PATQC to shared-experience multi-agent variants: MASAC, MAGAC, MATQC.
• Controlled single-agent vs. multi-agent comparison on identical environments, reward functions, and training configurations.
• Agent-scaling analysis across 3-, 5-, and 10-agent configurations.
• Statistical evaluation (ANOVA + Tukey HSD) of average evaluation return and training time across 10 runs.
• Characterization of the performance/compute trade-off as agent count increases.

Results

(Note: the paper text is truncated before numerical result tables appear; only qualitative findings from the abstract are available.) - MAGAC consistently improves over single-agent GAC on both Platform-v0 and Goal-v0. - MASAC and MATQC show comparatively modest gains over their single-agent counterparts. - Scaling from 5 to 10 agents yields limited additional performance improvement. - Training time increases substantially with agent count, particularly for MAGAC. - The 5-agent configuration represents the empirically observed performance/efficiency trade-off point.

Limitations

• No full CTDE implementation: no centralized critic over joint observations/actions, so results cannot be compared directly to MADDPG, QMIX, or MAPPO.
• Action fusion strategy is not formally specified or ablated; its effect on coordination is unquantified.
• Evaluation restricted to two benchmark environments (Platform-v0, Goal-v0); generalization to real-world parameterized tasks (e.g., actual match plan generation) is not demonstrated.
• Results section is not fully provided in the submitted text, making independent numerical verification impossible.
• No analysis of sample efficiency, only final evaluation return and wall-clock training time.
• Credit assignment within the shared-buffer framework is not addressed.

Relevance to Harnesses / Meta-Harnesses

This paper constructs a lightweight coordination harness for multiple RL agents: a shared replay buffer with action fusion acts as the glue layer that lets independently trained agents collectively drive a single environment, analogous to how a meta-harness coordinates multiple sub-agents toward a shared objective. The shared-experience architecture is a minimal, non-CTDE harness design choice — relevant to researchers studying how much coordination infrastructure is necessary to obtain multi-agent benefits without the overhead of centralized critics. The agent-scaling analysis directly informs harness designers about diminishing returns and cost cliffs when increasing parallelism beyond a threshold number of agents.