Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Continuous Soft Actor-Critic (CSAC/CMASAC), an off-policy deep RL algorithm grounded in stochastic control theory that operates natively in continuous time and space. Unlike standard discrete-time algorithms (SAC, DQN, DDPG), CSAC replaces TD-error minimization with martingale orthogonality conditions, making it robust to changes in the environment's time discretization step δt. The framework extends to multi-agent settings (both cooperative and competitive) and derives principled hyperparameter scaling laws as δt → 0.

Problem

Standard DRL algorithms trained under discrete-time frameworks (SAC, DQN, DDPG, MADDPG) degrade when the time step δt is changed, especially as δt → 0 in stochastic environments. The root cause: the mean squared temporal difference error (MSTDE) used for policy evaluation does not converge to zero at the true value function in stochastic continuous-time settings—instead it converges to the quadratic variation ⟨M⟩_T ≠ 0—invalidating gradient-based learning. No prior work addressed this for stochastic environments or for finite multi-agent continuous-time systems.

Method

Policy Evaluation via Martingale Orthogonality: The true value function J(t,x;π) defines a square-integrable martingale M_s. Instead of minimizing MSTDE, CSAC enforces the martingale orthogonality condition: E[∫ξ_t dJ^θ(t,X^π̃_t;π^φ) + r·dt − λ ln π^φ·dt − βJ^θ·dt] = 0 for all square-integrable test functions ξ_t. This is an off-policy condition: the behavior policy π̃ generates trajectories, while the target policy π^φ is evaluated.

Policy Gradient: Policy updates use a continuous-time policy gradient theorem (Theorems 3–4) that extends Jia & Zhou (2022b) to stochastic off-policy settings, enabling gradient backpropagation through the parameterized Gaussian policy.

Hyperparameter Scaling Laws (Theorem 5): For the discrete approximation to converge to the continuous-time limit, learning rates must scale as η_δt = α·δt (β=1). Additional scaling: discount γ_δt = e^{-βδt}, temperature λ_δt = λ·δt, reward r_δt = r·δt. The martingale difference δM^θ = O(δt) is normalized by δt to prevent vanishing gradients.

Multi-Agent Extension (CMASAC): Uses Centralized Training with Decentralized Execution (CTDE); agent value functions satisfy separate martingale orthogonality conditions (Theorem 2); agents update via alternating gradient steps.

Key Contributions

• First finite-agent continuous-time MARL algorithm for stochastic environments, covering both competitive and cooperative settings.
• Formal proof that MSTDE cannot converge to the true value function in stochastic continuous-time settings; martingale orthogonality condition proposed as the correct alternative.
• Off-policy learning algorithm: behavior policy π̃ ≠ target policy π^φ, enabling replay buffer reuse without importance sampling weights (via Gaussian policy reparameterization).
• Principled hyperparameter scaling laws (learning rate, discount, temperature, reward) as δt varies, with convergence proof.
• Empirical validation on VMAS benchmark against MASAC, MADDPG, MAPPO, IQL, QMIX.

Results

• CMASAC robustness (Navigation): IQM = 0.96 at δt=0.1 vs. IQM = 0.93 at δt=0.01 — near-stable performance across a 10× change in δt.
• CMASAC robustness (Sampling): IQM = 0.84 at δt=0.1 vs. IQM = 0.92 at δt=0.01 — performance actually improves at finer discretization.
• MASAC degradation (Sampling): IQM drops from 0.66 (δt=0.1) to 0.43 (δt=0.01).
• MASAC degradation (Navigation, Table 5): IQM = 0.49 at δt=0.01 (near-random), vs. 0.89 at δt=0.1.
• MAPPO at δt=0.01 (Navigation, Table 5): IQM = 0.87; IQL: 0.92; QMIX: 0.74 — all inferior to CMASAC's 0.93.
• MADDPG: More robust than SAC-based methods but still degrades; IQM ~0.90 in navigation at δt=0.01 (Table 5).

Limitations

• Experimental validation restricted to selected VMAS tasks (navigation and sampling) due to computational constraints; broader benchmark coverage is absent.
• The method targets continuous state-action spaces and is not applicable to discrete-valued action spaces.
• Multi-agent theory (convergence, equilibrium guarantees) is underdeveloped; the paper is explicitly exploratory for the MARL setting.
• Hyperparameter tuning uses defaults from BenchMARL rather than optimal settings for CSAC, potentially underrepresenting its peak performance.

Relevance to Agentic AI / LLM Agents

While focused on continuous-control RL rather than LLM-based agents, this work is directly relevant to any agentic system that uses RL fine-tuning or policy optimization in real-world (physical or simulated) environments where the time granularity of feedback varies—e.g., robotics, autonomous driving, and real-time decision systems. The principled hyperparameter scaling and martingale-based policy evaluation address a practical failure mode that arises when RL-trained agents are deployed at different inference frequencies than their training regime. The MARL extension is relevant to multi-agent coordination problems that underlie cooperative LLM-agent pipelines requiring continuous-time action selection.