HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HMARL-CBF introduces a hierarchical multi-agent reinforcement learning framework that integrates Control Barrier Functions (CBFs) to enforce pointwise-in-time safety guarantees — constraints that hold at every timestep, not just in expectation. A high-level policy selects cooperative skills across agents; low-level policies execute them safely via differentiable QP-based CBF filters. The method achieves ≥95–99% success rates on multi-vehicle navigation tasks where all baselines fail or achieve <75%.

Problem

Existing safe MARL methods rely on Constrained MDP (CMDP) frameworks that guarantee safety only in expectation over trajectories, which is insufficient for safety-critical systems requiring per-timestep constraint satisfaction. Flat MARL policies also suffer from poor sample efficiency and scalability as agent count grows. No prior HMARL method combines hierarchical skill decomposition with formal pointwise safety guarantees in partially observable, cooperative settings.

Method

The framework decomposes the MARL problem into two levels over a Multi-Agent Semi-Markov Decision Process (MSMDP):

• High level: A centralized-training–decentralized-execution (CTDE) policy selects skills for all agents (via MAPPO or QMIX), optimizing a cooperative extrinsic reward that includes penalty terms for safety violations.
• Low level: Each agent executes its assigned skill via a parameterized Quadratic Program (QP) whose constraints are CBFs (and CLFs for goal convergence). The QP parameters are neural network outputs conditioned on the agent's local observation and active skill, trained with policy gradient through the reparameterization trick.

CBF forward-invariance theorems guarantee that the safe set is never exited during skill execution. Both levels are trained jointly from the same trajectory data, with no added sample complexity from the hierarchy. Asynchronous skill termination (τ_continue) allows agents to update independently.

Key Contributions

• HMARL-CBF: first CTDE hierarchical MARL framework with provable pointwise-in-time CBF safety guarantees across both training and deployment.
• Skill-based hierarchy with safety-constrained skill definition as a 7-tuple including initiation sets, termination sets, CBF constraints, and a CLF-assisted QP policy.
• Demonstration that hierarchical decomposition eliminates the need for a pre-specified nominal policy (required by prior RL-CBF methods).
• Convergence in ≤300k iterations vs. 1M for baselines, without additional data.
• Validated across five METADRIVE environments (merging, intersection, roundabout, bottleneck, tollgate) with both on-policy (MAPPO+PPO) and off-policy (QMIX+SAC) implementations.

Results

• Success rate: HMARL-CBF achieves 0.97–0.99 across all five environments; best baselines (CoPo) reach 0.48–0.85 on the same tasks; MAPPO-Lagrangian and MACPO score 0.00–0.34.
• Merging: HMARL-CBF on-policy 0.99±0.01 vs. CoPo 0.48±0.01, MFPO 0.43±0.09, MACPO 0.32±0.04.
• Tollgate: HMARL-CBF 0.99±0.02; CoPo 0.00, MAPPO-Lagrangian 0.00, MACPO 0.00.
• Time/energy efficiency: HMARL-CBF also leads or matches on success-weighted time and energy across all five environments.
• Ablations: Flat policy with any penalty factor fails (0% success); removing CBFs and using LQR collapses to 0–14%; random constraints dropout reduces success by 4–39 pp depending on environment; removing the cooperative high-level policy reduces success by 5–31 pp.
• Converges in ≤300k iterations vs. baselines trained for 1M iterations plus 2000+ hours of pre-training data.

Limitations

• Requires an approximate dynamics model for CBF construction (affine control-affine form); extension to fully model-free settings is not demonstrated.
• Skill set Z must be pre-defined; joint learning of skill structure and cooperative behavior is left to future work.
• Validated only in simulation (METADRIVE); real-world sim-to-real transfer is not addressed.
• CBF constraint satisfaction relies on Lipschitz continuity and known system structure; robustness under significant model mismatch is handled only theoretically via robust CBFs (not empirically evaluated).
• Scalability beyond the tested agent counts is not systematically analyzed.

Relevance to Agentic AI / LLM Agents

HMARL-CBF is a high-fidelity study of how hierarchical decomposition — separating cooperative intent selection from safe primitive execution — dramatically improves both sample efficiency and safety compliance in multi-agent settings, a structural principle directly applicable to LLM-agent orchestration (high-level planner → low-level tool executor). The formal safety layer via CBFs is a concrete instantiation of constrained agentic execution, relevant to researchers designing safety wrappers or guardrails for autonomous agents operating in physical or high-stakes environments. The skill abstraction framework (goal-conditioned, termination-aware, safety-constrained primitives) mirrors how tool-use and subagent APIs are structured in LLM agent frameworks, and the CTDE training paradigm maps cleanly onto centralized-coordinator / decentralized-actor agent architectures. For researchers tracking safe and scalable multi-agent systems, this paper provides strong empirical and theoretical grounding.