Large Language Model-Based Reward Design for Deep Reinforcement Learning-Driven Autonomous Cyber Defense

🕒 Published (v1): 2025-11-20 15:54 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes using an LLM (Claude Sonnet 4) to automatically generate reward structures for deep reinforcement learning agents in autonomous cyber defense, replacing the cognitively expensive manual reward engineering by human subject matter experts. Multiple attack and defense agent "personas" (aggressive/stealthy attacker, proactive defender variants) are instantiated via prompt-based behavioral specifications, and the resulting reward structures train PPO-based blue agents in the Cyberwheel simulation environment. Results show LLM-guided proactive defenders outperform SME-baseline defenders against diverse adversarial behaviors.

Problem

Designing reward functions for DRL-based autonomous cyber defense agents is cognitively burdensome for cybersecurity SMEs: as network state spaces grow and agent action complexity increases, hand-crafting rewards that faithfully capture heterogeneous attacker/defender behavioral traits becomes intractable. No systematic method exists to translate qualitative behavioral characterizations (e.g., "stealthy attacker") into quantitative reward signals.

Method

The system follows the LLM4RL paradigm: Claude Sonnet 4 is given context about the Cyberwheel simulation environment (network topology, action spaces, baseline YAML reward files, MITRE ATT&CK/ART kill-chain structure) and prompted with qualitative behavioral constraints per agent persona. The LLM outputs numerical reward tables (immediate and recurring rewards per action) for each persona combination. These reward structures replace or augment SME-provided baselines and are used to train PPO-based blue agents with an actor-critic architecture. Nine experimental conditions cover all combinations of three blue agent policies (baseline, proactive-v1, proactive-v2) against three red agent personas (baseline, stealthy, aggressive) in a 15-host Cyberwheel network. Evaluation uses 50 episodes × 100 steps, measuring the complementary empirical CDF of time-to-first-impact.

Key Contributions

• LLM-driven reward design pipeline that translates qualitative agent behavioral traits into quantitative reward structures without requiring SME reward engineering for each persona.
• Systematic persona taxonomy: aggressive attacker (high immediate rewards for lateral movement/privilege escalation/impact), stealthy attacker (low immediate but high recurring rewards encoding gradual escalation), and two proactive defender variants with distinct cost structures for decoy deployment.
• Empirical evaluation across 9 persona-pair combinations in Cyberwheel, demonstrating that the LLM-suggested proactive-v2 policy yields the best 95th-percentile delay against both stealthy and aggressive attackers.
• Recommendation for a mixed defense strategy: deploy baseline policy against a baseline attacker, switch to proactive-v2 against stealthy or aggressive attackers.

Results

• Against baseline red agent: baseline blue agent achieves best 95th-percentile time-to-first-impact of 15 steps (vs. 11 steps for both proactive variants).
• Against stealthy red agent: proactive-v2 achieves 18 steps (vs. 13 for proactive-v1 and baseline).
• Against aggressive red agent: proactive-v2 achieves 15 steps (vs. 12 for proactive-v1, 10 for baseline).
• Proactive-v1 blue agent deploys decoys at 39.5% action frequency against aggressive red vs. 35.1% against stealthy; proactive-v2 inverts this (34.2% vs. 22.4%), showing reward-driven behavioral differentiation.

Limitations

• Red agent is heuristic-based (ART kill-chain), not a learned adversary; results do not generalize to adaptive attackers.
• Small 15-host network; scalability to larger, more realistic topologies is unverified.
• Reward design is one-shot from the LLM; the proposed LLM-in-the-loop iterative refinement is described as future work, not evaluated.
• No ablation isolating LLM reward quality from PPO training variance; baseline rewards are SME-provided, not LLM-provided, making the comparison partially confounded.
• Defense action space is minimal (deploy decoy / remove decoy / do nothing), limiting ecological validity.

Relevance to Agentic AI / LLM Agents

This work is a concrete instance of LLM4RL — using an LLM as a reward engineer rather than as a planner or policy — and demonstrates that language-specified behavioral constraints can be reliably translated to numerical reward signals sufficient to shape DRL policy behavior. For researchers tracking agentic AI, it illustrates a design pattern where LLM agents act as meta-controllers that configure the learning environment for downstream RL agents, decoupling domain expertise encoding from policy optimization. The Cyberwheel + MITRE ATT&CK framing also positions this work within the growing literature on autonomous cyber agents operating under adversarial conditions, a domain where reward misspecification has direct safety consequences. The proposed future direction of LLM-in-the-loop iterative reward refinement would make the LLM a first-class component of the training loop, moving toward fully agentic reward co-design.