Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving

🕒 Published (v1): 2025-11-26 17:01 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

End-to-end autonomous driving models fail in closed-loop settings due to observation mismatch (sim-to-real sensor gap) and objective mismatch (imitation vs. reward maximization). MPA addresses both by generating counterfactual trajectories via a 3DGS world model, then training a diffusion-based policy adapter and a multi-step Q-value model to guide inference-time trajectory selection. On nuScenes/HUGSIM, MPA raises HDScore from ~47 (best base) to ~67 in-domain, with strong out-of-domain and safety-critical generalization.

Problem

Pretrained E2E driving agents (UniAD, VAD, LTF) perform well in open-loop evaluation but degrade sharply in closed-loop rollouts due to: (1) observation mismatch — distribution shift between offline training sensors and closed-loop rendered observations; (2) objective mismatch — offline imitation learning (MSE) does not optimize cumulative closed-loop reward. Prior RL-based fine-tuning (e.g., RAD/PPO) is expensive and leaves value functions unused at inference. No prior work curates counterfactual data inside a photorealistic E2E simulator.

Method

MPA is a post-hoc adaptation framework applied to a frozen pretrained E2E agent:

1. Counterfactual Data Generation: Rollout the pretrained policy in a 3DGS-based simulator (HUGSIM), augmenting trajectories with rotation (±10°), warping (0.1–1.0), and Gaussian noise. Use beam search to retain high-reward candidates. Filter by distance-from-reference and minimum reward thresholds. Outputs (state, action, observation, reward) tuples.

2. Diffusion-Based Policy Adapter: A 1D U-Net latent diffusion model trained to predict residual trajectories Δa = a* − a_base in N modes, conditioned on scene encoding, ego history, and the base policy's predicted trajectory. Uses DDIM sampling at inference. The base policy remains frozen.

3. Multi-Step Q-Value Model: Four independent Q-functions trained on counterfactual data — Q_route, Q_dist, Q_collision, Q_speed — each predicting discounted cumulative returns under partial observability. Weighted sum forms the final Q. At inference, the adapter proposes multiple trajectory candidates; the Q-value model selects the highest-utility one.

Key Contributions

• Root-cause analysis decomposing closed-loop degradation into observation mismatch and objective mismatch, with empirical validation that 3DGS reconstruction fidelity is sufficient to address observation mismatch.
• A geometry-consistent 3DGS counterfactual data curation pipeline with beam-search filtering, producing diverse yet realistic behavioral trajectories inside a photorealistic simulator.
• A diffusion residual adapter + multi-principle Q-value model combination enabling inference-time scaling for E2E driving — claimed first such integration for E2E agents.
• Ablation showing longer counterfactual rollout horizons and more adapter modes monotonically improve performance.

Results

All metrics ×100, higher is better. Best base policy (LTF) vs. best MPA variant on HUGSIM/nuScenes:

In-domain (70 scenes, Singapore): - LTF: HDScore 46.7, RC 65.2 → MPA (VAD): HDScore 67.0, RC 94.9 - MPA (UniAD): HDScore 66.4, RC 93.6 vs. UniAD baseline: HDScore 19.4, RC 39.4

Unseen nominal scenes (70 Boston scenes, out-of-domain): - LTF: HDScore 41.9, RC 63.3 → MPA (VAD): HDScore 61.2, RC 90.9 - Diffusion baseline (counterfactual-trained): HDScore 35.1, RC 57.9 — MPA remains superior

Safety-critical scenes (10 scenes, adversarial agents): - LTF: HDScore 24.2, RC 35.1 → MPA (VAD): HDScore 74.7, RC 96.6 (+~207% HDScore) - MPA (UniAD): HDScore 70.4, RC 95.1

Ablation (Q-value components): Removing Q_route collapses RC to ~7 (unseen) / ~5 (safety-critical). All four Q components together with the adapter are required for peak HDScore.

Limitations

• 3DGS rendering fidelity degrades for trajectories deviating significantly from the reference; the framework assumes constrained behavioral diversity.
• Value modeling is decoupled from policy optimization (no online RL loop); joint training could yield better alignment.
• Evaluated only on nuScenes; generalization to diverse datasets (e.g., Waymo, nuPlan) is unverified.
• Safety-critical evaluation set is small (10 scenes), limiting statistical conclusions.
• Inference cost scales with number of sampled trajectory candidates (20 used in main experiments).

Relevance to Agentic AI / LLM Agents

MPA instantiates a core agentic pattern — inference-time scaling via a learned value model — that is directly analogous to LLM test-time compute methods (Best-of-N sampling guided by a reward/process model). The counterfactual data generation pipeline mirrors agentic self-play and world-model rollout strategies used in LLM agent training (e.g., model-based RL with learned world models). The framework's separation of a frozen base policy, a lightweight residual adapter, and a multi-principle value critic is a transferable architecture for any agent domain where offline-trained policies must be adapted to online, reward-maximizing deployment without full retraining. The paper also demonstrates that inference-time value guidance can substantially close the sim-to-real and train-to-deploy gaps — a recurring challenge in embodied and robotic LLM agents.