Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

🕒 Published (v1): 2026-07-09 07:48 UTC · Source: Arxiv · link

Why this paper was selected

Multimodal LLMs as policy inspectors for RL curricula; novel VLM-as-evaluator application

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Visual Inspection of Policies (VIP) uses a Video Language Model (VLM) to directly analyze recorded RL episode videos and recommend the next curriculum task, bypassing the information loss inherent in scalar or text-only policy summaries. Applied to the StarCraft Multi-Agent Challenge (SMAC), VIP with a 7B-parameter open VLM outperforms both its text-only ablation and scalar-score-based curriculum baselines on held-out maps. The core insight is that visual episode data can reveal learning progress (e.g., near-winning strategies despite zero win rate) that aggregate signals cannot.

Problem

Existing open-ended autocurricula rely on scalar learning signals (win rate, regret estimates) or text-only LLM summaries of agent behavior to select the next training task. Both modalities fail when the reward is sparse or deceptive: a policy that has nearly discovered a winning strategy but still loses all episodes is indistinguishable from a policy with no useful structure, making it impossible to identify which tasks remain productive. This limits the curriculum's ability to correctly judge task difficulty relative to the agent's current learning progress.

Method

VIP operates as a loop: (1) train agents on the current task \(\theta_t\) using MAPPO; (2) record episode videos \(\phi_t^{\text{video}}\) and a short text summary \(\phi_t^{\text{text}}\) (win rate + list of previously selected tasks); (3) prompt a VLM with both modalities to generate a free-text next-task recommendation \(\vartheta_t\); (4) sanitize \(\vartheta_t\) by finding the highest-cosine-similarity valid task in the enumerable task space (using sinusoidal embeddings), replacing hallucinated or out-of-format outputs. The VLM used is VideoLLaMa2-7B (7B quantized, openly accessible). No new hyperparameters are introduced beyond those of the VLM and underlying RL algorithm. The method is agent-count-agnostic: any number of agents rendered in a single video is processed in one VLM call.

Key Contributions

• Introduces VIP, the first open-ended autocurriculum that feeds raw episode videos to a VLM to select curriculum tasks.
• First application of unsupervised autocurricula to SMAC, a cooperative MARL benchmark with variable agent compositions.
• Demonstrates that a lightweight open-source 7B VLM suffices, with VLM inference consuming only ~1% of total training wall-clock time.
• Proposes a sentence-similarity sanitization module (sinusoidal embeddings) to robustly map hallucinated VLM outputs to valid task specifications.
• Ablation isolating the contribution of video vs. text-only input to curriculum quality.

Results

• On held-out map 3s vs 4z: VIP achieves \(0.84 \pm 0.16\) win rate after fine-tuning; all baselines (MAPPO, Random, Text-Only, PLR-PVL, PLR-MaxMC) achieve \(0.00\)–\(0.04\).
• On held-out map 3s5z: VIP achieves \(0.76 \pm 0.20\); next best is Random at \(0.59 \pm 0.09\); Text-Only drops to \(0.21 \pm 0.30\).
• On held-out map 10m vs 11m: VIP achieves \(0.53 \pm 0.22\); Text-Only \(0.51 \pm 0.18\); both PLR variants collapse to near zero.
• All results reported with 95% confidence intervals over 5 seeds.
• VLM inference accounts for ~1% of total curriculum compute; MAPPO training dominates at ~99%.

Limitations

• Evaluated only on SMAC; generality to other multi-agent or single-agent domains is not empirically verified.
• VideoLLaMa2 does not support video batching, so each curriculum step requires a separate VLM call per distinct policy.
• The built-in SMAC renderer can inaccurately depict game state, potentially degrading the visual information available to the VLM.
• Effectiveness is tightly coupled to VLM capability; the method inherits the specific failure modes of the chosen model.
• Sanitization via sentence similarity may break for obscure, non-natural-language task specification formats or non-enumerable task spaces.
• The text summary added alongside the video (task history list) was empirically required due to the limited context window of VideoLLaMa2, suggesting brittleness to VLM context constraints.

Relevance to Vision-Language Models

This paper is a direct application of video-capable VLMs (specifically VideoLLaMa2) in a novel non-generative role: policy evaluation and task recommendation within a reinforcement learning curriculum. It demonstrates that VLMs can extract actionable behavioral signals from raw video that are inaccessible to text-only LLMs, extending the practical utility of multimodal models beyond standard perception or QA tasks. The sanitization pipeline — using embedding similarity to ground open-ended VLM outputs into a constrained action space — is a transferable design pattern for any system that uses VLMs as decision-makers over discrete structured outputs. For researchers tracking VLMs, the result that a 7B open model suffices (outperforming text-only GPT-class prompting approaches) speaks directly to the threshold at which video understanding becomes practically useful for embodied AI applications.