Skip to content

Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

🕒 Published (v1): 2026-07-09 07:48 UTC · Source: Arxiv · link

Why this paper was selected

LLM-as-evaluator for open-ended RL curricula; directly advances auto-curriculum design for agentic systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VIP (Visual Inspection of Policies) uses a Video Language Model (VLM) to directly watch recorded episode videos of RL agents and recommend the next curriculum task, replacing scalar or text-based learning-signal proxies. Applied to StarCraft Multi-Agent Challenge (SMAC), VIP with a lightweight 7B VLM outperforms both text-only ablations and scalar-score-based unsupervised curriculum methods on held-out tasks. The core insight is that video captures behavioral information — e.g., near-winning strategies masked by zero win rate — that aggregate signals cannot.

Problem

Open-ended autocurricula must assess an agent's current learning progress to recommend the next useful task. Scalar task scores (e.g., win rate, learning potential estimates as in UED/PLR) fail when rewards are sparse or deceptive — an agent close to a winning strategy can still register 0% win rate. Text-only LLM summaries face the same limitation and describe candidate tasks rather than directly observing the policy. No prior method directly inspects policy behavior visually at curriculum time in a multi-agent setting.

Method

VIP is a loop: (1) train agents on the current task \(\theta_t\) with MAPPO; (2) evaluate and record episode video(s) \(\phi^{\text{video}}_t\) plus a short text summary \(\phi^{\text{text}}_t\) (win rate + task history); (3) feed both to a VLM prompted with domain context and task-space syntax to produce a raw next-task recommendation \(\vartheta_t\); (4) sanitize \(\vartheta_t\) via sentence similarity (sinusoidal embeddings) against the enumerable valid task set to yield \(\theta_{t+1}\). The VLM used is VideoLLaMa2-7B (quantized). No new hyperparameters beyond the VLM and RL algorithm are introduced. A fallback random-task selection fires when no task exceeds the similarity threshold (never triggered in experiments). The entire VLM inference costs ~1% of total compute; MAPPO dominates the remaining 99%.

Key Contributions

  • Introduces VIP, the first open-ended RL autocurriculum that uses raw episode videos (via a VLM) to assess learning progress rather than scalar or text-only signals.
  • First application of any unsupervised autocurriculum to SMAC, a complex multi-agent benchmark previously unexplored in this setting.
  • Demonstrates that a 7B open-weight VLM suffices for curriculum recommendations competitive with or superior to PLR-based baselines.
  • Shows empirically that the video modality is causally responsible for VIP's gains over its text-only ablation.
  • Proposes a sentence-similarity sanitization module to recover from VLM hallucinations and formatting errors without iterative re-prompting.

Results

On held-out SMAC maps (unseen by all curricula during training, evaluated after fine-tuning for \(10^7\) steps):

  • 3s vs 4z: VIP achieves 0.84 ± 0.16 win rate vs. Text-Only 0.00, Random 0.00, PLR-PVL 0.00, PLR-MaxMC 0.00, MAPPO-from-scratch 0.04.
  • 3s5z: VIP achieves 0.76 ± 0.20 vs. Text-Only 0.21, Random 0.59, PLR-PVL 0.03, PLR-MaxMC 0.00, MAPPO 0.16.
  • 10m vs 11m: VIP achieves 0.53 ± 0.22 vs. Text-Only 0.51, Random 0.49, PLR-PVL 0.04, PLR-MaxMC 0.00, MAPPO 0.40.
  • PLR variants (PLR-PVL, PLR-MaxMC) consistently underperform even the random curriculum baseline on most held-out maps.
  • All results reported with 95% CI over 5 seeds.

Limitations

  • Evaluated only on SMAC; generalization to other multi-agent or single-agent domains is not empirically validated.
  • VideoLLaMa2-7B does not support video batching, so only a single episode video is fed per curriculum step.
  • SMAC's built-in renderer can inaccurately depict game state, potentially limiting VLM perception quality.
  • The sentence-similarity sanitization only applies to enumerable task spaces; non-enumerable spaces require iterative prompting or alternative approaches.
  • VIP inherits the weaknesses of the chosen VLM; no ablation over different VLMs is presented.
  • Zero-shot transfer between SMAC configurations is not feasible, necessitating fine-tuning on test tasks — which reintroduces some domain knowledge implicitly through hyperparameter choices.
  • Text-only ablation parity on the 10m vs 11m map leaves unclear how much video contributes on tasks with simpler behavioral cues.

Relevance to Agentic AI / LLM Agents

VIP is a direct instance of LLM/VLM-as-agent-overseer: the VLM acts as a meta-controller that observes agent behavior and orchestrates the training curriculum, a pattern increasingly relevant to scalable oversight and automated agent training pipelines. The use of video — rather than text summarization — as the primary interface between the VLM and the RL agent aligns with emerging work on multimodal LLMs as world-model proxies and evaluators. The sanitization module (similarity-based grounding of free-form LLM outputs to valid discrete actions) is a lightweight pattern applicable to any LLM-in-the-loop agentic system that must map natural-language outputs to structured action spaces. More broadly, VIP demonstrates that open-weight 7B VLMs can serve as practical curriculum curators in multi-agent RL, lowering the barrier for VLM integration into agentic training infrastructure.