Skip to content

PANDA: Towards Generalist Video Anomaly Detection via Agentic AI Engineer

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PANDA is a training-free, manual-free agentic video anomaly detection (VAD) system built on MLLMs that generalizes across arbitrary scenes and anomaly types without domain-specific retraining. It combines scene-aware RAG planning, heuristic reasoning, tool-augmented self-reflection, and a chain-of-memory mechanism to iteratively refine decisions. It achieves state-of-the-art performance across multi-scenario, open-set, and complex-scene VAD benchmarks.

Problem

Existing VAD methods are specialist-oriented: training-dependent methods require annotated data per domain and degrade on out-of-distribution scenarios; training-free methods still rely on manually engineered pipelines (hand-crafted prompts, pre/post-processing, static rules) that lack adaptivity to novel scenes or unseen anomaly types. Neither paradigm achieves generalist VAD without human involvement.

Method

PANDA is an MLLM-based agentic framework (built with LangGraph) with four coordinated modules:

  1. Self-adaptive scene-aware strategy planning: Given a user query and sampled keyframes, a VLM extracts structured environmental context (scene, potential anomalies, weather, video quality). A FAISS-indexed anomaly knowledge base is queried via RAG (all-MiniLM-L6-v2 embeddings, top-k=5 rules) to produce a scene-specific detection plan including preprocessing steps and heuristic chain-of-thought prompts.

  2. Goal-driven heuristic reasoning: A VLM processes 5-frame clips (1 FPS sampling) with the RAG-derived plan and short chain-of-memory (l=5 steps of text+visual traces), outputting a ternary status (Normal/Abnormal/Insufficient) plus a score ∈ [0,1].

  3. Tool-augmented self-reflection: On Insufficient status, PANDA retrieves similar past reflection cases from long CoM, constructs a reflection prompt, and an MLLM selects tools (image deblur, denoising, super-resolution, object detection, image retrieval, web search) to acquire additional evidence. Updated rules and heuristic prompts drive a re-reasoning pass; up to r=3 reflection rounds are allowed.

  4. Self-improving chain-of-memory (CoM): Short CoM captures recent l reasoning/reflection outputs as rolling text+visual context. Long CoM accumulates full decision traces (initial reason → reflection plan → refined reason) across the video, retrieved via similarity for future reflection episodes.

Backbone: Qwen2.5VL-7B for perception/reasoning; Gemini 2.0 Flash for planning/reflection.

Key Contributions

  • First fully training-free and manual-free generalist VAD framework; no domain-specific pipelines or annotated data required.
  • Scene-adaptive RAG mechanism that dynamically retrieves anomaly rules conditioned on perceived environment context.
  • Progressive tool-augmented self-reflection loop with experience retrieval from long CoM to resolve decision uncertainty.
  • Dual chain-of-memory design (short: rolling window for temporal context; long: accumulating trace for cross-video experience).
  • CSAD benchmark: a new 100-video complex-scene anomaly detection dataset covering degraded conditions (low resolution, poor illumination, high noise, long-range temporal anomalies).

Results

  • UCF-Crime (AUC%): PANDA offline 84.89 vs. best training-free baseline AnomalyRuler 80.28; surpasses weakly-supervised TPWNG 88.96 only by a small margin but with zero training.
  • XD-Violence (AP%): PANDA offline 70.16 vs. best training-free AnomalyRuler 62.01; competitive with weakly-supervised HL-Net 73.67.
  • UBnormal open-set (AUC%): PANDA offline 75.78 — outperforms all methods including weakly-supervised TPWNG 64.94 and training-free AnomalyRuler 57.26.
  • CSAD complex-scene (AUC%): PANDA offline 73.12 vs. best training-free baseline 71.90 (Holmes-VAU, instruction-tuned); all training-dependent methods score below 65.
  • Online mode degrades modestly (82.57 / 72.41 / 72.41 / 71.25) due to fewer perception frames.
  • Ablation on UCF-Crime: base (no modules) 75.25% → +Planning +5.12% → +Reflection +2.26% → +Memory +2.26% = 84.89%.
  • Hyperparameter sensitivity: r=3 reflection rounds and k=5 RAG rules optimal; l=5 short CoM length optimal.

Limitations

  • Inference is computationally expensive due to repeated MLLM calls, multi-round reflection, and tool invocations; no latency figures reported.
  • Online mode degrades noticeably due to reduced perception sampling (10 vs. 300 keyframes); real-time applicability is limited.
  • Performance still below the best weakly-supervised methods on UCF-Crime and XD-Violence (e.g., TPWNG 88.96% vs. 84.89%).
  • CSAD benchmark is self-constructed and small (100 videos), limiting external validation of complex-scene results.
  • Relies on proprietary Gemini 2.0 Flash for planning/reflection, introducing API cost and reproducibility concerns.
  • Chain-of-memory grows unboundedly over long videos; no analysis of memory scalability or retrieval quality degradation.

Relevance to Agentic AI / LLM Agents

PANDA is a concrete instantiation of the agentic AI paradigm applied to a perception task: it closes the loop of perceive → plan → act → reflect → remember without any human in the loop, directly demonstrating how tool use, RAG, and memory architectures extend MLLM capabilities beyond single-pass inference. The chain-of-memory design — particularly the distinction between short rolling context and long episodic experience retrieval — is directly transferable to other long-horizon agentic tasks. The self-reflection mechanism triggered by model-expressed uncertainty (Insufficient status) is a principled approach to uncertainty-driven tool invocation that the broader agent community can adapt. This work exemplifies training-free generalization through agentic orchestration rather than fine-tuning, a growing trend in applying LLM agents to specialized computer vision domains.