Skip to content

MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

MoniTor is the first training-free online Video Anomaly Detection (VAD) framework that pipes streaming video frames through a VLM (BLIP-2 + GLM-4-Flash) and scores anomalies in real time using a hierarchical memory architecture and a structured scoring queue. It achieves 82.57% AUC on UCF-Crime without any task-specific training, outperforming all offline training-free baselines and matching weakly supervised methods. The system produces an anomaly score every 0.6 seconds with ~5–6 second end-to-end latency.

Problem

Existing LLM/VLM-based VAD approaches are exclusively offline: they require access to full video before scoring and cannot handle streaming input. Prior online VAD methods use specialized trained models that generalize poorly across domains. The gap is a training-free system that (1) operates causally on streaming frames, (2) maintains robust temporal context without being misled by anomalous historical memory, and (3) overcomes LLMs' sensitivity to ambiguous scoring instructions.

Method

MoniTor converts each incoming frame to text via five BLIP-2 captions, denoised using ImageBind cosine similarity to select the top-10 captions, then summarized by GLM-4-Flash. Six modules then operate in concert:

  1. Dynamic Memory Gating Module (LSTM-inspired): Long-term memory (10-frame window, filtered by a cosine-similarity forgetting gate with threshold Ξ=0.5) and short-term memory (2-frame window) are separately summarized via GLM-4-Flash and passed to the LLM.

  2. Anomaly Priors Integration: Wikipedia-sourced definitions of the dataset's anomaly categories (13 for UCF-Crime, 6 for XD-Violence) are injected into the LLM context prompt.

  3. Standard Scoring Queue: A queue Q = {Q₀, Q₀.₁, 
, Q₁} stores the most recent caption that received each discrete score level. The queue is passed to the LLM as in-context examples to anchor its scoring scale across frames.

  4. Behavior Prediction and Dynamic Analysis: The LLM generates a next-frame prediction conditioned on current context (prompted as a law-enforcement agency); this prediction is fed back as additional context when scoring the next frame, creating a prediction-reality discrepancy signal.

  5. Score Optimization and Weight Assignment: Final score is smoothed as ĂŁ_i = α·a_i + (1−α)·a_{i−1} (α=0.7) to ensure temporal coherence.

The full LLM scoring call combines long-term memory, short-term memory, scoring queue, anomaly priors, and current frame summary: a_i = Ω_GLM(P₁ ∘ Ml ∘ Ms ∘ Q ∘ PA ∘ Si).

Key Contributions

  • First training-free online VAD framework using LLMs/VLMs on streaming input.
  • LSTM-inspired dual-memory gating (long-term + short-term with forgetting gate) for causal temporal context without memory pollution from anomalous past frames.
  • Standard Scoring Queue that provides LLM with grounded scoring examples, mitigating instruction-sensitivity.
  • Behavior Prediction module that introduces an expectation–reality feedback loop for emergent anomaly detection.
  • Achieves 0.6 s decision period and 6.5 s end-to-end delay, beating REWARD's 6.9 s delay.

Results

  • UCF-Crime AUC:
  • MoniTor: 82.57% vs. online-LAVAD baseline: 76.06% (+6.51%)
  • Outperforms all offline training-free methods (best prior: EventVAD 82.03%)
  • Competitive with online weakly supervised REWARD (86.94%) without any training
  • XD-Violence:
  • AP: 55.01% vs. online-LAVAD: 52.63% (+2.38%)
  • AUC: 82.57% vs. online-LAVAD: 79.11% (+3.10%) (Note: paper reports both AP and AUC for XD-Violence; Table 2 values used)
  • Ablation on UCF-Crime (each module's marginal gain from 76.06% base):
  • Anomaly Priors: +1.79%
  • Dynamic Memory Gating: +2.82%
  • Standard Scoring Queue: +2.59%
  • Behavior Prediction: +2.24%
  • Weight Assignment: +0.96%
  • All combined: 82.57%

Limitations

  • Camera transitions disrupt the Dynamic Memory Gating Module; ~60% of detection errors occur around scene changes because the system loses established scene context.
  • Frame-level (vs. segment-level) processing means temporal context is shallower than video-segment-based methods, limiting performance against top weakly supervised approaches.
  • High computational load (two RTX 4090 GPUs) from LLM + VLM inference makes edge deployment impractical in current form.
  • Gains on XD-Violence are modest because the dataset contains frequent camera transitions (movies/YouTube) that degrade the memory and prediction modules.

Relevance to Vision-Language Models

MoniTor is a concrete demonstration of how VLMs (BLIP-2, ImageBind) and LLMs (GLM-4-Flash) can be composed in a training-free pipeline for temporally structured video understanding, extending beyond static image-question tasks into causal, streaming settings. It addresses a known failure mode of VLMs—sensitivity to prompt phrasing and lack of temporal memory—with a structured scoring queue and dual-memory architecture, offering a reusable design pattern. The paper directly benchmarks against other VLM-based VAD systems (LAVAD, LLaVA-1.5, Video-LLaMA2), providing a clear ablation of what the LLM needs from visual context to reason about anomalies. For VLM researchers, the LSTM-inspired forgetting gate applied to text-summary streams (not raw features) is a lightweight approach to temporal grounding that sidesteps the need for video-native architectures.