Skip to content

Gaussian Mixture Modeling for Event-Aware Visual Allocation in Long Video Understanding

🕒 Published (v1): 2026-07-14 09:27 UTC · Source: Arxiv · link

Why this paper was selected

Gaussian mixture keyframe selection for long video VLMs; event-aware visual allocation; addresses uniform-sampling information loss

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GMM-EVA is a training-free keyframe selection method for long video understanding that fits a Gaussian Mixture Model (via EM) to frame-wise query-relevance scores, decomposing the video into latent semantic events. It then applies a differentiated allocation strategy—high-resolution primary keyframes per event plus low-resolution secondary keyframes—to reduce visual token overhead. On multiple long-video benchmarks it matches or exceeds state-of-the-art methods while consuming roughly half their token budget.

Problem

Existing keyframe selection methods for LVLMs treat video frames as isolated, equally weighted units, ignoring higher-level temporal structure. This causes two compounding failures: (1) equal token budgets are wasted on frames that are temporally redundant within the same semantic event, and (2) the absence of event modeling makes it difficult to ensure each distinct event receives at least one representative keyframe, degrading multi-event question answering in particular.

Method

Given \(T\) candidate frames and a textual query \(Q\), GMM-EVA proceeds in three stages:

  1. Query-relevance scoring. CLIP computes cosine similarity \(s_t = \text{sim}(\text{CLIP}_V(f_t), \text{CLIP}_T(Q))\) for each frame, then linearly normalizes scores to a proper distribution \(S' = \{s'_t\}\).

  2. Latent event discovery via GMM. The temporal importance density is modeled as \(p(\tau) = \sum_{j=1}^{C} \pi_j \mathcal{N}(\tau|\mu_j, \sigma_j^2)\), where each component \(j\) encodes a latent semantic event by its center \(\mu_j\), duration \(\sigma_j\), and cumulative significance \(\pi_j\). Parameters are estimated by EM, with \(s'_t\) used as per-frame importance weights in the M-step. Overlapping components (1-\(\sigma\) IoU \(> \epsilon_m\)) are merged via a mixture-of-Gaussians closed-form merge, yielding a refined set \(\Theta^* = \{\pi_j^*, \mu_j^*, \sigma_j^*\}_{j=1}^{C^*}\).

  3. Event-aware visual allocation. The top-\(K\) frames by relevance are sorted descending. Each frame \(f_{(k)}\) is hard-assigned to event \(j^* = \arg\max_j \gamma^*_{(k),j}\) via GMM posteriors. The first frame assigned to each event becomes a primary keyframe (\(F_\text{prim}\), allocated \(B_p = 256\) tokens); subsequent frames from the same event become secondary keyframes (\(F_\text{sec}\), allocated \(B_s = 64\) tokens). This concentrates high-fidelity resolution on semantic anchors while preserving temporal context cheaply.

Key Contributions

  • A training-free GMM-based event discovery module that transforms noisy scalar relevance scores into a structured, parametric event layout without any learned components or gradient-based optimization.
  • A differentiated keyframe allocation strategy that assigns high-resolution budgets to one primary keyframe per event and low-resolution budgets to secondary keyframes, explicitly pruning intra-event redundancy.
  • A component merging scheme (IoU-based Gaussian fusion) that adaptively determines event count \(C^*\) without requiring a user-specified number of clusters.
  • Demonstrated generalization across four embedding backbones (CLIP, SigLIP, LanguageBind-Video, Qwen3-VL-Embedding-8B) and four downstream LVLMs (Qwen2-VL-7B, Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL-2.5-8B).

Results

  • Token efficiency vs. Top-K baseline (Qwen2.5-VL-7B): GMM-EVA w/CLIP uses 44.9%/52.7%/44.8% of full budget on LongVideoBench/LVBench/VideoMME, matching Top-K accuracy (61.2 vs. 61.3 / 49.6 vs. 49.6 / 61.1 vs. 61.3) at roughly half the tokens.
  • Multi-event split (LongVideoBench): GMM-EVA w/CLIP achieves 57.0 vs. 54.4–55.7 for AKS/Q-Frame/BOLT/VideoITG/MSJoE, the best score among all methods.
  • Strong embedding backbone: With Qwen3-VL-Embedding-8B, GMM-EVA scores 62.7 on LongVideoBench overall, surpassing training-based VideoITG (+2.7) and MSJoE (+2.3) at 44.2% budget.
  • Ablation—primary allocation: Event-aware strategy (57.9 average) outperforms Top (55.5) and K-means (55.4) given the same oracle event count.
  • Ablation—secondary budget: \(B_s=64\) is optimal; removing secondaries entirely (\(B_s=0\)) drops VideoMME by 1.7 points, but GMM-EVA still exceeds Top-K at comparable budget by 1.8 points.
  • Scaling: External event scaling at 28 additional events yields +4.8% on the VideoMME long split vs. +2.0% on the full VideoMME set, suggesting larger benefit for longer videos.
  • Training-based parity: GMM-EVA w/CLIP on Qwen2-VL-7B matches GenS (58.7) and TSPO (58.7) on LongVideoBench without training and at 44.9% vs. 84.4%/82.0% budget.

Limitations

  • Event count and event boundaries are inferred solely from scalar CLIP relevance scores; richer frame-level features are not exploited in the GMM fitting stage.
  • Performance with a lightweight backbone (CLIP) trails training-based methods on most benchmarks except the multi-event split; gains require pairing with a stronger embedding model.
  • Candidate frames are obtained by uniform 1 FPS sampling before GMM fitting, so very dense or very sparse action segments may still be poorly represented by the initial frame pool.
  • Hyperparameters (\(\Delta t_c\), \(\epsilon_m\), \(K\), \(B_p\), \(B_s\)) are fixed across videos of varying length and complexity; adaptive tuning is not explored.
  • Evaluation is limited to multiple-choice benchmarks; open-ended generation quality is not assessed.

Relevance to Vision-Language Models

GMM-EVA directly addresses the visual token bottleneck that constrains LVLMs on long video inputs—a core architectural challenge as these models scale to longer context windows. By treating keyframe selection as structured event modeling rather than independent frame scoring, it offers a principled, plug-and-play complement to any LVLM without modifying its architecture or fine-tuning procedure. The differentiated resolution strategy (primary vs. secondary keyframes) is an elegant instantiation of the broader trend toward adaptive token budgeting in VLMs, analogous to dynamic token compression schemes. Its demonstrated generalization across multiple LVLM families (Qwen-VL, InternVL) and embedding backbones positions it as a modular efficiency layer applicable to diverse VLM pipelines.