Skip to content

MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAGNET is a multi-agent retrieval-augmented framework for answering queries that require finding and reasoning over relevant segments scattered across large multi-video collections (up to 500 clips). It introduces AVHaystacks, a 3100-QA audio-visual benchmark requiring multi-video evidence linkage, along with two new evaluation metrics (STEM and MTGS), achieving up to 89%/65% relative improvement over baselines on BLEU@4 and GPT evaluation.

Problem

Existing video QA benchmarks pair each query with a single short clip, failing to capture real-world scenarios where answering requires localizing and integrating evidence from multiple videos in large corpora. No prior benchmark or model systematically evaluated multi-video temporal grounding with joint audio-visual reasoning.

Method

MAGNET operates in two stages:

  1. AV-RAG retrieval: Computes cosine similarity between the query and each video using two complementary representations — fused audio-visual features (Hadamard fusion via ImageBind) and LLM-generated captions (Gemini 1.5 Pro, encoded with ImageBind). The average similarity \(\text{Sim}_{avg}\) ranks videos; top-\(k\) are selected.

  2. Multi-agent grounded QA: For each shortlisted video, a dedicated Qwen 2.5 Omni instance is dynamically spawned as an AVLLM agent. Each agent independently predicts relevant temporal segments and a partial answer. A GPT-4o meta-agent then aggregates all agent outputs into a coherent, temporally grounded, step-wise final answer.

A Salient Frame Selector (SFS) module improves frame sampling: given \(m\) candidate frames, it computes pairwise audio-visual embedding cosine similarity \(\Gamma_{ab}\) augmented with a temporal separation penalty \(\Delta_{ab} = \gamma\left(\frac{1}{\sin(\frac{\pi}{2}|a-b|+1)} - 1\right)\), then selects \(k\) frames minimizing total pairwise affinity \(\sum_{i=1}^{k-1} Q_{t_i t_{i+1}}\) via dynamic programming (Algorithm 1).

Key Contributions

  • AVHaystacksQA task: Multi-video retrieval + temporal grounding + QA over pools of up to 500 videos; 82% of QA pairs require evidence from ≥2 distinct videos.
  • AVHaystacks benchmark: 3100 annotated QA pairs (2k train / 1k test) with \(\langle\text{videoID, start, end}\rangle\) groundings spanning 500 YouTube videos across 27 categories.
  • STEM (Step-wise Error Metric): Quantifies alignment errors between ground-truth and predicted step sequences via Hungarian matching; reports missing steps \(S_M\), hallucinated steps \(S_H\), wrong order \(S_O\), and video-ID false positives/negatives \(S_{FP}, S_{FN}\) plus step-wise IoU.
  • MTGS (Matched Temporal Grounding Score): Computes mean temporal IoU only over matched video IDs: \(\text{MTGS} = \frac{1}{|V_M|}\sum_{v \in V_M} \text{IoU}_v\).
  • Model-agnostic multi-agent framework with dynamic agent spawning and a meta-agent aggregator.

Results

  • MAGNET+Qwen 2.5 Omni-FT achieves BLEU@4 of 55.82 (AVHaystacks-50) and 53.69 (AVHaystacks-Full) vs. VideoRAG baselines of 43.16/41.59 — a ~29% absolute gain.
  • GPT Eval: 7.84 vs. 6.32 (VideoRAG) on AVHaystacks-50; 7.56 vs. 6.13 on Full.
  • The paper claims up to 89% relative improvement on BLEU@4 and 65% on GPT Eval over baselines.
  • Retrieval (AVHaystacks-Full): R@3 = 73.15, R@5 = 79.20 vs. VideoRAG R@3 = 70.43, R@5 = 74.96.
  • MTGS (AVHaystacks-50, FT): 0.83 for MAGNET+Qwen 2.5 Omni-FT vs. 0.54 (ZS).
  • SFS vs. uniform sampling: +0.17 BLEU@4, +1.03 Human Eval for Qwen 2.5 Omni-FT on AVHaystacks-Full.
  • Closed-source upper bound (MAGNET+Gemini 1.5 Pro): BLEU@4 57.67, GPT Eval 8.03; best open model nearly matches it.

Limitations

  • AVHaystacks covers only instructional/how-to style videos (cooking, music tutorials, language instruction); generalization to other genres is unverified.
  • Full dataset results only include ZS and one FT variant — FT results on AVHaystacks-Full for all backbones are not fully reported in Table 2.
  • The meta-agent (GPT-4o) is a closed-source component, introducing cost and reproducibility concerns.
  • Sensitivity to the temporal penalty hyperparameter \(\gamma\) is noted, with slight degradation at \(\gamma=25\), suggesting tuning is required per task.
  • Benchmark scale (500 videos, 3100 QA pairs) is modest relative to real-world archives with millions of clips.

Relevance to Harnesses / Meta-Harnesses

MAGNET is a direct instantiation of the meta-harness pattern: a coordinator agent (GPT-4o) dynamically spawns and orchestrates a variable-size fleet of specialized sub-agents (per-video AVLLM instances), then synthesizes their outputs. The architecture cleanly separates retrieval, per-document reasoning, and cross-document aggregation into distinct harness layers — exactly the kind of decomposition studied in meta-harness research. The SFS module illustrates how a harness can insert domain-specific preprocessing between retrieval and reasoning stages without modifying sub-agents. The paper also contributes evaluation methodology (STEM, MTGS) for assessing harness output quality on grounded, multi-source tasks, which is directly applicable to benchmarking other multi-agent harnesses.