Skip to content

MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAGNET is a multi-agent retrieval-augmented framework for answering queries that require finding and linking relevant temporal segments across large collections of up to 500 videos. The authors also introduce AVHaystacks, a 3100-QA audio-visual benchmark requiring multi-video reasoning, and two new evaluation metrics (STEM, MTGS). MAGNET achieves up to 89% and 65% relative improvement over baselines on BLEU@4 and GPT evaluation scores respectively.

Problem

Existing video QA benchmarks associate each question with a single short clip, failing to capture real-world retrieval scenarios that require searching hundreds of videos, localizing temporally relevant segments across sources, and synthesizing a coherent multi-modal answer. No prior benchmark systematically evaluates cross-video audio-visual retrieval and reasoning jointly.

Method

MAGNET is a two-stage retrieval-augmented multi-agent pipeline:

  1. AV-RAG retrieval: Given a query, ImageBind encodes fused audio-visual features (Hadamard fusion) and Gemini-1.5-Pro-generated captions separately. Cosine similarities are averaged to rank and select top-K videos from the corpus.

  2. Salient Frame Selector (SFS): A dynamic programming algorithm selects k maximally diverse, temporally dispersed frames from candidate videos by minimizing total pairwise cosine similarity with a temporal separation penalty term γ applied to adjacent frame pairs.

  3. Dynamic AVLLM agents: One dedicated Qwen 2.5 Omni instance is spawned per shortlisted video; each independently predicts relevant temporal windows and a partial answer. A GPT-4o meta-agent aggregates all partial responses into a final grounded, step-wise answer.

Two new metrics are introduced: MTGS (Matched Temporal Grounding Score) computes IoU over temporal intervals only for video IDs present in both prediction and ground truth; STEM (Step-wise Error Metric) uses Hungarian matching on step embeddings to count missing, hallucinated, out-of-order steps, and video-ID mismatches.

Key Contributions

  • Novel AVHaystacksQA task: multi-video linkage and reasoning over corpora of up to 500 clips; 82% of QA pairs require evidence from ≥2 distinct videos.
  • AVHaystacks benchmark: 3100 annotated QA pairs (2k/1k train/test) across 27 domains, with ⟨videoID, start, end⟩ grounding references.
  • MAGNET framework: model-agnostic, retrieval-augmented multi-agent architecture combining AV-RAG, SFS, dynamic per-video agents, and GPT-4o meta-aggregation.
  • STEM metric: Hungarian-aligned step-sequence error decomposition (missing, hallucinated, ordering, video-ID false positives/negatives, temporal IoU).
  • MTGS metric: video-ID-conditioned temporal IoU for balanced grounding evaluation.

Results

  • QA (AVHaystacks-50): MAGNET+Qwen2.5-Omni-FT achieves BLEU@4 = 55.82, CIDEr = 153.98, GPT Eval = 7.84, Human Eval = 4.15 vs. VideoRAG baseline BLEU@4 = 43.16, GPT Eval = 6.32 — ~29% BLEU@4 gain, ~24% GPT Eval gain (absolute); paper claims up to 89% relative improvement overall.
  • QA (AVHaystacks-Full): MAGNET+Qwen2.5-Omni-FT: BLEU@4 = 53.69, GPT Eval = 7.56 vs. VideoRAG 41.59 / 6.13.
  • Retrieval (AVHaystacks-Full): MAGNET R@3 = 73.15, R@5 = 79.20 vs. VideoRAG 70.43 / 74.96 — largest margin among all baselines.
  • Grounding (AVHaystacks-50): MAGNET+Qwen2.5-Omni-FT MTGSavg = 0.83 (ZS: 0.54); missing steps reduced from 0.37 to 0.11.
  • SFS ablation: Switching from uniform sampling to SFS increases BLEU@4 by ~17 points absolute for Qwen2.5-Omni-FT on AVHaystacks-Full (36.58 → 53.61).
  • Modality ablation: Joint audio+visual consistently outperforms audio-only or visual-only across all metrics.
  • Closed-source upper bound (MAGNET+Gemini-1.5-Pro): BLEU@4 = 57.67, GPT Eval = 8.03; best open-source model nearly matches it.

Limitations

  • AV-RAG and SFS rely on off-the-shelf ImageBind and Gemini caption encoders; no end-to-end fine-tuning of the retrieval backbone.
  • Each video requires spawning a separate AVLLM agent instance, which may not scale efficiently to corpora far exceeding 500 videos without additional engineering.
  • Benchmark videos sourced exclusively from YouTube how-to content (27 categories); domain coverage may not generalize to narrative, surveillance, or broadcast media.
  • γ hyperparameter in SFS shows sensitivity and slight performance degradation at high values, requiring per-dataset tuning.
  • No evaluation of latency, cost, or agent-count scaling behavior reported.

Relevance to Agentic AI / LLM Agents

MAGNET is a concrete instantiation of the multi-agent RAG pattern: specialized sub-agents process independent video streams in parallel, while a separate meta-agent synthesizes their outputs — a direct embodiment of the decompose-and-aggregate agentic paradigm. The dynamic agent spawning (one agent per retrieved video) demonstrates scalable, task-driven agent instantiation without static graph design. The GPT-4o meta-agent role mirrors orchestrator patterns studied in tool-use and planning literature, but grounded in multi-modal temporal evidence rather than text. For researchers tracking agentic AI, this paper provides a worked example of RAG+multi-agent coordination for perceptual tasks, with rigorous new metrics (STEM, MTGS) applicable to any step-grounded agentic evaluation.