Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition

🕒 Published (v1): 2026-06-16 07:31 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

"Divide, Deliberate, Decide" (D3) is a fully-local, zero-shot multi-agent framework for fine-grained egocentric action recognition that chains a VLM orchestrator (segmentation + initial ranking) with a heterogeneous ensemble of specialist VLMs that deliberate via structured peer-consultation, then aggregates via Borda count. The gain over the single-model baseline—+3.1% top-1 and +16.1% top-5 on MECCANO—comes from decorrelated priors across model families, not from extra compute alone.

Problem

Single VLMs performing fine-grained action recognition in egocentric video anchor on salient visual tokens and miss discriminative temporal/spatial cues (e.g., direction of hand motion, contact state). Scaling to larger models is often infeasible for on-premise industrial deployments due to latency, privacy, and cost constraints, so a different strategy is needed to correct these biased predictions without fine-tuning.

Method

The three-stage pipeline operates on a video \(V \in \mathbb{R}^{C \times W \times H \times T}\) with action label set \(\Lambda = \{\lambda_1, \dots, \lambda_m\}\):

1. Divide — The orchestrator (Qwen3.5-27B) temporally chunks the video, sub-samples frames, and generates an initial top-\(k\) candidate action ranking \(\hat{Y}^{(0)}\) per segment.

2. Deliberate — \(p\) heterogeneous specialist VLMs (Qwen3.5:9b, Ministral-3:8b, Gemma4:e4b) each produce a prior ranking \(\tilde{Y}^{(i)}\), share priors, then engage in a single-question peer-consultation round targeting disputed visual cues. Each agent produces a posterior ranking \(\hat{Y}^{(i)}\) after reading the full exchange; unanimity is not enforced.

3. Decide — Specialist posteriors are aggregated via Borda count: $\(B(y) = \sum_{i=1}^{p} w(r_i(y)), \quad w(r) = k - (r-1)\)$ where \(r_i(y)\) is \(y\)'s rank in agent \(f_i\)'s posterior (or \(k+1\) if absent). The orchestrator receives \(\hat{Y}^{(0)}, \hat{Y}^{(1)}, \dots, \hat{Y}^{(p)}, \hat{Y}^{(B)}\) and re-ranks to produce \(\hat{Y}^{(F)}\), constrained to labels already proposed by any agent.

All inference runs locally via Ollama on a single NVIDIA RTX 6000; no fine-tuning is performed.

Key Contributions

• A structured three-stage orchestrator-specialist pipeline for zero-shot fine-grained action recognition that runs entirely on local hardware.
• A deliberation protocol with a single-question peer-consultation round that lets heterogeneous VLMs probe each other's visual evidence without collapsing to trivial consensus.
• Empirical demonstration that performance gains stem from decorrelated model priors (heterogeneous families) rather than from additional compute or more votes: homogeneous ensembles peak at 15.9/43.2 top-1/top-5 vs. 16.8/45.0 for the heterogeneous configuration.
• First reported zero-shot evaluation on MECCANO, establishing a baseline for future work.

Results

• Heterogeneous D3 vs. orchestrator-only baseline (avg over 7 MECCANO test videos): 16.8% / 45.0% top-1/top-5 vs. 13.5% / 28.9% (+3.1% / +16.1%).
• With ground-truth boundaries (GTB): 20.8% / 38.6% vs. 17.5% / 36.9%.
• Orchestrator overrides its initial top-1 in 70.9% of segments; net override direction is positive: 21.6% correct flips vs. 10.3% incorrect flips; 28.1% top-5 correct re-labels vs. 2.9% incorrect.
• Homogeneous ablation: Gemma4:e4b×3 → 15.5/33.5; Ministral-3:8b×3 → 14.6/42.5; Qwen3.5:9b×3 → 15.9/43.2 — all below the heterogeneous result.
• Stage 1 segmentation: mean IoU = 32.9%; GT segment coverage = 47.5% — identified as the dominant performance bottleneck.
• Runtime: ~5 minutes per segment (full pipeline) vs. ~40 seconds (orchestrator alone); 11 VLM calls per segment.

Limitations

• Evaluation restricted to a single benchmark (MECCANO); generalization to other egocentric datasets is unverified.
• Zero-shot temporal segmentation is a hard ceiling: mean IoU of 32.9% and 47.5% GT coverage mean many ground-truth actions are never recoverable downstream.
• ~11 VLM calls per segment (~5 min/segment) precludes real-time deployment.
• Framework is evaluated only with one specific orchestrator and three specific specialist models; sensitivity to these choices is not fully characterized.
• The single-question constraint per agent in the deliberation round was chosen for compute budget reasons, not demonstrated as optimal.

Relevance to Harnesses / Meta-Harnesses

D3 is a concrete instantiation of an orchestrator–specialist meta-harness pattern: a coordinator model decomposes the task, routes sub-problems to specialized agents, collects structured outputs, and synthesizes a final decision—a pattern directly analogous to meta-harness architectures that wrap heterogeneous tool-using agents. The paper's deliberation protocol—where agents broadcast intermediate outputs, query each other, and revise before a rank aggregation step—operationalizes the "structured multi-round communication" design point that meta-harnesses often need when aggregating conflicting sub-agent results. The Borda count aggregation as a model-agnostic rank fusion mechanism is a lightweight primitive applicable to any harness that must reconcile ranked outputs from agents with different confidence calibrations. The finding that heterogeneous priors (not extra compute) drive gains directly informs harness design choices about agent diversity vs. agent count.