Skip to content

MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection

🕒 Published (v1): 2025-07-09 14:46 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MIND is a zero-shot harmful meme detection framework that avoids annotated data by retrieving visually/textually similar unannotated memes, extracting bidirectional insights from them via chained LMM agents, and resolving predictions through a multi-agent debate-then-arbitration mechanism. On three benchmarks, MIND built on LLaVA-1.5-13B matches or exceeds much larger open-source models and competitive closed-source ones.

Problem

Supervised harmful meme classifiers fail on newly emerging memes because annotation lags behind meme evolution. Few-shot in-context learning approaches still require pre-labeled examples, which are unavailable when novel harmful content first appears. No prior work tackled fully zero-shot harmful meme detection without any label access.

Method

MIND operates in three stages:

  1. Similar Sample Retrieval (SSR): For a target meme \(M = \{V, T\}\), a multimodal embedding is computed as \(E = \lambda_v \cdot V_{\text{enc}}(V) + \lambda_t \cdot T_{\text{enc}}(T)\). Cosine similarity \(s = \frac{E_{\text{target}} \cdot E_{\text{ref}}}{\|E_{\text{target}}\|\|E_{\text{ref}}\|}\) selects the top-\(K\) reference memes \(M_{\text{similar}}\) from an unannotated pool.

  2. Relevant Insight Derivation (RID): A deriving LMM agent processes \(M_{\text{similar}}\) sequentially in forward order (\(I_{\text{fwd},i} = \text{LMM}_{\text{deriving}}(M_{\text{similar},i}, I_{\text{fwd},i-1}, P_{\text{deriving}})\)) and then in reverse (backward pass), producing two complementary insight sets \(I_{\text{fwd},K}\) and \(I_{\text{back},K}\). The backward pass corrects recency bias introduced by sequential accumulation.

  3. Insight-Augmented Inference (IAI): Two debater agents each consume one insight set plus the target meme to produce binary harm judgments \(J_{\text{fwd}}\) and \(J_{\text{back}}\). If they agree, that label is adopted; if they disagree, a third judge agent arbitrates by weighing both reasoning chains: \(J_{\text{final}} = \text{LMM}_{\text{judge}}(J_{\text{fwd}}, J_{\text{back}}, V_{\text{target}}, T_{\text{target}})\).

Default configuration: \(K=3\), LLaVA-1.5-13B backbone for all agents.

Key Contributions

  • First multi-agent framework for fully zero-shot harmful meme detection requiring no annotated data.
  • Bidirectional insight derivation (forward + backward chaining) over retrieved similar memes to address sequential recency bias.
  • Debate-then-arbitration inference where a judge agent resolves disagreements between forward- and backward-insight debaters.
  • Demonstrated generalization across four model architectures (7B–34B open-source, closed-source Gemini) and three datasets.

Results

  • MIND (LLaVA-1.5-13B) vs. base LLaVA-1.5-13B: macro-F1 gains of +14.74% (HarM), +7.70% (FHM), +13.32% (MAMI).
  • On HarM, MIND (13B) outperforms base LLaVA-1.6-34B by +3.60% macro-F1 and GPT-4o by +4.90% macro-F1.
  • MIND applied to LLaVA-1.6-34B yields additional gains of +4.53%, +4.87%, +4.10% on the three datasets.
  • MIND applied to closed-source Gemini-1.5-Flash: +2.74%, +3.60%, +3.60% macro-F1.
  • Ablation: removing RID (insight derivation entirely) causes the largest single-component drop (up to −13.26% macro-F1 on HarM); removing SSR causes up to −4.27%; removing the debate mechanism causes up to −4.22%.
  • Optimal retrieved meme count is \(K=3\); performance degrades with higher \(K\).

Limitations

  • Relies on a reasonably representative unannotated reference pool; performance degrades with poor or irrelevant reference memes (SSR ablation shows 4.27% F1 drop with random selection).
  • Inference cost scales with \(K \times 2\) sequential LMM agent calls plus two debater calls and a conditional judge call per sample — substantially more expensive than single-pass baselines.
  • Backbone model is fixed; the bidirectional derivation adds latency proportional to \(2K\) LMM forward passes per query.
  • Evaluated only on binary harmfulness classification; multi-class or severity-graded settings are unexplored.
  • No evaluation under distribution shift where the reference pool is empty or domain-mismatched.

Relevance to Harnesses / Meta-Harnesses

MIND is a concrete example of a task-specific multi-agent harness: it composes retrieval, sequential reasoning chains, parallel debaters, and a conditional arbitration judge into a fixed orchestration graph — precisely the kind of structure meta-harnesses are designed to define and reuse. The bidirectional chaining pattern (forward pass then backward pass feeding separate downstream agents) and the consensus-or-judge dispatch are reusable orchestration primitives that a meta-harness could expose as first-class building blocks. The paper also demonstrates a key meta-harness design tension: adding more orchestration stages increases accuracy but multiplies LMM call counts, directly informing cost-vs-quality trade-offs in harness scheduling. Its model-agnostic generalization validates the principle that harness logic should be decoupled from the backbone model.