Skip to content

Multi-Agent VLMs Guided Self-Training with PNU Loss for Low-Resource Offensive Content Detection

🕒 Published (v1): 2025-11-14 08:03 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a self-training framework for low-resource offensive content detection that uses two VLM agents (moderator + user personas) to collaboratively verify classifier pseudo-labels, then trains a lightweight classifier with a novel Positive-Negative-Unlabeled (PNU) loss that differentiates high-consensus from ambiguous pseudo-labeled samples. The 428M-parameter CLIP classifier trained this way approaches the performance of 7B+ VLMs using as few as 50 labeled examples.

Problem

Supervised offensive content detection (hate speech, misogyny) requires expensive annotated data that is scarce due to low base rates and high annotation cost. Standard self-training propagates pseudo-label noise aggressively; LLM-based detection is accurate but too costly at inference scale. No existing method jointly handles multimodal inputs, label ambiguity from subjective content, and extreme label scarcity (n ≤ 100).

Method

A five-step iterative self-training pipeline: 1. A lightweight CLIP-Large classifier is trained on n labeled samples. 2. It ranks unlabeled samples by confidence and selects the top-k. 3. Two frozen Qwen-2.5-VL-72B agents — one playing a strict moderator (safety-first), one a lenient user (free-expression-first) — each independently label the top-k samples, then review each other's rationale before issuing a final decision. 4. Samples where all three (classifier + both VLM agents) agree become Agreed-Unknown with soft pseudo-labels (ŷ_p=0.67, ŷ_n=0.33); disagreements form Disagreed-Unknown. 5. The classifier is retrained with the PNU loss: standard PN supervision on labeled data + soft-PN on Agreed-Unknown + PU/NU-style risk estimation on Disagreed-Unknown, weighted by a dataset-specific γ ∈ [−1,1]. A validation gate rolls back rounds that degrade dev-set performance.

Key Contributions

  • MA-VLMs dual-persona prompting: moderator vs. user agents negotiate to produce more reliable and socially-grounded pseudo-labels than single-agent or CoT prompting.
  • PNU loss that unifies labeled, high-consensus pseudo-labeled, and ambiguous disagreement-prone data in a single objective, extending Sakai et al. (2017)'s framework with soft targets for Agreed-Unknown instances.
  • Agreement/disagreement partitioning of unlabeled data as an explicit signal for label confidence, rather than discarding uncertain samples entirely.
  • Task- and model-agnostic framework demonstrated across multimodal (memes) and text-only datasets.

Results

All results use n=100 labeled samples and CLIP-Large as the student classifier unless noted.

  • FHM (hate memes): SelfTrain CLIP + Qwen72B: 72.68 M-F1 vs. SupOnly CLIP: 59.24 M-F1 (+13.4 pts); trails SupOnly Qwen7B (70.41 M-F1 at same n) but with 16× fewer parameters.
  • MAMI (misogyny memes): SelfTrain: 73.49 M-F1 vs. SupOnly CLIP: 62.18 M-F1 (+11.3 pts); second best behind SupOnly Qwen7B (76.06).
  • HSOL (text hate/offensive): SelfTrain: 86.69 M-F1 vs. SupOnly CLIP: 85.30 M-F1 (comparable; CLIP-only pseudo-labeling suffices here).
  • Sent140 (sentiment): SelfTrain: 77.11 M-F1 vs. SupOnly CLIP: 64.22 M-F1 (+12.9 pts).
  • At n=50, SelfTrain maintains M-F1 >70 on FHM/MAMI while SupOnly CLIP drops to ~39–56 M-F1.
  • MA-VLMs prompt format outperforms Zero-Shot, Few-Shot, and CoT on both multimodal datasets (81.64 vs. 79.17 on MAMI).
  • CLIP + Qwen72B combined pseudo-labeling exceeds either model alone by >1.59 M-F1 points.

Limitations

  • Requires inference from a frozen 72B VLM at each self-training round, which is costly despite being inference-only (no fine-tuning).
  • γ is dataset-specific and requires ablation to tune per task; no automatic selection method is proposed.
  • RGCL outperforms the method in full-data (high-resource) settings, indicating the framework is primarily beneficial under label scarcity.
  • VLM agents are applied only to the top-k confident samples per round; highly ambiguous samples in later rounds yield declining pseudo-label quality.
  • Evaluation is limited to binary classification; multi-label offensive content tasks (MAMI Task B) are not addressed in the main experiments.

Relevance to Agentic AI / LLM Agents

This paper is a concrete instance of multi-agent LLM collaboration for classification: two VLM agents with deliberately contrasting system prompts (personas) engage in a structured negotiation — each providing rationale, reading the other's output, then issuing a revised decision — mirroring debate-style multi-agent architectures studied in the broader agent literature. The agreement/disagreement signal used to partition data is an emergent property of agent disagreement, which is a general mechanism applicable beyond content moderation. The work also illustrates a cost-effective deployment pattern: large agents (72B) act as infrequent oracle verifiers while a small student model handles inference at scale, directly relevant to agent system design where LLM calls must be budgeted.