Skip to content

The Label Imitation Game: Turing Test Network for Zero-Shot Pseudo-Label Pruning

🕒 Published (v1): 2026-06-29 20:06 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Jason Corso; Turing-test method cleans zero-shot pseudo-labels; novel + practical

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Introduces the Label Imitation Game (LIG), which formalizes pseudo-label pruning as adversarial interrogation, and the Turing Test Network (TTN), a lightweight transformer-based "judge" that evaluates whether a pseudo-label is contextually plausible against a dataset-wide reference set. A TTN trained purely on image classification can zero-shot transfer to prune object-detection pseudo-labels from three different VLMs, yielding up to 28% F1 gains on worst-performing categories and enabling "Category Revival" where detectors recover from zero recall on transfer-vulnerable classes.

Problem

Foundation VLM pseudo-labeling enables massive-scale annotation but VLMs produce systemic hallucinations that evade standard confidence thresholds. Traditional fixed-threshold pruning discards authentic low-confidence labels while retaining "confidently wrong" errors. There is no lightweight, post-hoc interrogator that can differentiate accurate labels from hallucinations without human supervision or changes to the VLM architecture.

Method

Label Imitation Game (LIG). Pruning is formalized as a game where a Judge receives a combined set of Ideal (accurate) and Candidate tokens and must identify which tokens are pseudo-labels (Pseudo-Label Game, PLG) or belong to out-of-category distributions (Out-of-Category Game, OCG). OCG is the primary training signal: the judge sees mostly in-distribution Ideal tokens plus a minority of disjoint-category tokens and must detect the outliers.

Turing Test Network (TTN). A transformer-based discriminator consisting of: (i) a frozen CLIP feature extractor encoding label-defined image patches as \(\mathbb{R}^{768}\), (ii) a learned tokenizer projecting to \(\mathbb{R}^{512}\), and (iii) an 8-layer transformer reasoning block producing Accept/Reject logits \(z_i\) per token. Training uses a weighted binary cross-entropy loss:

\[ \mathcal{L}_\text{BCE}(w) := -\sum_{i=1}^{n} \big[ p_w y_i \log(\sigma(z_i)) + (1-y_i)\log(1-\sigma(z_i)) \big] \]

with four training states: Ideal human labels (\(S=1\)), OCG-human (\(S=2\)), OCG-pseudo (\(S=3\)), and PLG (\(S=4\)). Each sequence uses 10 Reference tokens (\(S=1\) only) followed by 1 Candidate token sampled from the full distribution — forcing the TTN to reason about context.

Zero-shot cross-task transfer. The baseline TTN is trained exclusively on 8 image-classification datasets (1,303 classes, no pseudo-labels, \(\pi_3=\pi_4=0\)). At inference on detection data, it constructs a dataset-spanning "Ideal" set from the 1,000 highest-confidence pseudo-labels per target category \(c^*\) and iteratively judges all candidates via Self-Referential Pruning. Task-specific fine-tuning (TTN\(_D\)) adds VLM pseudo-labels and \(\leq\)100 human labels per class with a conservative pruning bias (\(p_w=0.2\)).

Key Contributions

  • Label Imitation Game (LIG): a framework formalizing pseudo-label pruning as adversarial interrogation, decoupled from any specific VLM or task head
  • Turing Test Network (TTN): a lightweight, task-agnostic transformer judge that evaluates semantic and spatial consistency without class labels or spatial-geometric verification
  • Zero-shot cross-task transfer: TTN trained strictly on classification directly prunes object-detection pseudo-labels from YOLO-World, YOLOE, and Grounding DINO
  • Category Revival: pruning "detoxifies" downstream training signals, enabling detectors to recover from zero recall on transfer-vulnerable classes (up to 27 classes)
  • 28% F1 gains on worst-performing baseline categories; 44% with task-specific fine-tuning

Results

  • Classification LIG validation (Tab. 1): TTN achieves >94% overall accuracy on 7/8 datasets, including 98.0% on CIFAR-10 and 96.2% on ImageNet
  • Detection LIG validation (Tab. 2): TTN\(_D\) achieves 73–90% overall accuracy; Accept accuracy >93% across all domains, Reject accuracy ranges from 50.6% (LVIS) to 84.2% (VOC) depending on instance density and class complexity
  • Overall cross-dataset performance (Tab. 3, averaged over VLMs, thresholds, classes):
  • TTN: precision 0.436, recall 0.426, F1 0.376, mAP50 0.355, mAP50:95 0.243
  • TTN\(_D\): precision 0.447, recall 0.437, F1 0.387, mAP50 0.362, mAP50:95 0.247
  • AL baseline (no pruning): F1 0.362, mAP50 0.350, mAP50:95 0.238
  • Oracle (LIG\(^*\), IoU < 0.5): F1 0.525, mAP50 0.382, mAP50:95 0.259
  • On VOC (Fig. 5), TTN consistently improves precision across all three VLM models and confidence thresholds \(\tau \in [0.05, 0.5]\), with more substantial precision gains than incidental recall drops

Limitations

  • TTN pruning consistently decreases recall alongside the precision improvement — not a Pareto improvement
  • Reject accuracy on detection datasets varies widely (50.6–84.2%), with the worst performance on the largest, most diverse dataset (LVIS, 1,203 classes)
  • Self-Referential Pruning uses top-1000 confidence pseudo-labels as the "Ideal" set, which may still contain residual noise
  • Requires at least 10 reference labels per category \(c^*\) for contextualized pruning; categories with fewer are left unpruned
  • TTN\(_D\) task-specific variant requires up to 100 human labels per class and 4.28 hours of fine-tuning
  • Training uses a fixed 10:1 reference-to-candidate ratio — not validated whether this is optimal across tasks

Relevance to Vision-Language Models

This paper directly tackles the hallucination bottleneck in VLM-based pseudo-labeling pipelines — arguably the most critical obstacle to scaling open-vocabulary annotation. The key finding that a judge trained on semantic context alone (no spatial geometry) can detect VLM hallucinations across architectures (GDINO, YOLOE, YOLOW) and tasks (classification → detection) implies that VLM errors share a learnable semantic signature. The Category Revival result is practically important: cleaning pseudo-labels can recover detection performance on classes that VLMs systematically fail on, which is otherwise a silent failure mode in automated annotation loops.