Skip to content

Towards Self-Refinement of Vision-Language Models with Triangular Consistency

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a self-refinement framework for VLMs that generates its own supervised training data from unlabeled images without external annotations or stronger teacher models. The core idea is a Triangular Consistency principle: any component of an image-question-answer (IQA) triplet, when masked and re-predicted, should match the original. Using LLaVA-1.5 as baseline, the framework achieves consistent benchmark improvements through purely self-generated data.

Problem

Supervised visual instruction tuning requires expensive image-question-answer triplets that cannot be crawled from the web and need human annotation or proprietary teacher models (e.g., GPT-4V). As VLMs improve, finding a superior teacher for data annotation becomes increasingly hard; environmental feedback (e.g., evaluation scores) is also an inefficient signal. The capability of VLMs to self-supervise from unlabeled images alone remains underexplored.

Method

Three-stage self-refinement framework applied iteratively:

  1. Instruction generation fine-tuning: The base VLM is fine-tuned on three tasks jointly—I→(Q,A) (generate a QA pair from image), (I,Q)→A (standard VQA), and (I,A)→Q (inverse: reconstruct question given image+answer). Loss is the sum of cross-entropy losses over all three tasks. This endows the model with bidirectional generation capability needed for filtering.

  2. Triangular Consistency filtering: For each generated IQA triplet (I, Q, A), the model re-predicts A' from (I, Q) and Q' from (I, A). A consistency score S = sqrt(Sim(Q, Q') × Sim(A, A')) is computed using type-appropriate metrics: Sentence Transformer for short text, BERTScore for long text, exact match for multiple-choice, IoU for localization. The top 20% by score are retained as D_filtered.

  3. Instruction tuning: D_filtered is merged with the original seed dataset (LLaVA-1.5's llava_v1_5_mix665k) and used to fine-tune the model, producing M^(1). The process iterates with new unlabeled images and M^(1) as the new base.

A causal theoretical analysis grounds the semi-supervised learning guarantee: language X and image Y share an underlying semantic concept S, enabling consistent cross-modal reconstruction from unlabeled data.

Key Contributions

  • Triangular Consistency principle: a model-intrinsic quality score for generated IQA triplets based on bidirectional re-prediction consistency.
  • Self-refinement framework (SRF-LLaVA): end-to-end pipeline that uses only unlabeled images and the model itself—no human annotations, no external VLMs/LLMs.
  • Causal theoretical analysis: semi-supervised learning guarantee that unlabeled-image self-supervision suffices, grounded in the causal structure linking language, images, and semantic concepts.
  • Iterative refinement: framework naturally extends to multiple rounds by treating M^(k) and D^(k) as the new starting point.
  • Public release of ~2M synthetic image-instruction pairs generated during experiments.

Results

  • LLaVA-1.5 (7B and 13B) used as baseline throughout.
  • Self-refinement produces consistent improvements across multiple benchmarks without any external supervision (specific numbers described as "deliberately modest" improvements; exact per-benchmark figures appear in Figure 1b and Table results not fully reproduced in the provided text).
  • Ablations (Table 6 referenced) show that naively applying existing VLMs to generate instructions without the multi-task instruction generation stage fails to produce precise, diverse data.
  • Top-20% consistency filtering is the chosen operating point (implied by the text; threshold sweep not fully shown in excerpt).

Limitations

  • Improvements are described as "consistent, though deliberately modest"—gains are incremental rather than large.
  • Triangular Consistency scoring relies on similarity metrics (BERTScore, Sentence Transformer, IoU) that are themselves imperfect proxies for semantic correctness.
  • The seed dataset (llava_v1_5_mix665k) is still required; the method is not truly zero-supervision from scratch.
  • The theoretical guarantee assumes the causal structure holds, which may not generalize to all visual domains.
  • Filtering retains only the top 20%, making data efficiency dependent on how well the generation step covers diverse question types.

Relevance to Vision-Language Models

This work directly addresses the data bottleneck in visual instruction tuning—a central challenge for VLM training—by demonstrating that a VLM can bootstrap higher-quality supervision from its own predictions on unlabeled images. The Triangular Consistency principle offers a principled, model-internal quality signal that complements RLHF/reward-model approaches and avoids dependence on proprietary APIs. It connects to the broader line of self-play, self-distillation, and iterative self-improvement research (e.g., STaR, SPIN) applied specifically to the multimodal setting, and is directly relevant for researchers interested in data-efficient VLM training, synthetic data generation, and autonomous curriculum design.