Skip to content

Large (Vision) Language Models are Unsupervised In-Context Learners

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a joint inference framework that enables fully unsupervised adaptation of foundation models to new tasks, eliminating the need for labeled examples or manual prompt engineering. Two efficient approximations are derived: unsupervised fine-tuning (LoRA + REINFORCE over joint predictive probability) and unsupervised ICL (iterative self-labeling via in-context prompting). Both methods approach supervised counterparts across NLP, vision, and math reasoning tasks.

Problem

Existing adaptation techniques—ICL, fine-tuning, prompt engineering—all require either labeled examples or significant manual effort. Zero-shot inference makes independent per-example predictions, ignoring cross-instance consistency signals available within a task's unlabeled data.

Method

The core idea is joint inference: instead of independently predicting y for each input x, the framework maximizes p(y₁,...,yₙ | x₁,...,xₙ) simultaneously over N inputs, capturing cross-example dependencies. Since exact optimization is combinatorially intractable, two approximations are developed:

  1. Unsupervised fine-tuning: Parameterizes the task encoder using the foundation model itself with LoRA. Uses a custom low-variance REINFORCE gradient estimator to optimize the joint objective (Eq. 7) with an entropy regularizer on the marginal answer distribution to prevent collapse to a single label. Applicable to open-weight models with output probability access.

  2. Unsupervised ICL: Iterative multi-turn refinement where zero-shot predictions seed the first round, then each subsequent round conditions on N randomly sampled (question, previous-answer) pairs as unlabeled in-context examples—effectively Gibbs sampling over the joint. Requires only token sampling, making it compatible with closed-weight APIs (GPT-4o). Shown to implicitly optimize the same joint objective.

Key Contributions

  • Formal joint inference objective that generalizes zero-shot inference (Jš reduces exactly to zero-shot)
  • Unsupervised fine-tuning via LoRA + structured REINFORCE with variance reduction tailored to the joint objective
  • Unsupervised ICL as an approximate, model-agnostic joint inference method with theoretical connection to Gibbs sampling
  • Demonstration that unsupervised adaptation often matches supervised counterparts without any labels
  • Test-time scaling result: 8B model with unsupervised ICL outperforms zero-shot 70B model on RTE

Results

  • NLP (Llama-3.1-8B, 13 datasets): Unsupervised FT: +23% absolute avg. over zero-shot; matches supervised FT on 6/13 datasets. Unsupervised ICL: +19.2% absolute avg.; matches supervised ICL on 10/13 datasets.
  • GSM8K (Qwen2.5-Math-7B): Unsupervised ICL achieves 91.4% vs. zero-shot 52.2% (+39.2% absolute); outperforms supervised ICL (89.9%).
  • Vision (OpenFlamingo): Unsupervised FT: +14% absolute avg. over zero-shot; +23% on Food101. Unsupervised ICL matches supervised ICL on 5/7 datasets.
  • GPT-4o (closed-weight): Unsupervised ICL: +3% on ImageNet-100, +2.2% on MMMU, +0.8% on MMMU-Pro over zero-shot+CoT.
  • MMLU / MMLU-Pro (Llama-3.1-8B): Unsupervised ICL 66.7%/33.1% vs. zero-shot 65.0%/23.7%.

Limitations

  • Unsupervised fine-tuning is restricted to close-ended tasks (finite answer set Y) due to the task encoder constraint; open-ended generation is not directly supported.
  • Unsupervised fine-tuning requires model weight access and output probabilities, excluding proprietary closed-weight models.
  • Both methods rely on the model already possessing ICL capability; if supervised ICL does not beat zero-shot, the framework cannot improve upon it.
  • Unsupervised ICL generally underperforms unsupervised fine-tuning when weights are accessible.
  • Context length limits how many joint examples can be processed per pass.

Relevance to Vision-Language Models

This work directly applies to VLMs via OpenFlamingo experiments (CIFAR-10/100, Food101, VQA), showing that the joint inference framework transfers to multimodal settings with consistent gains. For researchers tracking VLMs, the unsupervised ICL method is especially significant because it requires only API-level access and works with any VLM, including GPT-4o, enabling label-free task adaptation on vision benchmarks without any supervised data. The framework connects to the broader VLM adaptation literature (Flamingo-style ICL, parameter-efficient tuning) by showing that self-generated labels can substitute for ground truth, reducing the annotation bottleneck that is particularly acute in multimodal domains.