Skip to content

Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

🕒 Published (v1): 2026-06-17 20:39 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; novel paradigm using language to steer vision embeddings for controllable perception

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LIVE (Language-Instructed Vision Embeddings) trains a vision encoder to produce dynamically conditioned embeddings \(z^{(I)} = E_{live}(x, T(q))\) at inference time, guided by a textual instruction \(q\), eliminating the need for a large LLM decoder. Trained on ~16.4M synthetic image-instruction-answer triplets distilled from Gemini 2.0 Flash, LIVE achieves +34 points on MMVP and outperforms LLaVa on GQA with fewer than 10% of its parameters.

Problem

Standard vision encoders (CLIP, SigLIP, DINOv2) produce static, query-agnostic embeddings that cannot adapt to what a downstream task needs to attend to. LLM-centric VLM architectures compensate by routing all language integration through massive decoders, incurring high compute overhead and remaining unable to recover fine-grained visual details missed by the frozen encoder—leading to hallucinations. Prior modulation approaches either use spatial prompts (boxes, circles) that cannot specify which semantic attribute to focus on, or condition on descriptive captions rather than task instructions, risking shortcut learning.

Method

LIVE introduces a language-conditioned vision encoder \(E_{live}\) built on a pretrained SigLIP backbone. At inference, a frozen SigLIP text encoder encodes the task instruction \(q\) into an embedding \(T(q)\), which is projected and injected into the vision transformer alongside the image. The instructed embedding is: $\(z^{(I)} = E_{live}(x,\, T(q))\)$

Training objective. A sigmoid-based alignment loss (following SigLIP) aligns the instructed vision embedding with the text embedding of the correct answer \(a\): $\(\mathcal{L} = -\mathbb{E}_{i,j}\log\frac{1}{1+\exp\!\left(-y_{ij}\bigl(t\,(z_i^{(I)}\cdot z_j^{(T)})+b\bigr)\right)}\)$ where \(y_{ij}\in\{-1,+1\}\) encodes whether \((x_i, q_i)\) pairs with answer \(a_j\); \(t\) and \(b\) are learnable temperature and bias.

Training data. Gemini 2.0 Flash is applied offline to ImageNet images to generate diverse question-answer pairs per image, yielding ~16.4M \((x, q, a)\) triplets. This decouples the expensive LLM inference to data preparation; at inference only \(E_{live}\) and the text encoder are needed. The text encoder remains frozen; only the vision encoder is updated.

Key Contributions

  • A language-conditioned vision encoder paradigm that injects task instructions directly into the encoder rather than into a downstream LLM.
  • An offline LLM-distillation pipeline that synthesizes large-scale image-instruction-answer triplets from ImageNet via Gemini 2.0 Flash.
  • Demonstrating that explicit decoupling of instruction from answer description (unlike caption-conditioned methods) forces genuine visual grounding and avoids shortcut learning.
  • Zero-shot generalization to unseen datasets and instructions never seen during training.
  • Up to 10Ă— parameter reduction compared to LLM-heavy baselines while exceeding their accuracy.

Results

  • MMVP (hallucination benchmark): LIVE achieves 76.3% zero-shot accuracy vs. prior SOTA BRAVE at 42.0% — a +34-point gain. All static CLIP/SigLIP baselines score ≤37.8%.
  • GQA (visual question answering): LIVE with ViT-B-16 reaches 71.2% vs. LLaVa 63.3%, BRAVE 52.7%, InstructBLIP 49.5%, BLIP-2 44.7% — while using <10% of LLaVa/InstructBLIP's parameters (~891M vs. ~13–14B).
  • Knowledge transfer to Gemini oracle (ImageNet†): LIVE closes the gap to Gemini's 100% ceiling to 73–87% across model sizes, vs. 25–38% for static SigLIP and 33–61% for Menon et al.
  • Knowledge transfer (SUN397†/RefCOCO†): LIVE scores 49–53% / 54–59% vs. baselines at 9–26%.
  • LIVE narrows the gap to its Gemini teacher by up to 49 points across instruction-following benchmarks.

Limitations

  • Training data is derived entirely from ImageNet + Gemini annotations; generalization to domains far outside ImageNet distribution is not systematically evaluated.
  • The text encoder is frozen, preventing joint adaptation of vision-language alignment.
  • Training data generation requires access to a proprietary multimodal LLM (Gemini 2.0 Flash); reproducibility depends on API availability.
  • Evaluation is cast as retrieval (top-1 accuracy), not open-ended generation; downstream generative VLM tasks are not directly assessed.
  • A notable gap to the Gemini oracle remains on harder spatial benchmarks (e.g., RefCOCO†: 54% vs. 100%).

Relevance to Vision-Language Models

LIVE directly challenges the dominant design choice of treating the vision encoder as a passive feature extractor and scaling compute in the LLM decoder instead. By moving language-driven control into the encoder itself, it shows that hallucination and fine-grained perception failures—persistent problems in VLMs—are partly encoder-level phenomena addressable without larger decoders. The work demonstrates a parameter-efficient alternative to architectures like LLaVa and InstructBLIP, relevant to anyone studying efficient VLM design, modular multimodal architectures, or the tradeoff between encoder expressivity and decoder capacity. The LLM-distillation approach for synthetic instruction-answer data is also a broadly transferable training recipe for instruction-tuned encoders.