Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders¶

🕒 Published (v1): 2026-07-07 16:11 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; Daniel Cohen-Or/Or Patashnik; novel mechanistic insight into VLM localization signals

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

When VLMs are used as non-generative condition encoders inside diffusion-based editing pipelines, their localization ability degrades sharply: the same model achieves 89% localization accuracy autoregressively but only 57.5% when restricted to a single forward pass. This paper introduces Analysis-by-Proxy, a framework that trains a lightweight Q-Former proxy on the VLM's intermediate hidden states to reveal where spatial signals actually live. The core finding is that localization information is encoded in mid-layer, input-dependent positions rather than the final layer universally used for conditioning.

Problem¶

Diffusion-based image editing pipelines condition a Diffusion Transformer (DiT) on hidden representations extracted from a VLM in a single forward pass, discarding autoregressive decoding. Existing pipelines extract features from a predefined, input-independent set of layers (typically final-layer tokens, fixed layer subsets, or cross-layer pooling). This mismatch between the VLM's autoregressive training objective and its single-pass conditioning role causes a 31.5-point localization accuracy drop in complex, multi-entity scenes, producing failure modes including mislocalization, attribute leakage, hallucination, and excessive removal. Prior VLM interpretability methods require autoregressive sampling and thus cannot analyze this non-generative regime.

Method¶

Analysis-by-Proxy probes VLM hidden states indirectly via a supervised auxiliary task:

Proxy architecture: A Q-Former with \(N\) learnable queries \(Q \in \mathbb{R}^{N \times d}\) that cross-attends to a subset of VLM hidden states \(\{H^{(l)}\}_{l \in L}\) (where \(H^{(l)} \in \mathbb{R}^{M \times d}\)) through projected keys and values, followed by a coordinate head predicting bounding box \((x_1, y_1, x_2, y_2)\).
Training: Independent Q-Formers are trained per layer configuration on triplets \((I, T, B)\), where ground-truth bounding boxes \(B\) are obtained by prompting the VLM autoregressively. Loss is \(\mathcal{L} = \lambda_{\text{GIoU}} \mathcal{L}_{\text{GIoU}} + \lambda_{L_1} \mathcal{L}_{L_1}\).
Layer-level analysis: Proxy attention heads are categorized as "decisive" if their queries concentrate mass heavily on one VLM layer (mean rank-1 vs. rank-2 attention mass gap of 0.6). The dominant layer \(l^* = \text{mode}\{l^*_h \mid h \in H\}\) is determined per sample via majority vote over decisive heads.
Token-level analysis: Within \(l^*\), the same decisive heads concentrate attention onto a sparse set of tokens; decoding these tokens reveals they correspond to semantically significant nouns and adjectives in the edit instruction.

Applied to Qwen-Image-Edit (Qwen2.5-VL-7B backbone, 28 transformer layers).

Key Contributions¶

Identification of a 31.5% localization gap between standalone VLM autoregressive performance (89.0%) and its conditioning performance (57.5%) on 200 complex multi-entity scenes.
The Analysis-by-Proxy framework: a principled, training-based method for probing VLM hidden states in the non-generative conditioning regime without modifying the input or relying on generated tokens.
Empirical finding that final-layer representations are poor spatial carriers; mid-layers 15–24 (of 28) encode the strongest localization signal.
Discovery that the informative layer is input-dependent: no fixed layer subset is universally optimal, making static conditioning strategies inherently suboptimal.
Discovery that spatial signal is token-sparse: concentrated almost exclusively in tokens corresponding to semantically meaningful nouns and adjectives of the target edit.
Demonstration that recovering the proxy-predicted bounding box and injecting it as a conditioning input improves edit localization in the downstream pipeline.

Results¶

VLM standalone (autoregressive bounding box prediction): 89.0% localization accuracy on 200-sample multi-entity evaluation set.
Full Qwen-Image-Edit pipeline (single-pass final-layer conditioning): 57.5% localization accuracy — a 31.5 percentage point gap.
Q-Former trained on intermediate layers (L = {15,...,24}, user prompt tokens only): fastest convergence and lowest bounding-box center-distance error vs. proxies trained on final layer (slowest) or early/late layers (poor convergence).
Layer-wise attention sparsity in decisive heads: mean drop of 0.6 between rank-1 and rank-2 attention mass.
Using proxy-recovered bounding boxes as conditioning signal yields improved edit localization in complex scenes (qualitative; no aggregate number reported in the excerpt).

Limitations¶

Analysis is performed on a single backbone (Qwen2.5-VL-7B) and pipeline (Qwen-Image-Edit); generalization to other VLMs or editing architectures is not demonstrated in the provided text.
The evaluation set is small (200 samples) and relies on human evaluation, which may not scale.
Improvement from proxy-recovered signals is demonstrated qualitatively; aggregate quantitative gains of the improved pipeline over the baseline pipeline are not reported in the provided text.
The proxy requires a separate training phase with supervised bounding-box data derived from the VLM's own autoregressive outputs, adding overhead.
The framework addresses localization specifically; other aspects of conditioning quality (style, semantics, instruction following) are out of scope.

Relevance to Vision-Language Models¶

This paper directly illuminates a structural failure mode of VLMs when repurposed from autoregressive generators to non-generative encoders — a paradigm increasingly common in multimodal generative systems. It provides concrete mechanistic evidence that VLM internal representations follow an autoregressive information-flow pattern that does not align with fixed-layer conditioning extraction, contradicting the implicit assumption underlying current DiT-based editing architectures. For researchers studying VLM internals, the Analysis-by-Proxy methodology offers a general probing tool for the single-pass regime, complementing the growing literature on autoregressive VLM interpretability. The finding that spatially grounded information is encoded sparsely at semantically critical tokens has direct implications for cross-attention conditioning design, prompt engineering, and efficient feature extraction in VLM-conditioned generation systems.