Skip to content

Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models

🕒 Published (v1): 2026-06-18 08:40 UTC · Source: Arxiv · Venue: ECCV · link

Why this paper was selected

ECCV; text-in-image geometric anchoring directly addresses fine-grained spatial reasoning gap in MLLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Timage addresses the spatial misalignment between text queries and image content in MLLMs by rendering the natural-language query as a typeset overlay directly onto the image, using a Constrained Schrödinger Bridge (cSB) generative process to determine optimal placement and appearance. This input-level intervention is training-free, plug-and-play, and backbone-agnostic. On VMCBench, a 7B backbone with Timage achieves 87.7% average accuracy, surpassing GPT-4o, Qwen2-VL-72B, and full fine-tuning baselines.

Problem

Standard MLLMs receive the query as a discrete token stream and the image as a continuous tensor; cross-modal alignment is left to implicit cross-attention, which reliably fails under fine-grained spatial reasoning, dense text reading, or chained logical steps. Weight-space adaptation (LoRA, adapters) degrades general competence; input-space visual prompting (VPT) injects semantically opaque vectors that cannot exploit the backbone's OCR or spatial grounding priors. No prior method explicitly co-localizes a human-readable semantic cue with the image geometry at the input level.

Method

Timage converts multimodal understanding into an input-alignment problem: given image \(x_0\) and query \(q\), it synthesizes an inscribed image \(x_1 = x_0 \oplus \mathcal{T}\) where \(\mathcal{T}\) is a typeset overlay of \(q\).

Energy function. A candidate layout \(\mathcal{T}\) is scored by: $\(E(\mathcal{T}; x_0, q) = \lambda_{\text{zon}}\, E_{\text{zon}}(\alpha(\mathcal{T}); x_0, q) + \lambda_{\text{sty}}\, E_{\text{sty}}(\beta(\mathcal{T}))\)$ where \(\alpha \in [0,1]^{H \times W}\) is the placement field and \(\beta\) encodes appearance.

Placement (Ezon). A query-relevance heatmap \(H_q\) is extracted from cross-attention of a frozen VLM between text embedding \(e_q\) and visual patches. Placement is pulled toward high-\(H_q\) regions (soft semantic energy \(E_{\text{sem}}(\alpha) = -\int \alpha(u) H_q(u)\,du\)) while a hard Admissible Mask \(M_{\text{cand}}\) blocks any placement on salient foreground objects (via Mask R-CNN instance masks and a saliency map).

Style (Esty). Font size is governed by a Query-Adaptive Ink Budget: \(\text{ink}(\mathcal{T}) = \kappa \cdot \text{Length}(q) \cdot \text{Scale}(\alpha)\), adapting glyph size proportionally to query length and available area.

Constrained Schrödinger Bridge (cSB). Overlay generation is formulated as an entropic optimal-transport problem. A drift-diffusion SDE with velocity field \(v_\theta\) is discretized via a Projected Euler–Maruyama scheme: $\(\mu_{t+\Delta t} = \mathcal{P}_\Omega\!\Big(\mu_t + v_t\,\Delta t + \sqrt{2\mathcal{F}(t)}\,\Delta W_t\Big)\)$ The projection \(\mathcal{P}_\Omega\) onto \(\Omega = M_{\text{cand}}\) enforces hard non-occlusion; the drift \(v_\theta\) learns semantic direction. Two-stage generation: Stage 1 (Region Search) produces \(\alpha\); Stage 2 (Appearance Shaping) fixes glyph size and style via the ink-budget regularizer.

Training objective. \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{cSB}} + \lambda \mathcal{L}_{\text{task}}\), where \(\mathcal{L}_{\text{cSB}}\) is a flow-matching velocity regression loss and \(\mathcal{L}_{\text{task}} = -\log P(y_{\text{gt}}, y_{\text{pred}} = \mathcal{M}(x_1))\) backpropagates downstream task signal through the discrete trajectory via a straight-through estimator.

At inference, \(K=5\) independent overlay samples are generated and answers are selected by majority vote (self-consistency voting).

Key Contributions

  • Timage paradigm: renders the query as an optimized typeset overlay on the image, unifying both modalities through a single visual input channel without any MLLM weight modification.
  • Constrained Schrödinger Bridge (cSB): a principled generative formulation that enforces hard spatial feasibility (non-occlusion) via projected SDE dynamics and soft semantic alignment via a query-relevance heatmap.
  • Query-Adaptive Ink Budget: automatically scales glyph size to query length and placement area, ensuring legibility without crowding.
  • State-of-the-art on VMCBench (87.7%) in a zero-shot, training-free, inference-time regime with a 7B backbone, outperforming all proprietary and PEFT baselines.
  • Architecture-neutral generalization: demonstrated across five backbones (LLaVA-1.5-7B through Qwen3-8B) with consistent +4.8% to +8.6% gains.

Results

  • Overall VMCBench accuracy: Timage (Qwen2.5-VL-7B, K=5) = 87.7% vs. GPT-4o 80.3% (+7.4%), Qwen2-VL-72B 85.0% (+2.7%), Full Fine-Tuning 85.2% (+2.5%), LoRA (rank=64) 84.5% (+3.2%), CVPT 83.5% (+4.2%).
  • Category gains over Qwen2.5-VL-7B base (79.1%): General +5.7% (→91.3%), Reasoning +7.6% (→75.8%), OCR +3.2% (→98.5%), Doc&Chart +6.1% (→92.6%).
  • Ablation increments: Semantic Pull +2.5%, Admissible Mask +3.0% (single largest), Style Regularization +1.4%, Stochastic Ensemble (K=5) +2.2%.
  • Cross-architecture: average improvement +6.1% across five backbones; Qwen3-8B 87.3%→92.1% (+4.8%), LLaVA-1.5-7B 51.8%→57.4% (+5.6%).
  • Versus heuristic overlays: best heuristic (Saliency-Based Placement) = 80.5%; Timage = 87.7% (+7.2%).
  • Parameter efficiency: Timage surpasses LoRA scaled to 40M parameters (88.3%) while using far fewer trainable parameters.

Limitations

  • Evaluated exclusively on VMCBench (VQA format, multiple-choice); generalization to open-ended generation or other task formats is not demonstrated.
  • The cSB generator requires training on 16 H20 GPUs for 100 epochs; inference cost is multiplied by \(K=5\) overlay samples per query.
  • Relies on external segmentation (Mask R-CNN) and saliency models for admissible mask construction, adding pipeline complexity and potential failure modes.
  • The straight-through estimator used to backpropagate through the non-differentiable projection is an approximation; gradient fidelity is not analyzed.
  • Qualitative analysis is limited to one example; failure modes of the cSB placement are not systematically examined.

Relevance to Vision-Language Models

Timage directly targets a fundamental bottleneck in current VLMs: the lack of explicit spatial grounding between the linguistic query and the image domain. By framing VLM adaptation as input reconstruction rather than parameter modification, it offers a plug-and-play alternative to LoRA/adapters that preserves backbone generality and sidesteps distribution-shift degradation — a practically significant finding for the VLM fine-tuning literature. The use of cross-attention maps from a frozen VLM to build the placement heatmap also demonstrates a novel introspective use of VLM internals at inference time. The strong gains on OCR and document-understanding tasks are particularly relevant for researchers working on text-rich visual understanding, where query-to-region alignment failures are most costly.