Skip to content

Bridging Vision and Language Concepts through Optimal Transport Semantic Flow

🕒 Published (v1): 2026-06-25 11:24 UTC · Source: Arxiv · link

Why this paper was selected

OT-based visual-textual concept alignment; directly improves interpretable VLM reasoning in CBMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OTF-CBM replaces static cosine-similarity concept activation in Concept Bottleneck Models (CBMs) with a learned cross-modal geometry derived via Inverse Optimal Transport (IoT) and Unbalanced OT, then lifts discrete transport couplings into a continuous velocity field via flow matching. At inference, concept scores are computed from midpoint velocity agreement—no ODE integration required. The result is improved classification accuracy and spatially grounded, semantically faithful concept predictions.

Problem

Existing vision–language CBMs suffer three compounding failures: (i) reliance on CLIP's shared embedding space, which imposes a linguistic geometry misaligned with concept granularity; (ii) global-feature-to-concept inference that smears spatial evidence; and (iii) cosine similarity as a proxy for region-to-concept affinity, which is poorly calibrated for fine-grained, heterogeneous cross-modal spaces. Balanced OT variants (e.g., DOT-CBM) still use fixed hand-crafted costs and enforced mass conservation, causing hallucinated correspondences for background regions and many-to-one matches.

Method

Stage 1 — Learn cross-modal cost via Inverse OT (IoT). A parameterized cost \(c_\theta(x, c) = \langle \boldsymbol{\theta}, \Phi(f_\psi(x), c)\rangle\) is expressed as a weighted combination of \(K\) kernel bases (Euclidean, cosine, RBF, etc.). Given ground-truth region–concept coupling \(\hat{\pi}\) from component annotations, \(c_\theta\) is trained to minimize \(\mathcal{L}_{\text{IoT}} = \|({\pi}_\theta - \hat{\pi}) \odot c_\theta\|_{1,1} + \lambda_1\|\theta\|_1\), where \(\pi_\theta\) is the UOT plan induced by \(c_\theta\).

Stage 2 — Vision–Language Unbalanced OT (VLOT). Patch tokens are clustered via K-means into \(K\) prototype embeddings. An Unbalanced OT plan $\(\pi_\theta = \arg\min_{\pi \ge 0}\left\{\langle c_\theta, \pi\rangle + \varepsilon\,\mathrm{KL}(\pi\|\mu\otimes\nu) + \tau_1\,\mathrm{KL}(\pi\mathbf{1}\|\mu) + \tau_2\,\mathrm{KL}(\mathbf{1}^\top\pi\|\nu)\right\}\)$ is solved with KL-penalized marginals, enabling many-to-one and partial matching. Background prototypes are identified via CLS attention and penalized by \(c'_{kj} = c_{kj} + \lambda_{\text{bg}}\mathbf{1}(k\in B)\).

Stage 3 — OT-consistent flow matching. Transport displacements \(u = x_1 - x_0\) (concept embedding minus projected prototype, sampled from \(\pi_{\text{VLOT}}\)) supervise a conditional velocity field \(v_\phi\) via \(\mathcal{L}_{\text{FM}} = \mathbb{E}[\|v_\phi(x_t, t, \text{cond}) - u_t\|^2]\) with straight-line paths \(x_t = (1-t)x_0 + tx_1\).

Inference — velocity-based concept activation. At \(t=0.5\) (variance-maximizing midpoint of the Schrödinger bridge), concept alignment is scored as \(S_{k,j} = -\|v_\phi(x_{1/2}^{(k,j)}, 0.5, \text{cond}) - u^{(k,j)}\|^2\). Concept activation \(a_j = \frac{1}{K}\sum_{k\in\text{TopK}} S_{k,j}\) is passed through LayerNorm and a linear classifier.

Key Contributions

  • Learned cross-modal geometry: IoT infers \(c_\theta\) from annotated region–concept correspondences rather than using fixed cosine/Euclidean distances.
  • Unbalanced OT with foreground suppression: UOT with KL-penalized marginals handles many-to-one and background-to-none matches; CLS-attention-guided cost penalty suppresses background noise.
  • Velocity-based concept activation: Flow matching over OT-derived displacements replaces static similarity; midpoint velocity agreement provides geometry-aware concept scores without ODE integration at inference.
  • Theoretical grounding: Schrödinger bridge midpoint lemma justifies \(t=0.5\) as the maximally informative evaluation point.

Results

Experiments across CUB-200-2011, AwA2, ImageNet-1K, CIFAR-100, and Places365; baselines include classical CBM, P-CBM, Label-Free CBM, LaBo, PCBM-h, and DOT-CBM. Cross-modal flow quality evaluated on TRE, VMSE, MCR, and NPE metrics.

  • OTF-CBM outperforms state-of-the-art CBMs in classification accuracy across all five benchmarks (specific numbers truncated in the provided text).
  • Achieves lower TRE (transport reconstruction error) and VMSE, and higher MCR (directional flow consistency) than prior methods, indicating superior semantic flow geometry.
  • NPE reduction demonstrates fewer reversed flow directions compared to fixed-cost baselines.
  • Background-replacement robustness tests show OTF-CBM predictions rely more on object evidence than context (Places365 probes contextual bias).

(Full numerical tables were not included in the provided text excerpt.)

Limitations

  • Requires component-level annotation data to construct ground-truth coupling \(\hat{\pi}\) for IoT training, limiting scalability to datasets without part annotations.
  • K-means prototype clustering introduces a hyperparameter \(K\) and assumes spatially coherent regions, which may fail for cluttered or abstract scenes.
  • IoT cost learning and UOT solve are more computationally intensive than cosine similarity baselines; training uses two RTX 3090 GPUs.
  • Flow matching training adds a separate stage atop OT, increasing pipeline complexity and training stages.
  • ODE-free inference relies on single-step midpoint velocity, which may be less reliable if the velocity field is poorly conditioned away from training distribution.

Relevance to Vision-Language Models

OTF-CBM directly challenges the dominant paradigm of reusing CLIP's shared embedding space for cross-modal grounding, a foundational assumption in VLM-based interpretable models. The Inverse OT mechanism for learning data-driven cross-modal geometry is broadly applicable beyond CBMs—anywhere VLMs rely on cosine similarity for region-to-concept or patch-to-token alignment (e.g., grounding, open-vocabulary detection). The flow-matching formulation for semantic transport offers a differentiable, continuous alternative to attention-based cross-modal fusion, connecting to emerging directions in dynamic feature routing within VLMs. For researchers tracking VLMs, this work provides a geometric lens for diagnosing and correcting alignment failures that arise from modality gap and mass imbalance in heterogeneous encoder spaces.