Domain Generalization via Text-Anchored Information Bottleneck¶

🕒 Published (v1): 2026-07-02 03:31 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Text-anchored information bottleneck for domain generalization; VLM-grounded invariant representation method

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Visual expressiveness in VLM encoders entangles domain-invariant semantics with spurious domain cues, hurting generalization. This paper proposes using frozen CLIP text embeddings as class-conditional variational priors under a Conditional Entropy Bottleneck (CEB) to suppress domain-specific variation while preserving core semantics. The approach achieves state-of-the-art domain generalization across six benchmarks and multiple backbone architectures without relying on any visual guidance.

Problem¶

Standard domain generalization (DG) methods using VLMs assume preserving expressive visual representations improves robustness. The paper shows the opposite: visual encoders encode substantial domain-specific information $I(Z; \text{domain})$ alongside class semantics $I(Z; \text{class})$. Under finite model capacity $C$, this entanglement satisfies $I(S; X_\text{inv}) + I(S; X_\text{sp}) \leq C$, meaning learning from domain-contaminated visual guidance consumes capacity that should encode invariant semantics. Prior IB-based DG methods use uninformative priors (e.g., $\mathcal{N}(0, I)$) or visually-derived priors that inherit domain bias.

Method¶

Text-Anchored Information Bottleneck (TAIB) optimizes the Conditional Entropy Bottleneck objective: $$\mathcal{L}_\text{CEB} = -I(Z;Y) + \beta I(Z;X|Y)$$ by anchoring to frozen CLIP text embeddings $T = [t_1, \ldots, t_K]^\top$ as class-conditional priors.

Three tractable losses are derived:

Semantic distillation ($\mathcal{L}_\text{sem}$): maximizes $I(Z;Y)$ via contrastive cross-entropy between image features and text class prototypes: $$\mathcal{L}_\text{sem} = -\log \frac{\exp(\cos(z, t_k)/\tau)}{\sum_{k'} \exp(\cos(z, t_{k'})/\tau)}$$
Bottleneck compression ($\mathcal{L}_\text{comp}$): minimizes $I(Z;X|Y)$ by maximizing intra-class resultant vector length (concentration): $$\mathcal{L}_\text{comp} = -\frac{1}{|K_\mathcal{B}|}\sum_{k \in K_\mathcal{B}} \|\bar{z}_k\|$$
Alignment ($\mathcal{L}_\text{align}$): aligns each class mean toward its text anchor: $$\mathcal{L}_\text{align} = -\frac{1}{|K_\mathcal{B}|}\sum_{k \in K_\mathcal{B}} \cos\!\left(t_k, \frac{\bar{z}_k}{\|\bar{z}_k\|}\right)$$

The final objective is $\mathcal{L} = \mathcal{L}_\text{sem} + \beta_1 \mathcal{L}_\text{align} + \beta_2 \mathcal{L}_\text{comp}$ ($\beta_1=0.1$, $\beta_2=1.0$). The variational approximation uses von Mises–Fisher distributions on the unit hypersphere (matching CLIP's $\ell_2$-normalized embedding space), with $q(Z|y=k)$ centered at text anchor $t_k$ and $p(Z|x)$ centered at image feature $z$.

Key Contributions¶

Empirically demonstrates that visual encoders encode domain-specific information (via mutual information, t-SNE, and Lipschitz constant measurements), while text encoders encode near-zero domain information across multiple architectures (ViT, CLIP-Image, DINOv2 vs. MPNet, MiniLM, CLIP-Text).
Provides an information-theoretic argument: visual guidance increases $I(S; X_\text{sp})$, consuming capacity from $I(S; X_\text{inv})$; text guidance avoids this.
Proposes the first class-conditional, text-anchored IB prior for DG, replacing uninformative Gaussian priors with semantically structured CLIP text embeddings.
Demonstrates architecture-agnostic SOTA performance across six DG benchmarks (VLCS, PACS, OfficeHome, TerraIncognita, DomainNet, NICO++) over multiple backbone architectures.
Shows that existing KD-based (RISE, VL2V) and CLIP-specialized (CLIPood) methods exhibit architectural bias and degrade outside their original setup, while TAIB generalizes uniformly.

Results¶

Against baselines LP, MIRO, SAGM, GESTUR, INSURE, CLIPood, RISE, VL2V on ResNet-50 (ImageNet-1k pretrained):

Average across 6 benchmarks: Ours 75.4 vs. next-best VL2V ~68+ (exact second-best average not fully visible in truncated table)
PACS: Ours 96.9 vs. RISE 89.4, VL2V 86.7, CLIPood 88.8
OfficeHome: Ours 79.0 vs. VL2V 74.4, RISE 71.6
DomainNet: Ours 59.9 vs. VL2V 53.5, GESTUR 51.3
VLCS: Ours 81.7 (tied with RISE), vs. MIRO 79.0
TerraIncognita: VL2V 58.3 shown as second-best (ours not visible in truncated table)

ViT-B/16 results are partially visible; ours outperforms LP, MIRO, CLIPood on VLCS, PACS.

Limitations¶

Relies on simple class-name prompts ("a photo of a [class]") for text anchors, which may be insufficiently discriminative for fine-grained or ambiguous categories.
Fixed text anchors from a frozen CLIP text encoder; no mechanism to adapt the text prior for out-of-CLIP-vocabulary domains or classes.
Hyperparameters $\beta_1, \beta_2$ are tuned on PACS/ResNet-50 and transferred; sensitivity is deferred to appendix.
Evaluated only on image classification; applicability to detection or segmentation DG is not demonstrated.
The ViT-B/16 results table is incomplete in the paper as presented, limiting full comparison visibility.

Relevance to Vision-Language Models¶

This paper directly challenges a core assumption in VLM-based domain generalization—that preserving rich visual representations from CLIP benefits robustness—and shows empirically that CLIP's text encoder is the more domain-stable modality. It reframes the role of VLMs in DG: rather than distilling or fine-tuning visual features, the text encoder's embeddings serve as a fixed, domain-invariant semantic prior under an information-theoretic framework. This shifts the design focus from visual representation preservation to textual supervision design, with implications for prompt tuning, knowledge distillation, and any downstream task where CLIP's multimodal structure is used for generalization. The work is relevant to researchers studying CLIP adaptation, zero-shot robustness, and the use of language as a structural inductive bias in visual learning.