Skip to content

When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs

🕒 Published (v1): 2026-07-08 13:31 UTC · Source: Arxiv · link

Why this paper was selected

Graph-based attribute reasoning for VLM calibration — addresses overconfidence in prompt-tuned models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARGTCA replaces the flat, similarity-ranked attribute selection in TCA with a graph-structured selection pipeline: a Symbolic Attribute Graph (SAG) over (class, attribute) pairs is trained offline via a Graph Attention Network (GAT) and supervised contrastive loss, then two principled criteria—intra-class diversity (DIV) and inter-class discrimination (DISC)—select \(M'\) attributes per class. Across 9 benchmarks with CLIP ViT-B/16, ARGTCA-DIV reduces average Expected Calibration Error (ECE) by ~37% over the best prior baseline.

Problem

Test-time prompt tuning (TPT) improves zero-shot accuracy in VLMs but consistently degrades calibration through entropy minimization–driven overconfidence. TCA partially mitigates this by initializing prompts with LLM-generated visual attributes and applying contrastive regularization, but selects attributes via flat cosine-similarity ranking, producing two structural failures: (1) near-synonymous intra-class attributes cluster tightly on the hypersphere, and (2) cross-class ubiquitous attributes (e.g., "mammal" spanning leopard, dolphin, beaver) carry no discriminative capacity yet receive no suppression.

Method

Phase 1 (offline). Given \(K\) classes each with \(M = 10\) LLM-generated attributes, a Symbolic Attribute Graph (SAG) is built with \(N = K \times M\) nodes. Each node \((c_k, a_j)\) is initialized as the \(\ell_2\)-normalized EOS hidden state of "a {a_j} of a {c_k}" from the frozen CLIP text encoder. Edges are defined symbolically: \(E_\text{intra}\) connects all attribute nodes sharing the same class (\(c_i = c_j\)); \(E_\text{inter}\) connects nodes sharing the same attribute token across different classes (\(a_i = a_j, c_i \neq c_j\)). A 2-layer, 4-head GAT propagates information across these edges:

\[h_i^l = \text{ELU}\left(\Big\|_{h=1}^{H} \sum_{j \in \mathcal{N}(i)} \alpha_{ji}^{(l,h)} W_h h_j^{l-1}\right)\]

The GAT is trained with supervised contrastive loss (SupCon), treating same-class nodes as positives and all inter-class nodes as negatives, for 100 epochs (completes in under 2 minutes per dataset).

Attribute selection. From the learned embeddings \(\tilde{h}^{(L)}\): - ARGTCA-DIV: selects the attribute pair \((a_i^k, a_j^k)\) per class with minimum cosine similarity in graph-refined space — maximizing intra-class semantic complementarity. - ARGTCA-DISC: scores each node by mean angular distance from all other-class nodes and takes the top-\(M'\) — maximizing inter-class separation.

Phase 2 (test-time). Selected \(M'=2\) attributes per class initialize TCA's prompt tuning procedure unchanged; graph computation adds zero test-time cost.

Key Contributions

  • Identifies two structural failures in TCA's flat attribute selection: intra-class collinearity and cross-class ubiquity, both invisible to cosine-similarity ranking.
  • Proposes the Symbolic Attribute Graph (SAG) with symbolic edge rules encoding both intra-class attribute relations and cross-class attribute sharing.
  • GAT trained offline via supervised contrastive loss produces relational embeddings that capture inter-attribute dependencies without task supervision.
  • Two complementary selection criteria (DIV, DISC) derived from graph embeddings, both at zero additional test-time cost.
  • Evaluated across 9 benchmarks spanning fine-grained recognition, texture, and domain-shifted imagery.

Results

  • ARGTCA-DIV: best average ECE of 4.45%; reduces miscalibration vs. C-TPT (−26.7%), TCA (−31.5%), O-TPT (−42.4%), A-TPT (−48.0%); improves average accuracy simultaneously.
  • ARGTCA-DISC: best average accuracy of 64.68%, ECE of 5.90%; reduces ECE vs. A-TPT by 31% while matching its accuracy within 0.2%.
  • Caltech101 (ViT-B/16): ARGTCA-DISC ECE = 1.62% vs. TCA 2.56%, O-TPT 3.62%, A-TPT 2.67%, TPT 4.59%.
  • Largest gains on Aircraft (ECE 1.21% vs. A-TPT 8.19%), Flower (2.88% vs. 10.38%), Food101 (1.75% vs. 2.02%).
  • Ablation (DTD): removing edge structure raises DIV ECE from 3.17→4.91; skipping GAT training raises DISC ECE from 6.36→8.95; random selection raises DIV ECE to 5.16, isolating Div/Disc criteria as the operative mechanism.
  • Hyperparameters (\(\alpha, \beta\)) stable within ~2% ECE across full sweep.

Limitations

  • Only evaluated on CLIP ViT-B/16; generalization to other VLM backbones (e.g., ViT-L/14, LLaVA-family) is not demonstrated.
  • ARGTCA-DIV selects only \(M'=2\) attributes per class, which may be insufficient for classes with many independent visual dimensions.
  • LLM-generated attributes (GPT-4) are reused from TCA without ablation of attribute quality or alternative LLMs.
  • ARGTCA-DISC is less consistent across all nine datasets than DIV, underperforming on texture-heavy datasets (DTD ECE 6.36% vs. DIV 3.17%).
  • The offline graph must be retrained per dataset; scalability to large-vocabulary datasets is not addressed.

Relevance to Vision-Language Models

This paper directly targets a known failure mode of test-time prompt tuning in CLIP-family VLMs: that entropy minimization decouples accuracy from calibration and drives overconfidence. By showing that the geometric structure of textual feature space—not just predictive accuracy—governs calibration, it advances understanding of how prompt design affects uncertainty in VLMs. The graph-based attribute reasoning framework generalizes the emerging line of work (C-TPT, O-TPT, TCA) that treats calibration as a feature-space geometry problem rather than a post-hoc scaling problem. For researchers tracking VLMs, this work provides a principled, zero-test-time-cost mechanism for reliable uncertainty estimation, which is critical for deployment in safety-sensitive settings.