Skip to content

Reversible Primitive–Composition Alignment for Continual Vision–Language Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; reversible primitive-composition alignment for tight-budget continual VL learning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

COMPO-REALIGN addresses the phenomenon where continual VLMs preserve primitive (attribute/object) recognition but catastrophically forget compositional binding under sequential adaptation with tight rehearsal budgets and no task IDs. The method enforces reversible primitive–composition alignment through a Cayley-transform orthogonal composer, a multi-positive InfoNCE objective, and a spectral trust region that clips gradient steps when alignment sensitivity inflates. It sets new state of the art on compositional DIL and multi-domain MTIL retrieval, improving Avg R@1 by +2.4 and reducing forgetting by ~40%.

Problem

Under sequential task adaptation, VLMs retain first-order primitive representations (individual attributes, objects) while losing higher-order compositional binding—the ability to correctly associate a composition embedding with its constituent primitives. This structural degradation is especially severe when rehearsal budgets are tight (e.g., \(\leq\)64 samples/task) and task IDs are unavailable at test time. Existing continual VL methods (replay, regularization, adapters) focus on average task/domain accuracy rather than preserving the internal binding geometry.

Method

COMPO-REALIGN trains a lightweight head \((\Theta, A, \varphi)\) on top of frozen CLIP-style encoders \(f_v, f_t\).

Reversible Composer. Primitive embeddings \(e_{p,i} = f_t(p_i)/\|f_t(p_i)\|_2\) are adapted and averaged: $\(\bar{u} = \frac{1}{m}\sum_{i=1}^m \varphi(Ae_{p,i}), \quad \hat{e}_c = \frac{R(\Theta)\bar{u}}{\|R(\Theta)\bar{u}\|_2}\)$ where \(R(\Theta) = (I-S)(I+S)^{-1}\), \(S = \tfrac{1}{2}(\Theta - \Theta^\top)\) is the Cayley transform producing an orthogonal matrix, making the primitive→composition map bijective by construction (\(R(\Theta)^{-1} = R(\Theta)^\top\)).

Multi-Positive InfoNCE. The textual composition \(e_c\) and the composed embedding \(\hat{e}_c\) are treated as joint positives for the image \(z_v\) in a symmetric two-positive InfoNCE: $\(\mathcal{L}_{v\to c} = -\frac{1}{B}\sum_i \log \frac{\exp(s(z_{v,i}, e_{c,i})/\tau) + \exp(s(z_{v,i}, \hat{e}_{c,i})/\tau)}{\sum_j \left[\exp(s(z_{v,i}, e_{c,j})/\tau) + \exp(s(z_{v,i}, \hat{e}_{c,j})/\tau)\right]}\)$ Buffered paraphrases \(e'_c\) extend the numerator as additional positives without a new loss term.

Spectral Trust Region. The Jacobian \(J_p = \partial s(z_v, \hat{e}_c)/\partial\,\text{vec}(U_p) \in \mathbb{R}^{1\times md}\) is approximated via 1–2 power iterations on a random unit vector. Gradients are rescaled by \(\alpha = \min\{1,\,\gamma/\hat{\sigma}_{\max}\}\) whenever the estimated spectral radius \(\hat{\sigma}_{\max}\) exceeds threshold \(\gamma\), preventing harmful alignment sensitivity without adding a penalty term.

Training updates only \((\Theta, A, \varphi)\); encoders remain frozen throughout all tasks.

Key Contributions

  • Identifies and quantifies the primitive-stable / composition-degrading phenomenon in continual VLMs via Compositional Retention Ratio (CRR), cycle-consistency error (CCE), and Jacobian spectral indicators.
  • Shows empirically that composition error correlates strongly with Jacobian spectral radius (\(r=-0.82\)) and CCE, establishing a diagnostic framework linking alignment geometry to downstream behavior.
  • Proposes COMPO-REALIGN: a minimal three-component recipe (reversible composer via Cayley transform, multi-positive InfoNCE, spectral clipping) requiring no task IDs and operating under strict memory budgets.
  • Demonstrates that a text-centric micro-buffer (symbolic paraphrastic anchors) is more memory-efficient than image replay for preserving compositional structure.
  • Achieves SOTA on compositional DIL, multi-domain MTIL retrieval, and continual VQA under the same rehearsal budgets as prior work.

Results

  • Avg R@1 (Image→Text): 58.8 vs. 56.4 (C-CLIP) and 56.0 (DIKI), a +2.4 absolute improvement over the strongest prior.
  • Forgetting (AF): 3.2 vs. 5.0–5.1 for C-CLIP/DIKI, a ~36–40% relative reduction.
  • CRR: 0.91 vs. 0.88–0.89 for best prior methods.
  • ZSTD: −1.3 (smallest magnitude), vs. −1.9 for nearest competitor, indicating minimal harm to zero-shot transfer.
  • Continual VQA (CLOVE-scene / CLOVE-function / VQACL): 65.1 / 60.4 / 56.8 vs. 63.5 / 59.2 / 55.4 for CL-MoE (next best), with AF 3.6 vs. 4.7.
  • Ablation — composed positive is the main driver: removing it costs −1.9 R@1, −0.04 CRR, +0.8 AF.
  • Ablation — Cayley core matters: replacing with a linear mix reduces CRR by −0.03 and hurts retrieval/VQA by ~1–1.6 points.

Limitations

  • Requires primitive decomposition at training time: each sample must supply (image, composition text, \(\{\)primitive texts\(\}\)), which presupposes extractable primitive annotations or templates.
  • Frozen encoders limit adaptation capacity: the approach updates only a small head; tasks requiring deep backbone adaptation are not addressed.
  • Primitive inventory assumed stable across tasks: the exploratory setup and benchmarks share the same attribute/object vocabulary; domain shifts that introduce novel primitives are not studied.
  • Text micro-buffer depends on paraphrase availability: the buffer benefits rely on linguistically diverse templates, which may not exist for all domains.
  • Orthogonal core operates on mean-pooled primitives: the mean aggregation discards ordering and multiplicities, which may fail for compositions with repeated or ordered primitives.

Relevance to Vision-Language Models

This work directly targets CLIP-style contrastive VLMs deployed in non-stationary settings—a growing practical concern as VLMs are fine-tuned for specialized domains in sequence. The finding that primitives remain stable while compositional binding degrades is a non-obvious structural insight that challenges the assumption that overall task accuracy implies representation integrity. The spectral diagnostics (Jacobian radius, CCE, Grassmann subspace drift) are broadly applicable tools for auditing VLM geometry under any fine-tuning regime, not just continual learning. For researchers tracking VLMs, this connects to the broader question of how contrastive embedding spaces maintain compositional semantics under distribution shift—a property relevant to generalization, robustness, and compositional zero-shot transfer.