Skip to content

Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Bisecle is a continual learning framework for video-language models that addresses catastrophic forgetting and update conflict when only a small subset of parameters (projection/adapter layers) are trainable across sequential VideoQA tasks. Drawing from hippocampal neurobiology, it combines multi-directional supervision (rapid binding) and contrastive prompt learning (pattern separation) to achieve state-of-the-art results on three VideoQA benchmarks.

Problem

Parameter-efficient continual learning for VLMs—where the LLM and visual encoder are frozen and only lightweight adapters are updated—faces two compounding challenges: (1) catastrophic forgetting, where unidirectional answer-prediction training insufficiently anchors cross-modal associations from prior tasks in scarce trainable parameters; and (2) update conflict, where heterogeneous gradient signals from diverse video tasks compete for the same bottleneck parameters, causing inter-task interference.

Method

The backbone is LLaMA-Adapter (LLaMA-2-7B + ViT-L/14), with only projection layers and prompts trainable.

Multi-directional supervision (rapid binding): Beyond the standard answer prediction loss \(L_A = -\log P(A|V,Q)\), two auxiliary losses enforce bidirectional cross-modal binding: - Question prediction: \(L_Q = -\log P(Q|V,A)\) — reconstructs the question from video and answer, strengthening causal understanding. - Video prediction: \(L_V\) via InfoNCE-style mutual information maximization between frame visual tokens \(v_{k+1}\) and LLM output representations \(h_k\), approximating \(-\log P(V|Q,A)\).

Contrastive prompt learning (pattern separation): Shared task-aware prompt matrix \(P \in \mathbb{R}^{(N_p \times L_p) \times D}\) is injected into transformer layers. A per-task learnable type embedding \(e^t \in \mathbb{R}^D\) acts as a non-overlapping subspace. For each training sample, the prompt is reweighted by the mean-pooled question representation \(\tilde{p}^t = q^t \cdot P^\top \cdot P\), and a contrastive loss \(L_P\) aligns \(\tilde{p}^t\) with \(e^t\) against all other task embeddings (temperature \(\tau\)), explicitly separating task-specific knowledge in prompt space.

Total loss: \(L = L_A + L_Q + L_V + \gamma L_P\).

Key Contributions

  • Neurobiological reframing of continual learning challenges: maps hippocampal rapid binding → multi-directional supervision; dentate gyrus pattern separation → contrastive prompt learning.
  • Multi-directional supervision module that introduces question and video reconstruction objectives to deepen cross-modal binding without adding inference overhead.
  • Contrastive prompt learning that isolates task-specific knowledge in shared prompts via learnable task-type embeddings, eliminating the need for explicit task-matching at test time.
  • SOTA results across three VideoQA continual learning benchmarks with minimal additional parameters (~104k–51k extra over backbone).

Results

  • NExT-QA: 62.37% Acc / 5.34% Fog — vs. ColPro (best prior): 55.14% / 7.43%; vs. backbone: 46.58% / 13.83%.
  • DramaQA: 71.49% Acc / 10.37% Fog — vs. ColPro: 71.24% / 12.64%; vs. backbone: 60.99% / 24.39%.
  • STAR: 52.16% Acc / 7.60% Fog — vs. ColPro: 48.67% / 8.13%; vs. backbone: 46.89% / 11.54%.
  • Ablation: \(L_Q\) (question prediction) yields the largest single-component gain; all three auxiliary losses contribute positively.
  • Scales across LLaMA 1B/7B/13B; consistent improvement with notably reduced forgetting (13B: 4.52% → 3.03%).
  • Robust under low-data regimes (outperforms backbone at all training data fractions tested).
  • Increasing prompt injection layers from 8 → 32 monotonically improves both Acc and Fog.

Limitations

  • Evaluated only on multiple-choice VideoQA; applicability to open-ended video captioning or grounding tasks is untested.
  • Task identity must be known at training time to assign task type embeddings \(e^t\); open-world task-agnostic deployment is not addressed.
  • Task order for DramaQA is explicitly chosen to maximize forgetting, which may inflate reported gains relative to arbitrary real-world orderings.
  • Video prediction loss uses a mutual-information approximation (InfoNCE over frame tokens) rather than actual visual generation; fidelity of cross-modal binding is indirect.
  • Backbone is LLaMA-Adapter with ViT-L/14; generalization to more recent VLM architectures (e.g., LLaVA-style or instruction-tuned models) is not demonstrated.

Relevance to Vision-Language Models

Bisecle directly tackles a key gap in deploying large VLMs in real-world streaming settings—continual adaptation without retraining the full model or storing large replay buffers. The neurobiological framing provides a principled lens for designing auxiliary losses and prompt regularization that goes beyond heuristic approaches. For researchers tracking VLMs, this work is particularly relevant to the emerging subfield of lifelong/continual multimodal learning and demonstrates that lightweight architectural augmentations (multi-directional losses + contrastive prompt constraints) can substantially reduce forgetting in frozen-backbone multimodal systems. It also surfaces the tension between parameter efficiency and plasticity that will become increasingly important as VLMs are deployed on edge devices or in dynamic environments.