Skip to content

MemEIC: A Step Toward Continual and Compositional Knowledge Editing

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MemEIC tackles continual and compositional knowledge editing in large vision-language models (LVLMs), where visual and textual facts must be updated sequentially without catastrophic forgetting. It introduces a hybrid external-internal memory system with modality-separated dual LoRA adapters and a brain-inspired knowledge connector that activates only for cross-modal compositional queries. On the new CCKEB benchmark, MemEIC achieves 80.56% compositional reliability on LLaVA-1.5, outperforming the best baseline by +18.51 points.

Problem

Existing LVLM knowledge editing methods address a single modality in isolation and evaluate only single-step edits. This ignores two critical real-world requirements: (1) continual editing, where a sequence of alternating visual and textual updates must be retained over time without catastrophic forgetting; and (2) compositional reasoning, where a query requires integrating a visual edit (entity re-identification) with a separately stored textual fact update. External memory editors fail because they use text-only retrieval, missing visual cues. Internal methods (LoRA, fine-tuning) store heterogeneous modality edits in a shared parameter space, causing cross-modal interference and representation collapse under sequential edits.

Method

MemEIC operates through three coordinated components:

Query Decomposition. GPT-4o decomposes each input query into a visual sub-query \(Q_v\) and a textual sub-query \(Q_t\), enabling modality-specific downstream processing.

Mem-E (Dual External Memory). Two separate stores: \(M_t\) for textual QA pairs (retrieved via DistilBERT [CLS] cosine similarity) and \(M_v\) for visual image-QA triples (retrieved via a weighted combination of CLIP image embeddings and DistilBERT text embeddings, \(\alpha=0.5\)). For compositional queries, the visual retrieval resolves the entity first, then the entity name is substituted into the textual query for a second retrieval pass. Retrieved evidence is prepended to the LVLM prompt.

Mem-I (Dual LoRA Adapters). Two parallel LoRA adapters (rank 8 each) augment the FFN: a visual adapter \(\theta_v\) (right-hemisphere analogy) and a textual adapter \(\theta_t\) (left-hemisphere analogy). Only the modality-matched adapter is updated per edit and activated at inference; both activate for compositional queries.

Knowledge Connector. A LoRA-augmented self-attention module that activates only when both adapters are active (indicator \(I_{v,t}=1\)). It modifies the Q and K projections: \(Q^\ell = h^\ell W_Q + I_{v,t} \cdot \Delta W_Q^L\), enabling cross-modal token interaction for compositional reasoning while remaining an identity operation for unimodal queries.

Training uses two stages: (1) freeze the LVLM, train external memory retrieval; (2) freeze external memory, train the Knowledge Connector using an adversarial retriever that mixes factual and counterfactual evidence to prevent over-reliance on retrieved context.

Key Contributions

  • CCKEB benchmark: first benchmark for continual and compositional multimodal knowledge editing, extending VLKEB with paired visual+textual edits and a sequential evaluation protocol (500 edit steps, tested at gaps 0/10/20/50/100).
  • CompRel metric: measures whether a model correctly integrates a visual and textual edit to answer a single compositional query.
  • MemEIC framework: hybrid external-internal editor combining dual external memory, modality-specific dual LoRA adapters, and a selective Knowledge Connector.
  • Ablations confirming that visual cues in retrieval, modality separation in adapters, and the connector each independently contribute significant gains.

Results

  • MemEIC on LLaVA-1.5 (gap-averaged): visual reliability 98.93%, textual reliability 92.48%, compositional reliability 80.56%.
  • vs. WISE (best prior continual method): +16.94 visual reliability, +32.35 compositional reliability (gap-averaged, LLaVA-1.5).
  • vs. best single baseline (LoRA): +18.51 compositional reliability (62.05 → 80.56).
  • Knowledge Connector ablation (LLaVA-1.5): Dual-LoRA+RAG without connector reaches 78.16% CompRel at gap=0, degrading to 63.39% at gap=100; adding the connector raises this to 99.21% at gap=0 and 97.01% at gap=100.
  • Oracle RAG baseline: even with perfect retrieval, Base+RAG caps at 64.93% CompRel, demonstrating retrieval alone cannot enable compositional reasoning.
  • External memory ablation (VLKEB): Mem-E (tex+vis) achieves Rel=96.51 vs. SERAC text-only at 75.00 and Mem-E (tex) at 48.02; image locality (I-Loc) jumps from 4.02 to 57.10 when visual cues are added.
  • Dual vs. Single LoRA: Dual-LoRA (r=8+8) outperforms Single-LoRA (r=16) by +17.77% T-Loc for visual edits and +5.06% Loc for textual edits at matched parameter budget (p<0.05).

Limitations

  • Benchmark scope is restricted to a paired edit structure (one visual edit immediately followed by its associated textual edit); interleaved arbitrary edit sequences are not evaluated.
  • Query decomposition relies on GPT-4o, introducing inference-time cost and an external API dependency.
  • Evaluated on only two backbone models (LLaVA-1.5 and MiniGPT-4); generalization to larger or more recent LVLMs is untested.
  • CompRel ablation uses oracle retrieval; realistic multi-hop retrieval errors compound in ways not fully characterized.
  • Knowledge Connector training uses a fixed adversarial retriever; robustness to distribution shift in retrieval noise is not analyzed.

Relevance to Vision-Language Models

MemEIC directly targets a fundamental practical challenge for deployed LVLMs: keeping visual and textual world knowledge synchronized as facts change, without retraining. The brain-lateralization design—separating visual and textual editing pathways in the FFN via independent LoRA adapters—offers a principled alternative to the monolithic parameter updates that cause cross-modal interference, with implications for LVLM fine-tuning methodology more broadly. The finding that even oracle retrieval cannot enable compositional reasoning without an explicit cross-modal fusion mechanism (the Knowledge Connector) highlights a gap in pure RAG-based approaches to LVLM knowledge management. CCKEB fills an important benchmarking gap, as prior LVLM editing benchmarks did not jointly assess continual editing retention and compositional multi-modal query answering.