Skip to content

Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing

🕒 Published (v1): 2026-07-02 10:10 UTC · Source: Arxiv · link

Why this paper was selected

Online recursive MLLM editing with bounded overhead — essential for keeping deployed VLMs current

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Reliable online edits to multimodal LLMs do not guarantee correct scope control: 28.60% of successful edits fail to transfer to valid cross-modal variants (under-generalization), and 7.20% leak into unrelated inputs (over-generalization). ScopeEdit addresses this by decomposing each FFN weight update into a modality-local absorption branch and an evidence-gated shared generalization branch, both operating in orthogonal low-rank subspaces with Sherman–Morrison recursive preconditioners for constant per-edit overhead.

Problem

Existing online MLLM editors optimize for edit reliability (instance-level correctness) and long-horizon stability under sequential edits, but provide no mechanism to control the semantic boundary of each edit. This creates a scope gap: an edit absorbed correctly on its original input may fail to generalize to semantically equivalent cross-modal variants (paraphrases, alternative image views) or may propagate to wholly unrelated inputs. The problem is compounded by multimodal heterogeneity—visual and textual representations occupy different statistical scales, making naive shared-update statistics unreliable.

Method

ScopeEdit reformulates each per-step FFN output matrix update \(\Delta W_t\) as a two-branch decomposition:

\[\Delta W_t^{(l)} = \Delta W_{t,\text{loc}}^{(l)} + \Delta W_{t,\text{sh}}^{(l)} = \Delta B_{t,\text{loc}}^{(l)} A_{\text{loc}}^{(l)} + \Delta B_{t,\text{sh}}^{(l)} A_{\text{sh}}^{(l)}\]

where \(A_{\text{loc}}^{(l)} \in \mathbb{R}^{r_\ell \times d}\) and \(A_{\text{sh}}^{(l)} \in \mathbb{R}^{r_s \times d}\) are fixed orthogonal bases satisfying \(A_{\text{loc}} A_{\text{loc}}^\top = I\), \(A_{\text{sh}} A_{\text{sh}}^\top = I\), \(A_{\text{loc}} A_{\text{sh}}^\top = 0\).

Local branch: Always active; accumulates textual-only FFN keys into its branch-wise preconditioner, ensuring stable absorption and out-of-scope locality.

Shared branch: Activated by a scalar scope gate \(\gamma_t^{(l)} = \sigma\!\left(\beta\!\left(\cos_t^{(l)} - \tau\right)\right) \cdot \sup_t^{(l)}\), where \(\cos_t^{(l)}\) is the cosine similarity between projected text and vision FFN keys (directional agreement) and \(\sup_t^{(l)} = \min(\|z_{\text{text}}\|, \|z_{\text{vision}}\|) / \max(\|z_{\text{text}}\|, \|z_{\text{vision}}\|)\) measures bilateral support. The gate fires only in deeper language layers where text-visual energy peaks synchronize, as identified by the pilot analysis.

Each branch maintains its own inverse preconditioner \(P_{t,b}^{(l)} = (I + C_{t,b}^{(l)})^{-1}\) updated via Sherman–Morrison rank-1 recursion, yielding \(O(r_b^2)\) per-edit cost. The write coefficient is updated as \(\Delta B_{t,b} = -\eta_b \rho_{t,b} G_{t,b} P_{t-1,b}^{(l)}\), where \(\rho_{t,\text{loc}}=1\) and \(\rho_{t,\text{sh}}=\gamma_t\).

Key Contributions

  • Edit-Scoped Generalization criterion: Formalizes the joint requirement of in-scope cross-modal generalization (Eq. 5) and out-of-scope locality preservation (KL divergence, Eq. 6) as explicit optimization targets, distinct from reliability.
  • Pilot analysis: Quantifies the scope gap—among reliable edits on LLaVA-v1.5, only 62.20% achieve proper propagation; cross-modal responses concentrate in deeper language layers, validated via layer-wise energy profiling and attention visualization.
  • ScopeEdit architecture: Dual-branch online editor with orthogonal low-rank write geometry, cross-modal evidence gating (directional agreement Ă— bilateral support), and branch-wise Sherman–Morrison preconditioners.
  • Constant per-edit overhead: By fixing bases and using rank-1 recursions, per-edit cost is \(O(r^2)\) regardless of edit stream length.

Results

Specific headline numbers from benchmark tables are not present in the provided text excerpt. The paper reports: - Pilot analysis (M-ORE on LLaVA-v1.5): 62.20% proper propagation, 28.60% under-generalization, 7.20% over-generalization, 2.00% entangled failure among reliable edits. - ScopeEdit "consistently improves the trade-off between in-scope cross-modal transfer and out-of-scope locality" across controlled benchmarks, long-horizon edit streams (\(T\) sequential edits), real-world VLKEB scenarios, and multiple MLLM backbones. - Evaluation includes comparison against baselines including M-ORE, DualEdit, VisEdit, LiveEdit, MSCKE, and text-only editors (ROME, MEMIT, AlphaEdit) transferred to multimodal settings.

Limitations

  • Quantitative benchmark results are not reproduced in the provided excerpt, so the magnitude of improvement over baselines cannot be verified here.
  • The scope gate relies on FFN key alignment in the shared subspace as a proxy for semantic equivalence; this may fail for edits where visual and textual evidence are semantically aligned but differ greatly in representational geometry.
  • The rank budget \(r = r_\ell + r_s\) must be partitioned a priori between local and shared branches; the optimal split is not derived theoretically.
  • The method is restricted to parameter-modifying edits targeting FFN output matrices; applicability to attention layers or parameter-preserving paradigms is not addressed.
  • Applies per-layer orthogonal bases initialized once; accumulated edits may saturate the fixed low-rank subspace over very long streams.

Relevance to Vision-Language Models

This work directly targets a core weakness in multimodal LLM post-training: knowledge editing that respects semantic scope boundaries across vision and language modalities. The empirical finding that cross-modal generalization emerges specifically in deeper language layers—not uniformly—is architecturally significant and informs where editing interventions should be applied in VLMs. The evidence-gated branching mechanism provides a concrete design pattern for controlling cross-modal knowledge propagation, relevant to any VLM system that requires selective knowledge updates (e.g., fact correction, domain adaptation). The formalization of Edit-Scoped Generalization as a measurable criterion (distinct from reliability and stability) advances the evaluation framework for VLM editing research.