Skip to content

Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Tasks

🕒 Published (v1): 2025-11-30 15:45 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MIRA (Memory-Integrated Reconfigurable Adapters) is a unified framework that embeds Hopfield-style associative memory modules into each ViT layer to store per-task LoRA adapter weights and retrieve affine combinations of them on a per-sample basis at inference. A single architecture with only objective-function swaps handles domain generalization (DG), domain-incremental learning (DIL), and class-incremental learning (CIL), achieving state-of-the-art on all three settings. The core novelty is post-hoc key refinement via backpropagation, enabling the memory to learn optimal adapter retrieval without gradient computation at inference.

Problem

Domain generalization, domain-incremental learning, and class-incremental learning have been treated as isolated paradigms despite sharing the same underlying need: adapting to new tasks or distributions without forgetting prior ones. Existing unified approaches (e.g., ICON) lack explicit memory-based retrieval mechanisms, and adapter-based PEFT methods do not address catastrophic forgetting or OOD generalization. No prior work integrates associative memory modules into the training forward pass to jointly solve all three paradigms in one architecture.

Method

MIRA attaches Universal Hopfield Network (UHN) memory modules to every ViT-B/16 layer. Training proceeds in two stages:

  1. Adaptation: For each task, LoRA rank-4 adapters are trained on the task data using cross-entropy loss; adapter weight vectors are written into the UHN with randomly initialized placeholder keys.
  2. Consolidation: Adapter weights are frozen; only the retrieval keys and lightweight query modules \(g_\ell: \mathbb{R}^{d_h} \to \mathbb{R}^{d_k}\) are updated via backpropagation. At each layer \(\ell\), the query \(q_\ell = g_\ell(h_{\ell-1})\) is matched against stored keys to produce an affine-weighted combination of adapters: \(\hat\theta_\ell = \Theta_\ell \cdot \text{sep}(K_\ell^\top q_\ell)\).

At inference, the retrieved adapter ensemble is loaded per-sample with no gradient computation. For CL settings, standard forgetting-mitigation strategies (e.g., DualGPM) are applied during Consolidation. The affine separation function is used as default, allowing negative weights to actively suppress interfering adapters.

Key Contributions

  • Unified framework (MIRA) covering DG, DIL, and CIL with only objective-function differences, no architectural changes per setting.
  • Post-hoc key learning: retrieval keys are refined via backpropagation after adapter storage, aligning key geometry with layer activation geometry.
  • Theoretical grounding via Lemma 1: under a reproducing-kernel Hilbert space assumption, AM retrieval can recover the optimal adapter mixture from Equation 3.
  • Ablation establishing that affine separation (enabling negative adapter weights) and post-hoc key refinement are both necessary for peak performance.
  • Negligible inference overhead: ~0.4% latency and <0.4% parameter increase over standard LoRA.

Results

  • iDigits (CIL/DIL): MIRA 83.00%/82.46% avg accuracy vs. ICON 71.53%/84.83%; overall avg 82.73 vs. 78.18.
  • CORe50 (CIL/DIL): MIRA 83.39%/93.89% (0.00% forgetting on DIL) vs. ICON 80.85%/89.01%.
  • DomainNet (CIL/DIL): MIRA 67.29%/69.18% vs. ICON 65.43%/54.44%; avg 68.24 vs. 59.94.
  • DG (4 datasets, Table 3): MIRA avg 81.92 vs. PEGO 80.30; PACS 97.01, OfficeHome 87.36, DomainNet 61.19.
  • CDDB-hard (DIL): MIRA 77.37% vs. prior SoTA S-iPrompts 74.51%.
  • ImageNet-R CIL: MIRA 78.06% (5-task), 73.08% (10-task) vs. C-LoRA 75.85%/71.89%.
  • DN4IL (DIL, no replay): MIRA 78.40% vs. DARE++ 44.11% (uses 200-exemplar replay buffer).

Limitations

  • Memory capacity scales with the number of tasks (one adapter set + keys per task per layer); storage cost not bounded independently of task count.
  • Evaluated exclusively on ViT-B/16 with CLIP/LAION initialization; generalization to other backbones (LLMs, CNNs) unverified.
  • Retrieval is limited to affine combinations of seen adapters; extrapolation to genuinely out-of-training-distribution tasks (beyond the convex hull of sources) is noted as future work.
  • Task identity required during training (to route Adaptation/Consolidation), though not at inference.
  • Requires 10 adapters per task for best results; practical scaling to large task sequences not analyzed.

Relevance to Agentic AI / LLM Agents

MIRA's core idea—storing task-specific adapter weights in an associative memory and retrieving weighted combinations at inference without gradient computation—directly addresses a central challenge for multi-task LLM agents: how to rapidly switch behavioral policies across heterogeneous tasks without fine-tuning and without catastrophic forgetting. The two-stage training (task-specific Adaptation followed by memory Consolidation) mirrors the agent lifecycle of acquiring new skills and integrating them into a persistent memory store. The framework's compatibility with any LoRA-style PEFT method and its negligible inference overhead make it a candidate mechanism for long-running agents that accumulate skills incrementally. It also provides a principled alternative to prompt-based task conditioning (L2P, DualPrompt) for agents that must operate across many sequential tasks without task-id at inference.