Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

🕒 Published (v1): 2026-07-09 13:55 UTC · Source: Arxiv · link

Why this paper was selected

Cognitive-structured multimodal agent with selective context; addresses VLM context scaling

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Unified multimodal models fail at long-horizon dialogue because injecting all historical visual tokens into a shared context window causes exponential token growth and cross-turn retrieval degradation. This paper proposes a Cognitive-structured Multimodal Agent (CMA) that externalizes visual history into an Episodic Visual Memory and uses a dedicated retrieval module to selectively reactivate only relevant visual episodes per turn. An 8B CMA agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B unified baselines by +8.2% while halving per-turn inference time.

Problem

Monolithic unified multimodal models (e.g., BAGEL) treat long-horizon multimodal dialogue as autoregressive prediction over a growing token sequence. As dialogue length increases, this causes: (i) prohibitive visual-token growth crowding out reasoning budget, (ii) contextual interference from irrelevant historical images, and (iii) unstable attention over long histories leading to semantic drift. A strong baseline (BAGEL) retrieves the correct visual episode in fewer than 3% of the hardest evaluation turns, confirming that parameter scaling alone cannot solve this. Additionally, no existing dataset provides turn-level retrieval supervision needed to train cross-turn visual grounding.

Method

The agent formalizes long-horizon dialogue as selecting a minimal relevant image subset \(I_i^* = \pi(u_i, H_{<i}) \subseteq I_{<i}\) and conditioning on it: \(y_i = f(u_i, \text{Text}(H_{<i}), I_i^*)\). Three specialist modules implement policy \(\pi\):

• Perceptual Abstraction Engine (PAE): encodes each incoming image into a structured JSON entry (semantic tags, textual description, compressed thumbnail) written to the external Episodic Visual Memory (EVM).
• Cognitive Retrieval Engine (CoRE): given the current query and dialogue state, retrieves indices of relevant EVM entries. Trained via SFT then DAPO RL with a difficulty-weighted Jaccard reward \(r = \frac{|\hat{R} \cap R^*|}{|\hat{R} \cup R^*|} \cdot w_d\) (\(w_d = 1.2\) for hard/very-hard turns).
• Multimodal Executive Controller (MEC): classifies task type (generation, editing, composition, understanding, pure-chat) and dispatches to appropriate executor (Qwen-Image / Qwen-Image-Edit), forwarding only retrieved images to bound per-turn token cost.

PAE is optimized in a second RL stage where CoRE is frozen and retrieval accuracy of entries produced by PAE serves as the reward, directly aligning memory construction with downstream utility rather than captioning metrics.

Training data is generated by the Unified Scenario Engine: a closed loop where Gemini simulates users, a zero-shot multi-agent system (Qwen-VL + Qwen-Image-Edit) produces answers, and a GT-justify step verifies retrieval correctness before feeding back to the simulator. This produces 300 labeled 20-turn sessions (5,400–6,000 turns) across 55 topics in 8 domains, with structural inductive biases enforcing at least 3 long-range callbacks with span ≥ 8 turns.

The held-out M2CA-Bench consists of 100 sessions × 20 turns (2,000 annotated turns), stratified into four difficulty tiers (easy/medium/hard/very-hard) by topic-shift frequency, temporal span, multi-image interaction, and referential ambiguity, with hard negatives (near-duplicate confounders and no-retrieval-needed queries) to prevent shortcut learning.

Key Contributions

• CMA framework decoupling perception (PAE), episodic storage (EVM), retrieval (CoRE), and execution (MEC) into independently trainable modules.
• Unified Scenario Engine for programmatic generation of structured multi-turn multimodal conversations with dense turn-level retrieval annotations.
• M2CA-Bench: first benchmark targeting cross-turn episodic visual retrieval in long-horizon multimodal dialogue (100 sessions, 20 turns each, 4 difficulty levels).
• Staged SFT+DAPO-RL pipeline that optimizes both the retriever and the memory writer end-to-end for retrieval success rather than captioning quality.
• CMA-Harness: tool-augmented deployment extending the same architecture with web access, image generation/editing/composition, persistent multi-session memory, and OpenAI-compatible serving (17-tool MEC action space).

Results

• Retrieval accuracy (Full, 20-turn English sessions): 91.4% for CMA-8B vs. best 32B baseline at ~83.2% (+8.2%).
• Chinese sessions: 89.6%.
• Hard subset (very hard turns in turns 11–20): margin grows to +18.4% over the strongest baseline.
• Full vs. Medium subset margin: +9.6% on Full subset.
• Per-turn inference time: 23.1s → 12.7s (1.8× speedup) vs. all-context baseline.
• BAGEL (strong unified 32B model): <3% retrieval accuracy on hardest turns, confirming that implicit attention over visual histories is fundamentally insufficient.

Limitations

• M2CA-Bench is programmatically generated; real user dialogue distributions may differ from the controlled simulation.
• The Unified Scenario Engine relies on Gemini as a user simulator, which may introduce biases toward certain dialogue patterns or topic structures.
• The CMA-Harness is presented without full ablation; system-level evaluation details are deferred to appendices not included in the paper excerpt.
• Each module (PAE, CoRE, MEC) requires independent training pipelines, increasing engineering complexity relative to end-to-end unified models.
• Generation Quality metric relies on Gemini-3-Pro as a judge, introducing potential evaluator bias.

Relevance to Agentic AI / LLM Agents

This work directly addresses a central bottleneck in long-horizon agentic systems: how to manage a growing multimodal context without unbounded token costs. The modular decomposition (perceive → store → retrieve → execute) mirrors cognitive architectures proposed for LLM agents (working memory, episodic memory, executive control), and the RL-based alignment of memory construction to downstream retrieval utility is a transferable design principle beyond vision. The CMA-Harness instantiation, combining persistent memory with external tool use and OpenAI-compatible serving, is a concrete blueprint for deploying memory-augmented multimodal agents in practice. The finding that an 8B modular agent outperforms monolithic 32B models on long-horizon tasks provides empirical evidence against simply scaling unified models for agentic workloads.