Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

🕒 Published (v1): 2026-07-09 13:55 UTC · Source: Arxiv · link

Why this paper was selected

Cognitive memory architecture for unified understand/generate/edit — directly improves multimodal agent design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Unified multimodal models degrade in long-horizon dialogue because visual token counts grow unboundedly as history accumulates. This paper proposes the Cognitive-structured Multimodal Agent (CMA), which externalizes visual history into a structured Episodic Visual Memory (EVM) and retrieves only relevant episodes per turn. An 8B CMA achieves 91.4% cross-turn retrieval accuracy over 20-turn sessions, outperforming 32B unified-model baselines by +8.2% while halving per-turn inference time.

Problem

Monolithic unified multimodal models (e.g., BAGEL) handle long-horizon dialogue by repeatedly feeding all historical images into the shared context window. As dialogue length grows, this causes: (i) quadratic-like visual token explosion crowding out the reasoning budget, (ii) semantic drift from implicit attention over irrelevant historical images, and (iii) near-zero cross-turn retrieval accuracy on hard turns (\(<3\%\) for BAGEL on the hardest subset). No prior work simultaneously addresses cross-turn visual retrieval, mixed-task orchestration, and long-horizon context management in a unified dialogue framework.

Method

The problem is reformulated as visual-context selection: given dialogue history \(H_{<i}\) and current query \(u_i\), learn a policy \(\pi\) that selects a minimal relevant image subset \(I_i^* = \pi(u_i, H_{<i}) \subseteq I_{<i}\), so inference runs as \(y_i = f(u_i, \text{Text}(H_{<i}), I_i^*)\).

Three specialized modules implement \(\pi\): - Perceptual Abstraction Engine (PAE): converts each incoming image into a structured JSON memory entry (semantic tags, textual description, compressed thumbnail) written to the external EVM. - Cognitive Retrieval Engine (CoRE): given the current query and dialogue state, performs cross-modal reasoning over EVM entries and returns a ranked index of relevant episodes. - Multimodal Executive Controller (MEC): classifies the turn into one of five modes (generation, editing, composition, understanding, pure-chat), queries CoRE for context, fuses retrieved episodes with dialogue history, and dispatches to the appropriate executor (Qwen-Image for generation, Qwen-Image-Edit for editing/composition).

Training uses a staged SFT+RL pipeline on data from the Unified Scenario Engine, a closed-loop synthetic generator where Gemini acts as a user simulator producing 20-turn sessions with fine-grained turn-level retrieval annotations. CoRE is first trained with SFT (300 sessions), then optimized with DAPO using a difficulty-weighted Jaccard reward: $\(r = \frac{|\hat{R} \cap R^*|}{|\hat{R} \cup R^*|} \cdot w_d, \quad w_d = 1.2 \text{ for hard/very-hard turns}\)$ PAE is then optimized with CoRE frozen, rewarding memory entries by downstream retrieval Jaccard rather than captioning metrics (CIDEr/BERTScore).

Key Contributions

• Cognitive-structured Multimodal Agent framework with modular PAE, EVM, CoRE, and MEC, enabling bounded per-turn visual-token consumption regardless of dialogue length.
• Unified Scenario Engine: programmatic closed-loop generator producing 20-turn multimodal sessions with dense turn-level retrieval annotations across 55 topics and 8 domains.
• M2CA-Bench: 100 sessions × 20 turns (2,000 annotated turns) stratified by four difficulty levels (easy/medium/hard/very hard), with hard-negative confounders (near-duplicate images, no-retrieval-needed queries) to prevent shortcut learning.
• DAPO-based RL applied separately to CoRE (retrieval policy) and PAE (memory construction), closing the training loop between memory quality and retrieval reward.
• CMA-Harness: tool-augmented deployment extending the same architecture with web search, persistent multi-session memory, image generation/editing/composition, and OpenAI-compatible serving (17-tool MEC action space).

Results

• Retrieval accuracy (Full, 20 turns): 91.4% (8B CMA) vs. 83.2% (best 32B baseline), a +8.2% gain.
• Hard subset (very hard turns in turns 11–20): +18.4% over the 32B baseline; gap widens with horizon.
• Full subset vs. baselines: +9.6% on the Full subset.
• Per-turn inference time: 12.7s (CMA) vs. 23.1s (all-context baselines), a 1.8× speedup.
• Chinese sessions: 89.6% retrieval accuracy.
• BAGEL (strong unified 32B model): \(<3\%\) retrieval accuracy on the hardest turns of M2CA-Bench, confirming that implicit attention over long visual histories fails.

Limitations

• Training and evaluation data are entirely synthetically generated by the Unified Scenario Engine; real-world distribution shift is not assessed.
• M2CA-Bench is held out from the same generator pool, so benchmark and training data share the same procedural biases.
• PAE optimization freezes CoRE, preventing joint end-to-end optimization of both modules simultaneously.
• The paper text is truncated before the full ablation tables and generation-quality scores are presented, making it impossible to confirm all reported numbers from the provided excerpt.
• CMA-Harness deployment details (17-tool MEC action space, multi-scope memory schema) are deferred to appendices not included in the provided text.

Relevance to Vision-Language Models

This work directly targets a structural bottleneck of current unified VLMs—the inability to sustain grounded, consistent multimodal interaction beyond short contexts—and demonstrates that a modular memory-retrieval architecture at 8B parameters can decisively outperform monolithic 32B VLMs on cross-turn visual referencing. It introduces a retrieval-centric training paradigm (rewarding memory quality by downstream retrieval success rather than captioning metrics) that complements ongoing work on instruction-following and unified token-prediction VLMs. The M2CA-Bench fills a gap in existing VLM benchmarks (MMNeedle, Visual Haystacks), which expose retrieval degradation but do not evaluate full agent behavior across interleaved understanding, generation, and editing tasks. For researchers tracking VLMs, this signals that long-horizon multimodal capability may require architectural decoupling of memory and execution rather than simply scaling context windows or model parameters.