Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

🕒 Published (v1): 2026-07-09 13:55 UTC · Source: Arxiv · link

Why this paper was selected

Cognitive scaffolding separates modalities across context windows; novel agent architecture pattern

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Unified multimodal models degrade in long-horizon dialogue because visual token counts grow unboundedly as history accumulates. This paper proposes the Cognitive-structured Multimodal Agent (CMA), which externalizes visual history into a structured Episodic Visual Memory (EVM) and retrieves only relevant episodes per turn. An 8B CMA achieves 91.4% cross-turn retrieval accuracy over 20-turn sessions, outperforming 32B unified-model baselines by +8.2% while halving per-turn inference time.

Problem

Monolithic unified multimodal models (e.g., BAGEL) handle long-horizon dialogue by repeatedly feeding all historical images into the shared context window. As dialogue length grows, this causes: (i) quadratic-like visual token explosion crowding out the reasoning budget, (ii) semantic drift from implicit attention over irrelevant historical images, and (iii) near-zero cross-turn retrieval accuracy on hard turns (\(<3\%\) for BAGEL on the hardest subset). No prior work simultaneously addresses cross-turn visual retrieval, mixed-task orchestration, and long-horizon context management in a unified dialogue framework.

Method

The problem is reformulated as visual-context selection: given dialogue history \(H_{<i}\) and current query \(u_i\), learn a policy \(\pi\) that selects a minimal relevant image subset \(I_i^* = \pi(u_i, H_{<i}) \subseteq I_{<i}\), so inference runs as \(y_i = f(u_i, \text{Text}(H_{<i}), I_i^*)\).

Three specialized modules implement \(\pi\): - Perceptual Abstraction Engine (PAE): converts each incoming image into a structured JSON memory entry (semantic tags, textual description, compressed thumbnail) written to the external EVM. - Cognitive Retrieval Engine (CoRE): given the current query and dialogue state, performs cross-modal reasoning over EVM entries and returns a ranked index of relevant episodes. - Multimodal Executive Controller (MEC): classifies the turn into one of five modes (generation, editing, composition, understanding, pure-chat), queries CoRE for context, fuses retrieved episodes with dialogue history, and dispatches to the appropriate executor (Qwen-Image for generation, Qwen-Image-Edit for editing/composition).

Training uses a staged SFT+RL pipeline on data from the Unified Scenario Engine, a closed-loop synthetic generator where Gemini acts as a user simulator producing 20-turn sessions with fine-grained turn-level retrieval annotations. CoRE is first trained with SFT (300 sessions), then optimized with DAPO using a difficulty-weighted Jaccard reward: $\(r = \frac{|\hat{R} \cap R^*|}{|\hat{R} \cup R^*|} \cdot w_d, \quad w_d = 1.2 \text{ for hard/very-hard turns}\)$ PAE is then optimized with CoRE frozen, rewarding memory entries by downstream retrieval Jaccard rather than captioning metrics (CIDEr/BERTScore).

Key Contributions

• Cognitive-structured Multimodal Agent framework with modular PAE, EVM, CoRE, and MEC, enabling bounded per-turn visual-token consumption regardless of dialogue length.
• Unified Scenario Engine: programmatic closed-loop generator producing 20-turn multimodal sessions with dense turn-level retrieval annotations across 55 topics and 8 domains.
• M2CA-Bench: 100 sessions × 20 turns (2,000 annotated turns) stratified by four difficulty levels (easy/medium/hard/very hard), with hard-negative confounders (near-duplicate images, no-retrieval-needed queries) to prevent shortcut learning.
• DAPO-based RL applied separately to CoRE (retrieval policy) and PAE (memory construction), closing the training loop between memory quality and retrieval reward.
• CMA-Harness: tool-augmented deployment extending the same architecture with web search, persistent multi-session memory, image generation/editing/composition, and OpenAI-compatible serving (17-tool MEC action space).

Results

• Retrieval accuracy (Full, 20 turns): 91.4% (8B CMA) vs. 83.2% (best 32B baseline), a +8.2% gain.
• Hard subset (very hard turns in turns 11–20): +18.4% over the 32B baseline; gap widens with horizon.
• Full subset vs. baselines: +9.6% on the Full subset.
• Per-turn inference time: 12.7s (CMA) vs. 23.1s (all-context baselines), a 1.8× speedup.
• Chinese sessions: 89.6% retrieval accuracy.
• BAGEL (strong unified 32B model): \(<3\%\) retrieval accuracy on the hardest turns of M2CA-Bench, confirming that implicit attention over long visual histories fails.

Limitations

• Training and evaluation data are entirely synthetically generated by the Unified Scenario Engine; real-world distribution shift is not assessed.
• M2CA-Bench is held out from the same generator pool, so benchmark and training data share the same procedural biases.
• PAE optimization freezes CoRE, preventing joint end-to-end optimization of both modules simultaneously.
• The paper text is truncated before the full ablation tables and generation-quality scores are presented, making it impossible to confirm all reported numbers from the provided excerpt.
• CMA-Harness deployment details (17-tool MEC action space, multi-scope memory schema) are deferred to appendices not included in the provided text.

Relevance to Harnesses / Meta-Harnesses

CMA is a direct existence proof that harness-level memory management — externalizing visual history into a structured store and retrieving per-turn — can substitute for raw model scale, with an 8B model outperforming a 32B baseline by +8.2% on cross-turn retrieval. For harness builders, the Episodic Visual Memory is a deployable scaffolding primitive: instead of naively concatenating all visual tokens into context (the default behavior of unified models), the harness indexes episodes and injects only retrieved ones, cutting inference cost in half. This maps cleanly onto the general meta-harness pattern of context budget management — the same principle used for tool-call history or RAG in text-only agents, now validated empirically for the visual modality. Anyone building a long-session multimodal agent harness (medical imaging review, iterative image editing, visual QA over document sets) should treat this as a concrete recipe for the memory subsystem.