Skip to content

Mixture of Cognitive Experts in Large Vision-Language Models

🕒 Published (v1): 2026-07-12 15:09 UTC · Source: Arxiv · link

Why this paper was selected

Mixture-of-cognitive-experts for LVLMs; novel framing of metacognition and diverse representations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoCE introduces a training-free, evidence-driven multimodal reasoning framework for LVLMs that routes queries through specialized CV experts, converts their outputs into atomic evidence statements, and applies a Bloom's taxonomy-inspired staged verbalization before final LLM generation. The framework also includes a lightweight Reasoning Trace Module that quantitatively tracks evidence usage and cognitive entry levels across queries.

Problem

Most LVLM inference pipelines apply uniform perception-reasoning routing regardless of the cognitive demands of a query, and lack interpretable intermediate representations—making outputs prone to hallucination and difficult to verify. Existing work on integrating specialized CV experts (detection, OCR, segmentation) into LVLMs does not systematically organize their outputs into trustworthy, citeable evidence grounded by a cognitive hierarchy.

Method

MoCE operates at inference time without any additional training. Given an image \(I\) and query \(q\):

  1. LLM-guided Router: selects a top-\(K\) subset of vision experts \(\mathcal{G}\) from a pool comprising DETR (object detection), SAM (segmentation), Pix2Struct/Vary (OCR/document understanding), and DePlot (chart parsing), using the query and a downsampled visual embedding.

  2. Literal Evidence Summary: each selected expert \(e\) produces outputs \(O^{(e)}\), which are deterministically decomposed by a per-expert verbalizer \(v_e: O^{(e)} \rightarrow \{(s_k, m_k)\}\) into atomic, non-inferential statements. These are stored in a unified evidence store \(\mathcal{E} = \{e_i\}_{i=1}^{M}\) where each item carries an ID, a type (\(\in \{\text{TEXT}, \text{ENTITY}, \text{REGION}\}\)), a literal text, and a grounding reference (bounding box, mask, or cell index).

  3. Bloom Verbalization: a verbalizer \(f_\text{Bloom}(q, \mathcal{E})\) determines a query-conditioned starting level \(\ell_0 \in\) [LIST, EXPLAIN, ILLUSTRATE, COMPARE, HYPOTHESIZE, FINAL ANS.] and executes all subsequent Bloom steps, producing a staged draft answer \(y_\text{Bloom}\) with explicit evidence citations \(U_\ell \subseteq \{1,\ldots,M\}\) at each step.

  4. Final LLM: \(\hat{y} = f_\text{LLM}(q, I, y_\text{Bloom})\) conditions on the image, query, and Bloom draft tokens.

  5. Reasoning Trace Module: deterministically parses \(y_\text{Bloom}\) to recover \(\ell_0\), executed steps, per-step cited evidence sets, and evidence-type usage distributions—enabling dataset-level diagnostics without hidden chain-of-thought.

Visual expert features are fused via an adapter (from MoVA) using \(L\) stacked transformer blocks that iteratively refine base CLIP-L (336px) features with routed expert representations, followed by an MLP projector. The language backbone is Vicuna-7B.

Key Contributions

  • Training-free evidence-driven framework that integrates specialized CV expert outputs into LVLMs via atomic evidence verbalization and Bloom's taxonomy-structured staged reasoning.
  • Literal Evidence Summary with typed, citeable, grounding-referenced atomic statements (TEXT/ENTITY/REGION) that enforce fidelity and inspectability before any reasoning occurs.
  • Bloom Verbalizer that operationalizes Bloom's cognitive taxonomy as an ordered, query-conditioned reasoning protocol producing explicit intermediate scaffold tokens for the downstream LLM.
  • Reasoning Trace Module enabling quantitative, deterministic analysis of evidence usage, cognitive entry level, and step-wise reasoning progression across a dataset.

Results

  • TextVQA: 67.8 vs. LLaVA-NeXT 7B (64.9), LLaVA-1.5 13B (61.3), Qwen-VL (63.8).
  • MMBench: 75.7 vs. LLaVA-NeXT 7B (75.7), LLaVA-1.5 7B (67.7), Qwen-VL-Chat (64.3).
  • MMVet: 43.9 (LLaVA-1.5 7B) / 49.0 (13B) are listed baselines; MoCE value not fully visible in truncated text.
  • POPE: 87.8 vs. LLaVA-1.5 13B (86.5), showing reduced hallucination degradation under adversarial conditions.
  • SEED: 72.6 vs. LLaVA-1.5 13B (70.2).
  • MMBench ablation (Bloom contribution): Vision Experts + Bloom achieves MMB 75.7 vs. Vision Experts Only 70.4 (+5.3); SEED text understanding also improves.
  • MMBench Level-2 breakdown: MoCE leads all four categories—Coarse Perception (CP), Fine-grained Perception (FP-S), Logical Reasoning (LR), and Relation Reasoning (RR)—against InstructBLIP, Qwen-VL-Chat, and LLaVA-1.5.
  • Outperforms several 13B baselines at 7B scale, suggesting architectural design is more impactful than scale alone.

Limitations

  • Ablation results are truncated in the provided text; the full quantitative isolation of each component is not fully reported here.
  • Bloom Verbalization adds inference latency and GFLOPs (exact numbers cut off in text); the efficiency cost is acknowledged but not fully characterized.
  • Vision expert pool is fixed and borrowed from MoVA; domain coverage depends on the quality and breadth of those pretrained specialists.
  • The framework is validated on a single 7B LLM backbone (Vicuna-7B) with CLIP-L; generalizability to other backbones is undemonstrated.
  • The Reasoning Trace Module's diagnostic value depends on the Bloom verbalizer consistently emitting parseable step headers and citation tokens, which is not guaranteed under distribution shift.
  • The authors acknowledge that most evaluated queries require only basic recall/understanding levels, meaning higher-order Bloom steps are exercised less frequently in the current benchmarks.

Relevance to Vision-Language Models

This paper directly advances the LVLM architecture design space by showing that structured cognitive scaffolding—rather than end-to-end training or scale—can meaningfully improve multimodal reasoning and reduce hallucination. The Bloom-verbalization mechanism is a concrete, operationalizable alternative to opaque chain-of-thought prompting, offering interpretability without requiring hidden reasoning traces. For researchers tracking VLMs, the Reasoning Trace Module introduces a new diagnostic lens for understanding how different queries engage different perceptual and reasoning capacities, which could inform future benchmark design and training objectives. The training-free nature makes this directly composable with existing LVLM backbones, lowering the barrier to adoption.