Skip to content

Towards Resource-Efficient Multimodal Intelligence: Learned Routing among Specialized Expert Models

🕒 Published (v1): 2025-11-09 16:14 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a modular AI framework that routes queries — text, image, audio, video, or document — to the most cost-appropriate expert model using a learned, multi-signal routing function. A two-stage complexity scorer for text and a semantic similarity dispatcher for non-text inputs selectively invoke expensive LLMs only when necessary. On MMLU and VQA benchmarks the system matches or exceeds always-premium monolithic baselines while reducing reliance on costly models by over 67%.

Problem

Deploying large multimodal LLMs uniformly across all query types is computationally wasteful and impractical for real-time, resource-constrained deployment. Existing routing systems (HybridLLM, RouteLLM, FrugalGPT) handle only text, while orchestration frameworks (HuggingGPT) keep a heavyweight controller active even for trivial subtasks. No prior architecture jointly addresses routing across all modalities with dynamic cost–quality optimization.

Method

The framework decomposes each incoming request through three layers:

  1. Modality classification: a hierarchical decision tree using MIME analysis, file extension mapping, and content-based magic-number detection assigns inputs to text-only, non-text, hybrid, or follow-up categories.

  2. Complexity scoring (text path): a scalar score \(C(Q) = \alpha \cdot I(Q) + \beta \cdot L(Q) + \gamma \cdot S(Q)\) combines intent alignment \(I\), linguistic length \(L\), and structural/semantic density \(S\). Queries with \(C(Q) < \tau\) route to efficient open-source models (QwenCoder, WizardMath); those above \(\tau\) escalate to premium LLMs (GPT-4, Claude, Gemini).

  3. Multi-signal routing objective: the selected route \(r^*\) maximizes $\(R(Q) = \arg\max_{r \in \mathcal{R}} \left[\delta_m S_m(Q,r) + \delta_u S_u(Q,r) + \delta_t S_t(Q,r) - \lambda_c C_r(r)\right]\)$ where \(\delta_m, \delta_u, \delta_t, \lambda_c\) are \(\ell_1\)-normalized tunable weights (modality alignment, user preference, task affinity, cost penalty).

Non-text inputs bypass complexity scoring and go directly to modality-specific pipelines. The Couplet subsystem handles vision by pairing classical detectors (CLIP, YOLO, DETR) with a small language model (Phi-3, Mistral-7B) to avoid invoking full vision-LLMs by default. Complex multi-dependency queries flow into a LangGraph-based multi-agent planner that builds a modality/task DAG at runtime, executes independent subtasks in parallel, and fuses outputs via a Mixture-of-Experts (MoE) aggregator using modality-aligned relevance weights.

A Context Agent maintains five memory layers (short-term window, full session log, modality-specific banks, relevant-query retrieval, compressed context) scored by \(\text{ContextScore}(L_i) = \theta_s R_i + \theta_t T_i + \theta_m M_i\).

Key Contributions

  • Unified routing framework spanning text, image, audio, video, and document modalities in a single architecture.
  • Learned complexity scorer \(C(Q)\) with AST-based code detection (94% accuracy), Shannon-entropy lexical analysis, and dynamically calibrated threshold \(\tau\).
  • Formal routing objective (Eq. 2) with \(\ell_1\)-normalized, Pareto-tuned weights balancing modality fit, user policy, task affinity, and cost.
  • Couplet subsystem reviving classical vision components (CLIP/YOLO/DETR) as cost-efficient alternatives to end-to-end vision-LLMs.
  • Layered memory management with per-modality isolation to reduce hallucination in multi-turn multimodal sessions.
  • User-facing preference toggle that propagates open-source/paid constraints globally across the routing graph.

Results

  • Matches or exceeds always-premium monolithic LLM performance on MMLU (language understanding) and VQA (visual question answering).
  • Reduces reliance on costly premium models by over 67% relative to always-premium baselines.
  • No ablation tables or granular per-benchmark numbers are provided in the text; specific accuracy figures are not reported.

Limitations

  • No ablation numbers isolating the contribution of individual routing components (Couplet, complexity scorer, MoE aggregator).
  • Threshold \(\tau\) and routing weights \(\delta_m, \delta_u, \delta_t, \lambda_c\) are described as empirically derived but calibration datasets and sizes are not disclosed.
  • Evaluation covers only MMLU and VQA; audio, video, and document pipeline performance is not quantified.
  • The 67% cost-reduction figure lacks a statistical confidence interval or variance estimate.
  • LangGraph multi-agent orchestration overhead for truly complex queries is not characterized.
  • The Couplet path is described architecturally but its accuracy/latency versus full vision-LLMs is not benchmarked directly.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a meta-harness: a control layer that wraps multiple heterogeneous sub-models and selects among them at query time based on learned signals, rather than hard-coded dispatch. The routing objective (Eq. 2), the LangGraph-based planner, and the MoE aggregator together constitute a programmable orchestration spine — exactly the architectural pattern that distinguishes a meta-harness from a simple ensemble. The Couplet subsystem demonstrates how a meta-harness can revive classical (non-neural) modules as cost-efficient experts, a design principle applicable to any harness that manages a heterogeneous tool pool. The layered memory and fallback mechanisms also inform how meta-harnesses should maintain state and recover from sub-agent failures.