Skip to content

ContextNav: Towards Agentic Multimodal In-Context Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ContextNav is the first agentic framework that reformulates multimodal in-context learning (ICL) context selection as a closed-loop, tool-driven workflow rather than a static retrieval step. It combines automated vector-database retrieval with MLLM-driven semantic denoising and structural alignment, coordinated by an Operational Grammar Graph (OGG) that adapts across timesteps via downstream ICL feedback. It achieves an average ICL gain of 16.8% across six MLLMs, more than doubling the prior state-of-the-art gain of 7.6%.

Problem

Existing multimodal ICL contextualization faces a robustness-scalability tradeoff. Manual curation produces clean, high-quality contexts but is labor-intensive and task-specific. Similarity-based retrieval is automated and scalable but introduces two noise types: semantic noise (off-topic or contradictory candidates) and structural noise (interrogative/imperative/narrative format mismatches between retrieved candidates and the query). Neither approach adapts its selection strategy based on observed downstream ICL performance.

Method

ContextNav unifies three synergistic modules under an MLLM policy \(\pi_\theta\) (Gemini-2.0-flash by default):

  1. Agentic Context Management: The agent selects embedding model pairs \((E_T, E_V)\) via a resource-aware prompt \(P_\text{emb}\) that encodes hardware state and available model zoo. It builds and incrementally updates a multimodal vector database \(D\) from corpus \(C = \{(T_i, I_i)\}_{i=1}^N\), then retrieves an initial candidate pool \(R^\text{init}_\tau = f_\tau(q, D_\tau, k)\) using top-\(k\) similarity.

  2. Context Denoising: Two sequential MLLM-driven filtering steps:

  3. Agentic Retrieval: Conditioned on coherence prompt \(P_\text{coh}\), the policy evaluates each candidate for semantic alignment with query \(q\) and removes weak matches: \(R^\text{sem}_\tau = \pi_\theta(q, P_\text{coh}, R^\text{init}_\tau)\).

  4. Structural Alignment: Conditioned on \(P_\text{str}\), the policy rewrites structurally inconsistent candidates to mirror the query's textual form: \(R^\text{alin}_\tau = \pi_\theta(q_t, P_\text{str}, R^\text{sem}_\tau)\).

  5. Graph-driven Workflow Orchestration via OGG: A directed graph \(G = (V, E)\) encodes atomic operations as nodes and valid execution dependencies as edges. The agent instantiates operation sequences \(S_\tau = \pi_\theta(P_\text{wop}, M_\tau, G)\) where each transition \((v_i, v_{i+1}) \in E\) enforces execution validity. A memory module \(M\) stores pairs \((S_i, \phi_i)\) coupling past workflows with downstream ICL feedback \(\phi_\tau\) from the target MLLM \(\Phi\), enabling adaptive optimization across timesteps: \(M_{\tau+1} = M_\tau \cup (S_\tau, \phi_\tau)\).

Key Contributions

  • First agentic framework for contextualization in multimodal ICL, replacing static one-shot retrieval with an adaptive, tool-driven workflow.
  • Operational Grammar Graph (OGG) that formally encodes permissible operation sequences and dependency constraints, enabling valid multi-step workflow planning.
  • Memory-augmented closed-loop optimization: downstream ICL feedback continuously refines the agent's toolchain selection strategy across timesteps.
  • Resource-aware embedding pipeline that adapts model selection to user hardware constraints.
  • 16.8% average ICL gain across six MLLMs on eight diverse benchmarks, vs. 7.6% for the prior SOTA (MMICES).

Results

  • Average ICL gain: ContextNav 16.8% vs. MMICES (prior SOTA) 7.6%; VL-ICL 7.7%; Random Sampling −2.4% (degrades performance).
  • Per-model average accuracy (base → +ContextNav):
  • Gemini-2.0-flash: 0.542 → 0.604 (+0.062)
  • Gemini-1.5-flash: 0.505 → 0.574 (+0.069)
  • GPT-4o: 0.484 → 0.547 (+0.063)
  • Qwen2.5-VL-7B: 0.440 → 0.480 (+0.040)
  • InternLMX2.5-7B: 0.287 → 0.312 (+0.025)
  • Phi-3.5V-4.2B: 0.332 → 0.346 (+0.014)
  • ContextNav outperforms all baselines (zero-shot, random sampling, VL-ICL, MMICES) on every model across BlindTest, MME-RealWorld, CharXiv, GVL, MathVision, CLEVR, FOMI, and TextOCR.
  • Strongest absolute gains observed on larger/closed-source models (Gemini, GPT-4o).

Limitations

  • Agent policy defaults to Gemini-2.0-flash (closed-source), introducing API dependency and cost.
  • Multi-timestep closed-loop operation introduces inference latency compared to one-shot retrieval; not characterized in the provided text.
  • Embedding pipeline requires GPU resources (A100 80G), limiting accessibility for resource-constrained deployments.
  • Gains are smallest on smaller open-source models (Phi-3.5V: +1.4%), suggesting the approach may be less impactful when the MLLM itself has limited reasoning capacity.
  • OGG structure must be manually specified per domain; generalization to entirely new task types is unclear.

Relevance to Harnesses / Meta-Harnesses

ContextNav is directly a harness design paper: it wraps any MLLM's inference step with an adaptive, tool-driven pipeline (retrieval → semantic denoising → structural alignment) coordinated by the OGG, showing that the harness layer — not the underlying model — accounts for most of the 16.8% gain across six different MLLMs. The OGG's feedback-driven adaptation across timesteps is a concrete instantiation of the meta-harness pattern: a controller that monitors downstream task outcomes and reshapes the pipeline's behavior accordingly. For someone building agent harnesses, the key transferable idea is that static retrieval-then-prompt patterns leave large gains on the table, and closed-loop context refinement (with structured semantic filters between retrieval and the model) is both feasible and highly effective at inference time without any fine-tuning. The model-agnostic, 2×-over-prior-SOTA result also serves as a strong empirical argument for investing in the harness tier rather than chasing better base models.