Skip to content

ContextNav: Towards Agentic Multimodal In-Context Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; ContextNav: agentic multimodal in-context learning for novel tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ContextNav is the first agentic framework that reformulates multimodal in-context learning (ICL) context selection as a closed-loop, tool-driven workflow rather than a static retrieval step. It combines automated vector-database retrieval with MLLM-driven semantic denoising and structural alignment, coordinated by an Operational Grammar Graph (OGG) that adapts across timesteps via downstream ICL feedback. It achieves an average ICL gain of 16.8% across six MLLMs, more than doubling the prior state-of-the-art gain of 7.6%.

Problem

Existing multimodal ICL contextualization faces a robustness-scalability tradeoff. Manual curation produces clean, high-quality contexts but is labor-intensive and task-specific. Similarity-based retrieval is automated and scalable but introduces two noise types: semantic noise (off-topic or contradictory candidates) and structural noise (interrogative/imperative/narrative format mismatches between retrieved candidates and the query). Neither approach adapts its selection strategy based on observed downstream ICL performance.

Method

ContextNav unifies three synergistic modules under an MLLM policy \(\pi_\theta\) (Gemini-2.0-flash by default):

  1. Agentic Context Management: The agent selects embedding model pairs \((E_T, E_V)\) via a resource-aware prompt \(P_\text{emb}\) that encodes hardware state and available model zoo. It builds and incrementally updates a multimodal vector database \(D\) from corpus \(C = \{(T_i, I_i)\}_{i=1}^N\), then retrieves an initial candidate pool \(R^\text{init}_\tau = f_\tau(q, D_\tau, k)\) using top-\(k\) similarity.

  2. Context Denoising: Two sequential MLLM-driven filtering steps:

  3. Agentic Retrieval: Conditioned on coherence prompt \(P_\text{coh}\), the policy evaluates each candidate for semantic alignment with query \(q\) and removes weak matches: \(R^\text{sem}_\tau = \pi_\theta(q, P_\text{coh}, R^\text{init}_\tau)\).

  4. Structural Alignment: Conditioned on \(P_\text{str}\), the policy rewrites structurally inconsistent candidates to mirror the query's textual form: \(R^\text{alin}_\tau = \pi_\theta(q_t, P_\text{str}, R^\text{sem}_\tau)\).

  5. Graph-driven Workflow Orchestration via OGG: A directed graph \(G = (V, E)\) encodes atomic operations as nodes and valid execution dependencies as edges. The agent instantiates operation sequences \(S_\tau = \pi_\theta(P_\text{wop}, M_\tau, G)\) where each transition \((v_i, v_{i+1}) \in E\) enforces execution validity. A memory module \(M\) stores pairs \((S_i, \phi_i)\) coupling past workflows with downstream ICL feedback \(\phi_\tau\) from the target MLLM \(\Phi\), enabling adaptive optimization across timesteps: \(M_{\tau+1} = M_\tau \cup (S_\tau, \phi_\tau)\).

Key Contributions

  • First agentic framework for contextualization in multimodal ICL, replacing static one-shot retrieval with an adaptive, tool-driven workflow.
  • Operational Grammar Graph (OGG) that formally encodes permissible operation sequences and dependency constraints, enabling valid multi-step workflow planning.
  • Memory-augmented closed-loop optimization: downstream ICL feedback continuously refines the agent's toolchain selection strategy across timesteps.
  • Resource-aware embedding pipeline that adapts model selection to user hardware constraints.
  • 16.8% average ICL gain across six MLLMs on eight diverse benchmarks, vs. 7.6% for the prior SOTA (MMICES).

Results

  • Average ICL gain: ContextNav 16.8% vs. MMICES (prior SOTA) 7.6%; VL-ICL 7.7%; Random Sampling −2.4% (degrades performance).
  • Per-model average accuracy (base → +ContextNav):
  • Gemini-2.0-flash: 0.542 → 0.604 (+0.062)
  • Gemini-1.5-flash: 0.505 → 0.574 (+0.069)
  • GPT-4o: 0.484 → 0.547 (+0.063)
  • Qwen2.5-VL-7B: 0.440 → 0.480 (+0.040)
  • InternLMX2.5-7B: 0.287 → 0.312 (+0.025)
  • Phi-3.5V-4.2B: 0.332 → 0.346 (+0.014)
  • ContextNav outperforms all baselines (zero-shot, random sampling, VL-ICL, MMICES) on every model across BlindTest, MME-RealWorld, CharXiv, GVL, MathVision, CLEVR, FOMI, and TextOCR.
  • Strongest absolute gains observed on larger/closed-source models (Gemini, GPT-4o).

Limitations

  • Agent policy defaults to Gemini-2.0-flash (closed-source), introducing API dependency and cost.
  • Multi-timestep closed-loop operation introduces inference latency compared to one-shot retrieval; not characterized in the provided text.
  • Embedding pipeline requires GPU resources (A100 80G), limiting accessibility for resource-constrained deployments.
  • Gains are smallest on smaller open-source models (Phi-3.5V: +1.4%), suggesting the approach may be less impactful when the MLLM itself has limited reasoning capacity.
  • OGG structure must be manually specified per domain; generalization to entirely new task types is unclear.

Relevance to Agentic AI / LLM Agents

ContextNav is a direct instantiation of the tool-using, memory-augmented, closed-loop agent paradigm applied to a concrete NLP problem (ICL context selection), making it a strong case study for agentic workflow design. The OGG is a formalized action-space representation that enforces compositional constraints on multi-step agent plans—a technique directly applicable to other complex agentic pipelines where operation ordering matters. The feedback-driven memory update mechanism (\(M_{\tau+1} = M_\tau \cup (S_\tau, \phi_\tau)\)) demonstrates how agents can self-improve orchestration strategies without gradient updates, a key concern in autonomous agent research. For researchers tracking LLM agents, this paper shows that agentic context curation yields substantially better downstream task performance than heuristic retrieval, validating the argument that agentic scaffolding can replace brittle static pipelines.