Skip to content

PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

🕒 Published (v1): 2025-11-18 04:30 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PRISM is a training-free framework for financial document and chunk ranking that combines structured prompt engineering, embedding-based in-context learning (ICL), and optional multi-agent coordination. Its primary contribution is a systematic empirical study showing that simpler non-agentic configurations with well-designed prompts frequently outperform complex multi-agent pipelines. The best configuration achieves NDCG@5 of 0.71818 on FinAgentBench, ranking 3rd overall as the only training-free method in the top three.

Problem

Financial IR over long, domain-specific documents (SEC filings, earnings transcripts) is challenging due to dense text, high relevance sparsity, and variable chunk length. Existing supervised ranking methods require fine-tuning, while unsupervised LLM-based approaches lack systematic guidance on when prompt engineering, ICL, or multi-agent coordination actually helps—and practitioners lack latency/cost baselines for deployment decisions.

Method

PRISM operates on three independent modules applied to document and chunk ranking tasks:

  1. Prompt Engineering (P1–P4): Four progressively refined prompt variants informed by EDA on FinAgentBench (document-type distributions, keyword ratios). P1 uses ReAct-style corpus-informed instructions; P2 adds quantitative domain priors; P3 adds CoT/ToT reasoning scaffolds; P4 streamlines to maximize reasoning clarity and minimize hallucination. P4 outperforms earlier variants, showing reasoning structure matters more than informational density.

  2. ICL Sample Retrieval: A FAISS vector store is built offline from training query-document pairs embedded with OpenAI text-embedding-3 (small or large). At inference, top-k semantically similar exemplars are retrieved and prepended as few-shot demonstrations. ICL is applied selectively—document-level only yields best results; extending to chunk level causes context overload.

  3. Multi-Agent System (MAS): Four architectural variants (A1–A4) of decreasing complexity: A1 is a four-agent parallel ensemble (CEO, Financial Analyst, Operations Manager, Risk Analyst) with score averaging; A2 is a three-stage pipeline with noise filtering before scoring; A3 is a two-stage filter+three-scorer pipeline; A4 is the minimal two-agent configuration. Agents are evaluated across GPT-4o-mini, GPT-4.1, GPT-5-mini, and GPT-5.

Key Contributions

  • Training-free PRISM framework combining prompt engineering, ICL, and MAS for financial IR, evaluated across FinAgentBench, FiQA-2018, and FinanceBench.
  • Systematic ablation study (34 configurations) establishing conditions under which each component provides value, specifically showing that non-agentic P4+document-ICL outperforms all MAS variants.
  • Empirical finding that MAS benefits are model-size dependent: smaller models (GPT-4o-mini) fail in deep agent graphs due to error propagation; GPT-5-mini recovers stability in A3.
  • First latency, token usage, and cost analysis for training-free financial IR on FinAgentBench.

Results

  • FinAgentBench (NDCG@5): Best configuration (Run 19: P4, GPT-5, ICL-5/TE3-S at document level, no chunk ICL) achieves 0.71818 (private), ranking 3rd overall; top two methods used fine-tuned embeddings (30-epoch and three-MiniLM respectively); PRISM trails top by only 0.007 (0.72497 vs 0.71818).
  • FiQA-2018 (NDCG@10 / Recall@100): PRISM (GPT-5, P3, ICL-5) achieves 0.6197 / 0.8206 vs. BM25+Cross-Encoder baseline of 0.3470 / 0.5390—gains of 78.59% and 52.24%.
  • FinanceBench (oracle context): PRISM P1+ICL-9 achieves 147/150 correct (98%), vs. baseline 141/150.
  • MAS degradation: A2 with GPT-4o-mini scores 0.51518 (NDCG@5 private); scaling to GPT-5-mini on A3 recovers to 0.66291.
  • Hybrid (doc-level agentic, chunk-level non-agentic): 0.70685 (Run 33), competitive with best non-agentic.
  • Latency (Run 19): 10.43s per document query, 131.13s per chunk query; per-query cost $0.0155 and $0.2768 respectively; CV < 1.6% across repeated runs.

Limitations

  • All experiments use OpenAI models only (GPT-4o-mini, GPT-4.1, GPT-5-mini, GPT-5); generalization to open-source models unvalidated.
  • FinAgentBench evaluation system (Kaggle) became defunct post-competition, preventing further experiments on that benchmark.
  • ICL at chunk level degrades performance due to context overload; the boundary for optimal ICL scope is task-specific and requires empirical search.
  • MAS gains are conditioned on model size; coordination overhead exceeds benefits for smaller models and fine-grained chunk tasks.
  • Cost ($0.28/complex query) may be prohibitive for high-volume, lower-value retrieval use cases.

Relevance to Agentic AI / LLM Agents

PRISM directly challenges a common assumption in agentic AI: that more agents and more complex coordination graphs yield better performance. Its ablations show that multi-agent pipelines degrade on fine-grained sub-document tasks due to error propagation and coordination overhead, and only recover with sufficiently capable frontier models—a concrete, quantified data point on the model-size threshold for MAS viability. For researchers building agentic retrieval systems (a key component of RAG-based agents), the finding that selective, scope-limited ICL at document level outperforms both naïve ICL and full MAS provides a practical design heuristic. The MAS architectural study (A1–A4) also contributes a grounded taxonomy of agent graph complexity versus task granularity tradeoffs, directly relevant to anyone designing multi-agent pipelines for structured document understanding.