Skip to content

FinAcumen: Financial Multimodal Reasoning via Self-Evolving Experience Memory Harness

🕒 Published (v1): 2026-06-16 08:00 UTC · Source: Arxiv · link

Why this paper was selected

Self-evolving experience memory harness; multimodal financial reasoning; novel memory loop

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FinAcumen wraps a frozen 8B vision-language model with a self-evolving experience memory bank (FM) and a deterministic financial tool suite (FT) to perform financial multimodal reasoning without any parameter updates. The FM separates reusable strategies (from successful trajectories) from failure-derived guard rules, activating retrieved experience only when cosine similarity to the query exceeds a calibrated threshold \(\tau\), with a hard fallback when no entry qualifies. Across four financial benchmarks the framework raises a Qwen3-VL-8B baseline by up to 41 accuracy points and surpasses GPT-4o on FinMMR Easy/Medium.

Problem

Tool-augmented agents (ReAct-style) are stateless across episodes: they repeatedly rediscover the same reasoning strategies and failure modes. Prior cross-episode memory systems (Reflexion, ExpeL, ReasoningBank) conflate successful and failed experiences, retrieve weakly related guidance, and expand the bank without controlling retrieval quality — problems especially damaging in finance, where irrelevant or hallucinated context directly corrupts numerical conclusions and hallucination-penalizing benchmarks (FinTMMBench, FinMME) make this explicit.

Method

Memory bank construction. For each training problem, the framework samples \(K\) diverse solution trajectories from the frozen \(\pi_\theta\) and scores each against the gold answer \(y^*\). A summary agent synthesizes: $$ (S, C) = \text{Summarize}!\left({(\text{Traj}k, s_k)}^K,\, y^*\right), $$ where \(S\) are generalised strategies from correct trajectories and \(C\) are guard rules from incorrect ones. Each \((S, C, q, y^*)\) tuple is stored as a bank entry \(m \in M\) with a pre-computed embedding \(z_m\).

Selective memory activation. At inference, query \(x\) is embedded via shared encoder \(E(\cdot)\). Candidate entries are those satisfying: $$ \text{sim}(x, m) = \frac{E(x) \cdot z_m}{|E(x)|\,|z_m|} \geq \tau. $$ The top-\(k_{\max}\) qualifying entries are deduplicated, ranked, and rewritten by an annotator agent into problem-specific directives prepended to the solver prompt. If \(M_x = \emptyset\), the model falls back to base reasoning — no memory is injected.

Financial Tools (FT). Four deterministic tools share the same calling schema across all configurations: (1) a Numerical Reasoning Engine executing code in a persistent namespace \(\sigma_t\) to enable stateful cascaded arithmetic \((o_t, \sigma_t) = \text{Execute}(\text{code}_t, \sigma_{t-1})\); (2) a Grounded Data Retrieval tool returning OHLC/news/quarterly rows via \(r = \text{Lookup}(\text{symbol}, \text{date}, \text{indicator})\); (3) a Visual Structure Decoder for OCR and chart re-parsing; (4) an Answer Consolidation Gate enforcing source-traceability, unit compatibility, and format validity before any answer is emitted.

The backbone (\(\pi_\theta\), Qwen3-VL-8B-Instruct) is frozen throughout; FM operates strictly at the prompt level.

Key Contributions

  • Selective experience memory architecture that explicitly separates strategy entries (from successes) from guard-rule entries (from failures), unlike prior systems that conflate both.
  • Threshold-gated retrieval with an empty-return fallback, preventing noisy injection under retrieval uncertainty.
  • Stateful persistent-namespace code execution enabling multi-step financial arithmetic without re-initialising intermediate quantities.
  • Source-traceability enforcement at answer emission via the Answer Consolidation Gate, not as a post-hoc auditor.
  • Empirical evidence that a frozen 8B VLM with this harness matches or exceeds finance-specialised fine-tuned models and proprietary GPT-4o results on multiple settings.

Results

  • BizBench SEC-NUM (2000 items): Base 27.67 → FinAcumen 68.65 (+41.0 pp); best General LLM (GPT) 79.30.
  • FinMMR Easy (1200 items): Base 59.17 → 81.67 (+22.5 pp); GPT-4o 78.00 (FinAcumen exceeds).
  • FinMMR Medium (1200 items): Base 45.75 → 67.00 (+21.25 pp); GPT-4o 63.33 (FinAcumen exceeds).
  • FinMMR Hard (1000 items): Base 31.40 → 46.10; GPT-4o 63.42 (gap remains).
  • FinTMMBench (1136 items): Base 16.20 → 19.98; best General LLM 21.53 (gap remains).
  • FinMME (2220 items, raw avg): Base 27.84 → 51.08.
  • Ablation (FinMMR Easy): Base 59.17 → +FT 74.08 (+14.91 pp) → +FM 81.67 (+7.59 pp additional).
  • Memory accumulation (BizBench): FT-only at \(n=0\): 55.15%; at \(n=2400\): 68.65% (+13.5 pp); saturation plateau beyond \(\approx1800\) entries.
  • Threshold sensitivity: \(\tau=0.65\) (moderate regime) achieves 79.8% hit rate at 1200 entries; \(\tau \geq 0.75\) (strict) stays below 37% even at 1200 entries.
  • Outperforms all financial LLMs tested (Llama-Open-Finance-8B, FinLLaVA, Fin-R1) across shared evaluation columns.

Limitations

  • Still trails the best General LLM on BizBench, FinMMR Hard, and FinTMMBench, indicating the 8B backbone imposes hard ceilings on planning and temporal retrieval.
  • FM bank saturates around \(n=1800\) entries; marginal gains become negligible, limiting scalability of naive bank expansion.
  • FM is frozen read-only at test time — no online/continual updating from test trajectories.
  • Evaluated on a single backbone (Qwen3-VL-8B-Instruct); generalisability to other VLMs is untested.
  • FinMME scoring required author-supplemented domain labels because official per-sample labels were unavailable, introducing a methodological caveat for that benchmark.
  • \(\tau\) and \(k_{\max}\) require per-domain calibration; optimal values likely shift across benchmarks.

Relevance to Harnesses / Meta-Harnesses

FinAcumen is a canonical example of a inference-time meta-harness: it wraps an unmodified frozen model with a persistent external memory and deterministic tool layer, altering behavior purely through prompt conditioning and execution routing — no gradient flows into the backbone. The self-evolving memory construction loop (multi-trajectory sampling → strategy/caution distillation → bank expansion) is a harness-level self-improvement mechanism that generalises to any domain where labeled trajectories can be scored, making it a transferable harness design pattern. The threshold-gated fallback is a principled harness intervention policy — deciding when to augment vs. when to trust the base model — a key unsolved design problem in meta-harnesses. The strict separation of the FT execution environment from the FM memory layer also demonstrates a clean composability principle: swapping either module leaves the other's interface intact, which is the architectural ideal for modular harness stacks.