Skip to content

IPO Finance Agent: Evaluation of LLM Financial Analysts beyond Finance Agent v2, with Automated Rubric Generation -- the Case of the SpaceX (SPCX) IPO

🕒 Published (v1): 2026-06-22 08:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IPO Finance Agent extends the Finance Agent v2 benchmark to SEC S-1 registration statements (IPO filings), replacing naive chunk retrieval with contextual retrieval and introducing an automated evaluator-optimizer pipeline for rubric generation. The best evaluated model (Qwen 3.7 Max) reaches 79.4% accuracy at $0.30/query, outperforming Finance Agent v2's leaderboard ceiling (Gemini 3.5 Flash at 57.9%) while costing less.

Problem

Finance Agent v2 evaluates LLMs only on periodic 10-K/10-Q filings from public companies; its naive fixed-size chunk retrieval fails entirely on the longer, cross-referential structure of S-1 IPO registration statements. No existing benchmark targets IPO due diligence tasks (pro forma accounting, governance disclosures, underwriting-sensitive risk factors). Rubric construction for open-ended financial QA also does not scale manually to new document types.

Method

Two architectural changes over Finance Agent v2:

  1. Contextual retrieval harness: Each document chunk is augmented with an LLM-generated document-aware description before embedding and indexing, following Anthropic's contextual retrieval approach. This allows cross-referential S-1 passages (risk factor + MD&A paragraph + pro forma footnote) to be retrieved against a single query with no lexical overlap.

  2. Evaluator-optimizer rubric pipeline (4 stages):

  3. Stage 1 — Fact extraction: GLM-5.1 pulls atomic, verifiable claims from answers of an ensemble (GLM-5.1, Qwen 3.7 Max, Kimi K2.6, MiMo-2.5 Pro).
  4. Stage 2 — Consolidation: Quantitative facts are deterministically clustered; qualitative facts are merged via incremental LLM consolidation. Only facts with agreement \(\geq 2\) models advance.
  5. Stage 3 — Rubric induction: Consolidated facts induce a draft rubric with tiered criteria; a deterministic schema-validation pass runs before LLM evaluation.
  6. Stage 4 — Quality evaluation: GLM-5.1 scores on specificity, atomicity, tier correctness, and coverage, then routes to stop (deploy), repair (structural reorganization, no new facts injected, hallucination guard on numeric payloads), or enrich (re-extraction targeting specific coverage gaps, then deduplication). Both repair and enrich loop back to Stage 4. Final human expert review gates deployment.

Key Contributions

  • 1,000-question IPO due-diligence benchmark across six domains (segment economics, KPI quality, governance/control, accounting recast mechanics, execution risk, valuation/underwriting); 70 SpaceX S-1 questions released publicly.
  • Contextual-retrieval harness that fixes the failure of Finance Agent v2's naive chunking on long S-1 filings.
  • Automated evaluator-optimizer rubric-generation pipeline reducing manual annotation effort; human experts only review final rubrics.
  • Cost-accuracy Pareto analysis across frontier models on IPO diligence tasks.

Results

  • Qwen 3.7 Max: 79.4% accuracy at $0.30/query (best accuracy).
  • MiMo-2.5 Pro: 76.8% accuracy at $0.05/query (Pareto-optimal cost efficiency).
  • Finance Agent v2 leaderboard ceiling: Gemini 3.5 Flash at 57.9% for $2.51/query.
  • Finance Agent v2 cheapest entry: MiniMax M3 at 48.3% for $0.32/query.
  • Both IPO Finance Agent top models exceed FABv2's accuracy ceiling while undercutting even its cheapest entry on cost.

Limitations

  • Benchmark publicly releases only 70 of 1,000 questions (SpaceX S-1 only); the remainder are private, limiting full reproducibility.
  • Hallucination safeguards during repair check against the preceding rubric version, not the source filing directly.
  • NVIDIA Nemotron 3 Ultra was excluded from the fact-extraction ensemble due to excessive failures, reducing ensemble diversity.
  • Single flagship case study (SpaceX S-1) for the public split; generalization across other S-1 filings relies on the private held-out set.
  • Rubric quality ultimately depends on the ensemble models' answer quality; systematic blind spots shared across all ensemble members propagate into rubrics.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instantiation of the meta-harness pattern: a harness (the agentic retrieval + evaluation system) that itself orchestrates other LLM calls to generate and validate the evaluation rubrics used to score models within the same harness. The evaluator-optimizer pipeline is an explicit meta-harness loop — one model generates artifacts (rubrics), another evaluates them, and a controller routes repair/enrich/stop branches, instantiating the generate-evaluate-repair cycle that is the canonical skeleton of iterative meta-harnesses. The contextual retrieval component illustrates how harness-level architecture choices (not model capability alone) dominate benchmark outcomes — a recurring finding in harness design research. The paper also demonstrates automated rubric generation as a scalable alternative to human-authored evaluation criteria, which is directly relevant to anyone building meta-harnesses that must auto-construct their own grading logic for novel task domains.