Skip to content

FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation

🕒 Published (v1): 2025-11-10 17:22 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FinRpt is the first benchmark, dataset, and evaluation system for automated Equity Research Report (ERR) generation. The authors also propose FinRpt-Gen, a nine-agent framework that decomposes ERR generation into extraction, analysis, and prediction modules. A small fine-tuned open-source model (Qwen2.5-7B-SFT+RL) outperforms GPT-4o and Gemini-2.5-Pro on most metrics.

Problem

No formal task definition, training dataset, or domain-appropriate evaluation framework existed for automatic ERR generation. Existing financial NLP benchmarks focus on short-form tasks (QA, sentiment), and generic metrics like ROUGE-L and BERTScore are insufficient for assessing financial accuracy, stock-trend prediction correctness, or professional writing quality in multi-section reports.

Method

Task formulation: Given a stock ticker \(s\) and analysis date \(t\), the system retrieves a structured input \(S = [O, F, A, N, P, M]\) (company info, financial indicators, announcements, news, stock prices, market indices) and generates a six-section ERR \(R = [R_{fin}, R_{news}, R_{manage}, R_{risk}, R_{invest}, R_{rec}]\).

Dataset Construction Pipeline: The pipeline runs FinRpt-Gen (with GPT-4o as each agent) to auto-generate ERRs for 6,825 (stock, date) pairs from the CSI800 index (Sep–Nov 2024). Three enhancement steps then correct quality: (1) a Recommendation Rating Corrector re-infers samples where the generated buy/sell label contradicts ground-truth trend; (2) an Expert-written ERRs Corrector fetches real analyst reports from Eastmoney and prompts GPT-4o to reconcile accuracy and logical consistency; (3) an LLM Polisher improves readability.

FinRpt-Gen architecture: Nine agents in three sequential modules: - Information Extraction Module (4 agents): news ranker, income/balance/cash statement extractors. - Information Analysis Module (4 agents): finance, news, status, and risk analysts that synthesize sections from extracted data. - Prediction Module (1 agent): integrates all analysis plus historical prices/indices to produce \(R_{invest}\) and \(R_{rec}\).

Training: The four analysis/prediction agents are fine-tuned with SFT+LoRA (objective: \(\max_{\Delta\theta} \sum_{(X,Y)\in\mathcal{D}_{demo}} \log P(Y|X; \theta_0+\Delta\theta)\)). The Prediction Agent is further trained with DAPO (a GRPO variant) using a composite reward \(\text{Reward}(Y,Y^*) = 0.6\cdot\text{ACC}(R_{rec}) + 0.2\cdot\text{ROUGE-1}(R_{invest}) + 0.2\cdot\text{ROUGE-L}(R_{invest})\).

Evaluation system: 11 metrics: CompletionRate, Accuracy (buy/sell), ROUGE-L, BERTScore, NumberRate (ratio of numeric content), plus six LLM-judged dimensions (Financial Numeric, News, CMI, Invest, Risk, Writing) scored via pairwise GPT-4o judge with position-bias cancellation and an Adjusted Win Rate formula.

Key Contributions

  • First formal definition of the ERR generation task and a generalized pipeline for similar report generation domains.
  • FinRpt benchmark: 6,825 bilingual (Chinese + English) ERR samples with automated quality enhancement; 5,556/617/652 train/val/test split.
  • 11-metric evaluation system combining NLP metrics with financial-domain LLM judges.
  • FinRpt-Gen: nine-agent, three-module framework; publicly available with all code and datasets.
  • Demonstration that SFT+RL on the auto-constructed dataset enables a 7B open-source model to match or exceed frontier closed-source models.

Results

  • FinRpt-Gen (Qwen2.5-7B-SFT-RL): Accuracy 55%, ROUGE-L 49.06, BERTScore 82.43, NumberRate 95.15% — best overall.
  • FinRpt-Gen (GPT-4o): Accuracy 51%, ROUGE-L 48.44, BERTScore 82.09 — best without fine-tuning.
  • Single GPT-4o (no framework): Accuracy 48%, ROUGE-L 40.72, BERTScore 79.57 — substantially below FinRpt-Gen (GPT-4o) on ROUGE-L (+7.7).
  • Single Gemini-2.5-Pro: ROUGE-L 41.79, BERTScore 80.29 — best single-LLM baseline.
  • After SFT alone, Qwen2.5-7B achieves ROUGE-L 48.83, beating FinRpt-Gen (Gemini-2.5-Pro) at 48.58.
  • RL adds ~1 accuracy point and ~0.2 ROUGE-L over SFT-only for Qwen2.5-7B.

Limitations

  • Dataset covers only CSI800 (Chinese large-cap stocks, Sep–Nov 2024); generalization to other markets or time periods is not validated.
  • Ground-truth ERRs are themselves LLM-generated (GPT-4o + human correction), not human-written from scratch; silver-standard ceiling may be limiting.
  • The "Expert-written ERRs Corrector" relies on retrieved analyst reports whose quality and availability vary.
  • LLM-as-judge (GPT-4o pairwise) evaluation inherits known biases (verbosity, self-preference) despite position-bias mitigation.
  • RL training requires A100-class hardware; applicability to resource-constrained settings is not discussed.

Relevance to Harnesses / Meta-Harnesses

FinRpt's Dataset Construction Pipeline is itself a meta-harness: it invokes FinRpt-Gen (the task-level multi-agent harness) as a sub-process to auto-generate training data, then wraps it in three automated correction stages — a recursive pattern where a harness both produces and quality-controls its own training corpus. This exemplifies a self-bootstrapping harness architecture where the orchestration layer (pipeline) sits above the agent framework (FinRpt-Gen) and applies rule-based and LLM-based validators to enforce output quality. The nine-agent decomposition with explicit module boundaries (extraction → analysis → prediction) is a concrete example of how multi-stage harnesses can be designed for structured, heterogeneous output tasks, and the SFT+RL training loop shows how harness-generated data can close the loop to specialize component agents.