Skip to content

What Factors Affect LLMs and RLLMs in Financial Question Answering?

🕒 Published (v1): 2025-07-11 06:37 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically benchmarks 5 LLMs and 4 RLLMs (reasoning LLMs with Long CoT) across three intervention axes—prompting methods, agentic frameworks, and multilingual alignment—on the FAMMA financial QA benchmark. The central finding is that all three axes improve LLMs by lengthening reasoning chains (simulating Long CoT), but are largely ineffective for RLLMs, which already possess native Long CoT. The paper concludes that unlocking RLLM performance requires richer agent collaboration mechanisms rather than prompt engineering.

Problem

Existing financial QA work focuses on improving model performance, but no prior study systematically isolates why various methods help or fail for both standard LLMs and the newer class of RLLMs. The factors that govern performance—prompting strategy, agentic orchestration, multilingual alignment—have not been studied comparatively across these two model families, leaving practitioners without principled guidance on which interventions to apply.

Method

The study evaluates 9 backbones (Llama-3.1-8B, GPT-4o-mini, Gemini-1.5-flash, Qwen-2.5-32B, DeepSeek-V3, DeepSeek-R1-Distill-Qwen-32B, GPT-OSS-20B, Qwen-3-14B, O4-mini) on FAMMA's 1,945-entry text-only subset (multimodal data OCR-converted). Three intervention axes are tested:

  1. Prompting: Direct, Zero-shot CoT ("let's think step by step"), Plan-and-Solve (two-stage decomposition prompt).
  2. Agentic frameworks: Self-Refine (one-iteration self-critique loop) and S3 Agent (three parallel agents for superficial/semantic/sentiment perspectives).
  3. Multilingual alignment: Direct (English prompt + local-language question), Translate-en (translate to English first), Cross-lingual Prompting (CLP; two-stage alignment + task solver).

Performance is measured as accuracy on arithmetic and non-arithmetic sub-splits at Easy/Medium/Hard difficulty. Output token counts are tracked to quantify reasoning length.

Key Contributions

  • Empirical evidence that prompting and agentic frameworks improve LLMs by increasing output token count (proxy for reasoning depth), not through task-specific knowledge.
  • Demonstration that RLLMs' native Long CoT makes standard prompting interventions redundant and sometimes harmful (Plan-and-Solve degrades RLLM performance).
  • Evidence that multilingual alignment methods (Translate-en, CLP) improve LLMs by extending reasoning length; RLLMs self-align through Long CoT, making explicit alignment methods largely unnecessary.
  • Comparison of DeepSeek-R1-Distill-Qwen-32B vs. Qwen-2.5-32B showing Long CoT distillation yields +7.4% average gain, and that LLMs cannot close this gap via complex frameworks without matching RLLM token budgets.
  • Scaling-law confirmation: within Qwen-3, larger non-thinking models produce longer outputs and higher accuracy; activating thinking mode provides large gains but can introduce overthinking at equal or smaller scale.

Results

  • Plan-and-Solve is the best single prompting method for LLMs on most models; e.g., Llama-3.1-8B: Direct 16.50% → Plan-and-Solve 22.21% overall.
  • S3 Agent yields the largest gain for small LLMs: Llama-3.1-8B Direct 16.50% → S3 Agent 24.62%.
  • DeepSeek-V3 (strongest LLM): 58.86% overall (Direct); prompting/agentic methods show minimal delta (< 2%).
  • O4-mini (strongest RLLM): 65.29% overall (Direct); all methods stay within ±1.5% of baseline—no meaningful gain.
  • Multilingual (CLP): best for LLMs, e.g., Qwen-2.5-32B Chinese+French: Direct 38.44% → CLP 40.91% (+2.65%); for RLLMs (e.g., Qwen-3-14B): Direct 55.20% → CLP 52.21% (−2.99%).
  • Distillation gap: DeepSeek-R1-Distill-Qwen-32B outperforms Qwen-2.5-32B by ~7.4% on average across all methods.
  • Token–performance correlation: within LLMs, more tokens → higher accuracy; Plan-and-Solve averages ~1,132 tokens for Llama-3.1-8B vs. 464 for Direct.

Limitations

  • All models evaluated with a single run; no confidence intervals or significance testing reported.
  • FAMMA's text-only (OCR) subset is used because RLLMs are unimodal; results may not generalize to the full multi-modal benchmark.
  • S3 Agent and Self-Refine were not designed for financial QA; only minor task-specific modifications were made, potentially underestimating agentic gains.
  • CLP and other multilingual methods are evaluated on only two non-English languages (Chinese, French).
  • Overthinking in RLLMs is identified as a problem but no mitigation is proposed or tested.

Relevance to Harnesses / Meta-Harnesses

This paper is a controlled ablation study of agentic harness components—specifically Self-Refine (iterative self-critique) and S3 Agent (multi-role parallel reasoning)—measured against prompt-only baselines on a domain-specific task. Its core finding—that standard harness primitives (reflection loops, multi-agent ensembles) work by extending reasoning length rather than through architectural novelty—is directly relevant to harness design: it implies that harness value for RLLM backends must come from genuine task decomposition and structured output enforcement rather than reasoning augmentation. The evidence that "more complex agent mechanisms" are the path forward for RLLMs suggests that future meta-harness work should focus on coordination protocols (routing, role specialization, output schema enforcement) rather than CoT scaffolding.