Skip to content

Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment

🕒 Published (v1): 2026-03-24 18:25 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EnterpriseArena is a 132-month CFO simulator that benchmarks LLM agents on long-horizon resource allocation under uncertainty in a FinTech lending firm. Across 23 LLMs and four agent frameworks, only 15.4% of trials survive the full horizon, and the best agent (\(34.7M terminal valuation) reaches just 7% of human expert performance (\)476.7M). Model scale does not reliably predict success; framework scaffolding has a larger and more surprising effect.

Problem

Existing agent benchmarks test largely reactive competence with immediate feedback, explicit action spaces, and locally recoverable errors. No benchmark simultaneously requires: (1) hard resource budgets where both actions and observations consume scarce capacity; (2) long horizons (132 months) demanding coherent multi-step strategy; (3) latent consequences where state and outcomes are only partially revealed over time; (4) non-stationary dynamics driven by exogenous macroeconomic regime shifts. The capability gap of LLM agents under these joint conditions is unmeasured.

Method

The simulator is formalized as a constrained partially observable MDP with latent rewards under non-stationary dynamics. Internal enterprise state (revenue, cash, user base) evolves via Gaussian perturbations \(x'_i = x_i + \epsilon_i\), \(\epsilon_i \sim \mathcal{N}(0, \sigma_i^2)\) calibrated to real-world indicator volatility. External dynamics follow a deterministic but hidden trajectory derived from anonymized macroeconomic data spanning expansion/neutral/recession phases. At each monthly timestep the agent selects exactly one of three actions: book_closing (reconciliation, which unlocks ground-truth financial statements), fund_raising_request (specifying instrument type and amount \(A_\text{requested}\); actual receipt is \(A_\text{actual} = f \cdot A_\text{requested}\) with \(f \sim U(0.7, 1.0)\), delayed \(d \sim U_\mathbb{Z}(1,6)\) months), or pass. The agent has at most 20 tool calls per timestep across four observation tools. Survival requires cash \(\geq 0\) at every step; performance is scored as \(\text{Score}_T = \text{Rev}_T \times 5 + \text{Cash}_T - 5000 \cdot N_\text{tools}\). Four frameworks are evaluated: ReAct (23 backbone LLMs), Claude Code (Opus 4.7), Codex CLI (GPT-5.5), and OpenClaw (DeepSeek-V4). Human baseline: five finance experts with 10+ years of experience.

Key Contributions

  • Formal definition of long-horizon resource allocation under uncertainty as a distinct LLM agent evaluation target, operationalized as a constrained POMDP
  • EnterpriseArena: a 132-month CFO simulator grounded in transformed firm-level financials, anonymized macroeconomic trajectories, and GAAP-validated accounting rules
  • Large-scale empirical evaluation (23 LLMs × 4 frameworks) revealing systematic failure modes: wrong observation allocation, mis-timed fundraising, and capital under-sizing

Results

  • Overall survival: 15.4% across all trials; 13.0% for ReAct alone, 33.3% for other frameworks
  • Human baseline: 60% survival, $476.7M avg terminal score
  • Best agent: Codex CLI + GPT-5.5, 60% survival, $34.7M — ~7% of human score
  • Scale paradox: Llama-3.1-8B achieves \(30.6M and 40% survival, outperforming Qwen3.5-397B-MoE (\)16.0M) and all seven tested closed-source frontier models under ReAct
  • Framework effect: GPT-5.5 achieves 0% survival under ReAct but 60% under Codex (114.6 actions vs. 17.2); DeepSeek-V4 achieves 60% under ReAct (16.2 tools/month) but drops to 20% under OpenClaw (0.27 tools/month)
  • Crisis cascades: 100% of models pass the 1st crisis, ~50% survive the 2nd, <20% reach the 3rd
  • 12 of 23 ReAct backbone models never survive a single full run

Limitations

  • Each model is evaluated on only 5 runs per configuration, making survival percentages (in steps of 20%) statistically coarse
  • Human baseline consists of only 5 experts, limiting statistical power for that reference point
  • The simulator is calibrated to a single domain (FinTech consumer lending); generalization to other enterprise settings is untested
  • Anonymization and stochastic noise may not fully prevent models from exploiting distributional patterns in the macro trajectory
  • The tool penalty coefficient (\(\lambda = 5{,}000\)) and revenue multiple (\(m = 5\)) are manually set; sensitivity to these hyperparameters is not fully characterized

Relevance to Harnesses / Meta-Harnesses

This paper is directly relevant to harness research because it treats the agent framework itself as an independent variable — holding backbone constant and measuring how ReAct, Claude Code, Codex CLI, and OpenClaw produce dramatically different outcomes with the same underlying model. The GPT-5.5/Codex vs. GPT-5.5/ReAct gap (60% vs. 0% survival) is among the clearest empirical demonstrations that harness scaffolding can be a more decisive factor than model capability. The finding that OpenClaw suppresses DeepSeek-V4's tool-use strategy (0.27 vs. 16.2 tools/month) illustrates how harness design can inadvertently prune the very behaviors that make a model effective. For anyone building or evaluating meta-harnesses, EnterpriseArena offers a reproducible stress-test where harness interaction effects are measurable and consequential.