Skip to content

Automated Analysis of Sustainability Reports: Using Large Language Models for the Extraction and Prediction of EU Taxonomy-Compliant KPIs

🕒 Published (v1): 2025-12-30 15:28 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a structured benchmark of 190 EU corporate sustainability reports and evaluates five LLMs across four tasks: multi-label activity classification, binary activity verification, quantitative KPI regression, and a multi-step agentic pipeline. LLMs achieve moderate qualitative performance (best F1=0.311) but completely fail zero-shot KPI regression (all R²<0). A parallel "generate-then-verify" agentic architecture modestly outperforms single-step inference (F1=0.3285 vs. 0.311).

Problem

EU Taxonomy compliance requires companies to manually identify relevant economic activities and extract financial KPIs (Turnover, CapEx, OpEx) from dense legal and internal documents—a process that is error-prone, inconsistent, and scales poorly. No public structured benchmark existed for this multi-task IE problem, blocking systematic LLM evaluation.

Method

Dataset construction: 190 annual PDF reports (2023 fiscal year, primarily Germany/Austria) converted to Markdown via the Marker tool; ground truth extracted via a semi-automated LLM+human pipeline (Gemini Flash 2.0 zero-shot → LLM self-verification → human review).

Four evaluation tasks: 1. Multiclass activity prediction: multi-label classification under three context conditions—company name only, structured metadata, or full annual report (Taxonomy section removed). 2. Binary activity classification: single-activity yes/no classification on 30 companies × 116 active Taxonomy activities. 3. KPI regression: predict Turnover/CapEx/OpEx eligible percentages and absolute values from metadata or report text. 4. Multi-step agentic workflow (via Google ADK): two architectures—(a) sequential: summarize report → generate candidates from summary → binary verification agent; (b) parallel: Gemini Flash 2.5 generates high-recall candidates from full report while simultaneously generating a summary, then a verification agent filters candidates using the summary.

Models: Gemini 2.0 Flash, Gemini 2.5 Flash, Llama 4 Maverick 17B, Gemma-3-12b-it, Mistral-Small-3.1-24B.

Key Contributions

  • First publicly released structured benchmark for EU Taxonomy compliance combining qualitative activity labels and quantitative KPI ground truth (190 companies).
  • First systematic multi-task LLM evaluation covering both activity classification and KPI regression in the same framework.
  • Empirical demonstration of the "paradox of context": full annual reports degrade zero-shot classification versus concise metadata.
  • Comparative evaluation of two agentic pipeline architectures; identification of sequential summarization as a lossy preprocessing step.
  • Confidence calibration analysis showing ECE as high as 0.684, invalidating self-reported model confidence for this domain.

Results

  • Best single-step F1: Gemini Flash 2.5 with metadata = 0.311 (precision 0.274, recall 0.493).
  • Parallel generate-then-verify pipeline: F1 = 0.3285; verification step raised precision from 0.211 to 0.344 at modest recall cost (0.540 → 0.402).
  • Sequential pipeline: F1 = 0.2957, below the single-step baseline—information loss from prior summarization step.
  • Binary classification context trade-off: metadata → precision 67.86%, recall 10.80%; full report → precision 48.65%, recall 20.45%.
  • KPI regression (all models fail): Turnover R²=−0.2106, CapEx R²=−0.2856, OpEx R²=−0.0020; absolute-value KPIs R²<−194. Models are outperformed by a mean-prediction baseline.
  • Paradox of context: average F1 with full report = 0.2264 < metadata = 0.2493 < company name = 0.2492.
  • Feature importance (Random Forest on F1): log_employees (0.239) and log_revenue (0.238) account for >47% of variance in model performance; sector Utilities highest F1 (0.423), Financials lowest (0.155).
  • Calibration: ECE ranges 0.353–0.684; Gemini Flash 2.5 (highest recall) is most poorly calibrated.

Limitations

  • Geographic concentration: 63% Germany, 20% Austria—limits generalizability to other EU regulatory contexts.
  • Evaluation is prompting-only; fine-tuned or RAG-augmented models are not tested, so results represent a lower bound.
  • Dataset reflects 2023 mandatory reporting scope (large public-interest entities under NFRD), skewing toward very large multinationals.
  • Results are a time-bound snapshot; rapidly advancing models may shift the quantitative findings.

Relevance to Harnesses / Meta-Harnesses

This paper operationalizes two distinct harness patterns—sequential chaining and parallel generation with downstream verification—and benchmarks them head-to-head on a real-world IE task, producing actionable design lessons. The finding that sequential summarization as an intermediate harness step is lossy (F1 drops below single-step baseline) directly informs how preprocessing stages should be architected in multi-step pipelines. Conversely, the parallel generate-then-verify pattern—where a high-recall generator and a summary are produced concurrently, then a precision-focused verifier filters the candidates—validates a canonical harness composition pattern. The "paradox of context" result (noisy full-document input degrades performance relative to structured metadata) has broad implications for how harnesses should preprocess and chunk documents before passing them to LLM stages.