Skip to content

AGC-Bench: Measuring Artificial General Creativity

🕒 Published (v1): 2026-07-01 16:31 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AGC-Bench is a meta-benchmark for artificial general creativity assembled via a PRISMA-compliant systematic review of 3,101 papers (497 unique benchmarks identified), paired with an agentic onboarding harness that converts source-paper benchmarks into runnable HELM-style scenarios. Across 83 LLMs evaluated on 67 text-only datasets spanning six creativity domains, factor analysis recovers a single creativity factor \(C\) explaining 81.5% of variance, analogous to the \(g\) factor of general intelligence. Frontier proprietary models lead, but domain-level dissociations are substantial even among top models.

Problem

The AI creativity evaluation landscape is fragmented across hundreds of heterogeneous benchmarks, each using its own scoring protocol, model roster, and prompting convention. This makes it empirically intractable to ask whether LLM creative ability generalizes across domains (the "artificial general creativity" question), and existing multi-domain consolidations are too small (e.g., 9 tasks × 17 models in CreativityPrism) to support psychometric inference at scale.

Method

Systematic curation. A PRISMA-compliant six-stage pipeline (literature harvesting → LLM pre-screening with gpt-4.1 → data-access verification with gemini-2.5-flash → dual human review → structured extraction with gemini-2.5-pro → hybrid deduplication) processed 3,101 papers, yielding 497 unique benchmarks; 327 Tier-1 (API-runnable) candidates entered onboarding.

Agentic onboarding harness (benchmark-onboarder). Built with Claude Code on claude-opus-4.6, a nine-state pipeline converts each source-paper benchmark into a HELM-compatible scenario. Design principles: verbatim prompt extraction, source-paper scoring metric preserved, regex/code-based extraction before LLM inference, human review at every accept step. Output: 159 valid scenarios, 78 launched in this release.

Evaluation. 83 models × 67 text-only datasets, scored \(n = 50\) items per (model, dataset) cell with a frozen seed. For 24 LLM-judge benchmarks, a three-judge planned-missing 2-of-3 panel (gemini-3-flash-preview, grok-4.1-fast, gpt-4.1-mini) is calibrated with Judge Response Theory (JRT)—a Bayesian graded-response model recovering per-unit ability \(\theta\) alongside per-judge discrimination and severity \(\beta\)—to correct systematic leniency/severity bias. The AGC-Bench composite is the per-model mean of dataset-level \(z\)-scores.

AGC-Judge. Qwen3-30B-A3B-Instruct-2507 fine-tuned via LoRA on 48,299 JRT-corrected ratings to reproduce the three-judge ensemble at low cost.

Factor analysis. Per-(model, domain) composite matrix (\(83 \times 6\)) submitted to exploratory factor analysis to test dimensionality of creative ability.

Key Contributions

  • AGC-Bench: 78-dataset meta-benchmark (67 text-only, 11 multimodal) covering brainstorming, problem solving, STEM, narrative, figurative language, and humor; public leaderboard over 83 models.
  • Agentic onboarding harness: reproducible nine-state pipeline converting literature benchmarks into HELM-compatible scenarios; designed for incremental expansion.
  • AGC-Judge: open-weight Qwen3-30B LoRA scorer trained on 48,299 JRT-calibrated ratings; matches the three-judge ensemble (\(\rho \geq 0.97\) in-distribution, \(\rho = 0.83\) on held-out benchmarks).
  • PRISMA inventory: full annotated catalog of 497 creativity benchmarks (modality, scoring protocol, source paper) released for community use.
  • AGC-Human: paired human–LLM subset across five tasks (224 humans, 125 LLM evaluations) enabling direct scale-comparable comparison.

Results

  • \(C\) factor: first eigenvalue = 4.89 (only above Kaiser); single creativity factor explains 81.5% of variance (\(\alpha = 0.96\)) across six text-only domains; correlated with but separable from general knowledge and reasoning.
  • Leaderboard: claude-opus-4.7, gpt-5.4, gpt-5.5 top the overall composite; open-weight glm-5.1, glm-5-turbo close behind.
  • Domain dissociations: kimi-k2.5 tops Problem Solving (+1.02 \(z\)) without appearing in other domain top-5s; claude-opus-4.6-fast leads Figurative Language by a wide margin (+1.36 \(z\)); gemma-4-31b-it enters three domain top-5s despite ranking 9th overall.
  • Creativity vs. reasoning prompt: "be creative" prompting boosts performance substantially more than enabling reasoning mode, confirming the benchmark tracks creativity rather than general reasoning.
  • Human comparison: LLM creativity is more domain-general than human creativity; the top human still leads the top LLM on the five-task AGC-Human subset.
  • AGC-Judge held-out: \(\rho = 0.94\) on held-out items/models; \(\rho = 0.83\) on three fully held-out benchmarks.
  • Rank stability: Spearman \(\rho = 0.99\) between \(k=10\) and \(k=50\) items; Top-10 Jaccard = 1.0 at \(k=50\).

Limitations

  • Only 78 of 497 identified benchmarks are active in this release; the remaining 419 are queued or excluded for scope/methodology reasons.
  • Sampling cap of \(n = 50\) items per (model, dataset) cell trades depth for breadth; absolute scores on any single benchmark may be noisy.
  • No multimodal results are reported despite 11 multimodal scenarios being released as artifacts.
  • JRT calibration applied only to the 24 LLM-judge benchmarks; non-judge benchmarks retain their source-paper metric without cross-benchmark bias correction.
  • Human comparison restricted to five tasks and 201 humans; generalizability of the domain-generality finding is limited.
  • Benchmark contamination (data leakage into model pretraining) is flagged as a known confound in the broader literature but not directly measured here.

Relevance to Harnesses / Meta-Harnesses

AGC-Bench is a direct example of a meta-harness architecture: it does not introduce new evaluation tasks but instead systematizes the conversion of existing heterogeneous benchmarks into a unified, runnable evaluation infrastructure, exactly the pattern that harness/meta-harness research studies. The benchmark-onboarder—a nine-state agentic pipeline with explicit state-machine specification, human-in-the-loop accept gates, and reproducibility guarantees—is a concrete instantiation of an LLM-driven harness-construction workflow, and its design principles (verbatim prompt extraction, source metric preservation, code-before-LLM extraction) are transferable design patterns. The PRISMA-compliant curation layer feeding into the harness demonstrates how systematic literature review can be operationalized as a harness-population mechanism, a scalable model for other domains. The paper also contributes a scoring meta-layer (JRT + AGC-Judge) that abstracts over individual benchmark evaluation protocols, which is a recurring challenge in multi-benchmark harness design.