Skip to content

Language Modeling by Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Genesys is a multi-agent LLM system that autonomously discovers novel language model architectures by simulating the research pipeline—ideation, literature search, code generation, pretraining, and evaluation. It uses genetic programming over a factorized code representation (GAU trees) with a Ladder-of-Scales verification strategy to discover 1,062 new architectures at 14M–350M parameters, several of which outperform GPT2 and Mamba2 on standard benchmarks.

Problem

Existing automated scientific discovery (ASD) systems (AI Scientist, CodeScientist) focus on open-ended, loosely verifiable tasks. Neural architecture discovery offers a concrete, verifiable ASD target—clear success criteria and quantitative metrics—but requires deep literature understanding, unbounded code generation, and expensive compute management, none of which prior ASD systems address well.

Method

Genesys combines two subsystems:

LMADE (environment): A knowledge engine (297 LM papers in a vector store, ArXiv/Semantic Scholar/web access) and a verification engine that runs pretraining on SmolLM corpus and evaluates on 29 LM-Eval benchmarks. Architecture designs are represented as GABBase PyTorch modules with symbolic checks for differentiability, causality, and numerical stability.

Genesys (discovery system): Each design is factorized into a GAU (generalized autoregressive unit) tree enabling GP-style mutation and crossover. Designer agents operate in two stages: (1) a proposer–reviewer pair generates and adversarially scores research proposals against the literature; (2) a planner–coder–observer pipeline implements proposals unit-by-unit (Viterbi-style), freezing each validated unit before generating the next—this reduces expected LLM calls exponentially vs. direct prompting. Verifier agents select designs using a four-quadrant fitness×confidence strategy and run a Ladder-of-Scales budget (1,000 trials at 14M → 5 trials at 350M parameters), with designer and verifier nodes running in parallel.

Key Contributions

  • Genesys system: end-to-end multi-agent pipeline for autonomous LM architecture discovery.
  • LMADE environment: reusable discovery environment with knowledge and verification engines.
  • GAU tree factorization: enables GP operations (mutation/crossover) on executable neural architecture code; formally shown to cover the full Σ→Σ composition space.
  • Unit-based code generation: Viterbi-style, unit-by-unit implementation achieving 92% valid code vs. 6% for direct prompting (~86 percentage point gain).
  • Ladder-of-Scales verification: budget-aware multi-scale pretraining strategy enabling cost-effective verification of 1,062 architectures.
  • Largest ASD experiment of its kind: >1B tokens, 2.76M lines of code, 86K agent interactions.

Results

  • Discovered designs outperform/match seed baselines on 7/9 benchmarks at 125M and 6/9 at 350M parameters (vs. GPT2, Mamba2, RWKV7, RetNet, TTT).
  • Top discovered model (VQH) achieves avg. accuracy 59.05–60.61% vs. GPT2's 61.78% and Mamba2's 61.45% at 350M; some designs exceed both on specific tasks (e.g., BLiMP: 94.37 vs. 92.70 for GPT2).
  • Unit-based generation vs. direct prompting: 92% valid code vs. 6%; removing symbolic checker drops validity from 92% to 30%.
  • Full system evolutionary fitness improvement (∆) over 1,000 designs: 7.13% vs. 1.69% for w/o Exp.; Sharpe Ratio 26.5 vs. 3.1.
  • Removing literature access reduces ∆ by 0.73% and SR from 0.69 to 0.567; removing experimental verification raises error rate from 8.61% to 27.31%.

Limitations

  • Cannot integrate hardware-optimized kernels (e.g., FlashAttention) due to complex hardware-specific evaluation requirements.
  • Constrained to ≤350M parameter verification; billion-parameter scale is computationally intractable with current resources.
  • Discovered architecture intelligibility (what the AI-designed blocks actually compute) is not analyzed.
  • Agent learning from feedback is static; no online RL-based adaptation of the discovery policy.
  • Evolutionary fitness shows signs of convergence around 1,000 designs, suggesting diminishing returns in the current search formulation.

Relevance to Agentic AI / LLM Agents

Genesys is a concrete instantiation of a closed-loop, multi-agent autonomous research system where agents handle the full research lifecycle—proposal, peer review, implementation, and empirical verification—without human intervention. The unit-based Viterbi-style code generation strategy (freezing valid sub-components before generating the next) is a broadly applicable design principle for LLM agents operating in structured, validity-constrained output spaces. The adversarial proposer–reviewer dynamic and four-quadrant exploration–exploitation selection strategy are reusable patterns for agentic systems that must manage scarce compute budgets. As an ASD benchmark, LMADE offers a rigorous testbed for evaluating autonomous agent capability on tasks with clear, executable success criteria—directly relevant to the evaluation gap in agentic AI research.