Automated Design Optimization via Strategic Search with Large Language Models¶

🕒 Published (v1): 2025-11-27 17:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AUTO is a two-agent LLM framework that casts design optimization as a gradient-free strategic search: a Strategist selects between refine, combine, and innovate strategies, while an Implementor executes and repairs generated designs. Applied to GPU CUDA kernel optimization (chemical kinetics ODE integration and dense matrix multiplication), AUTO produces solutions competitive with expert implementations at 50–70% the search cost of Bayesian optimization.

Problem¶

Traditional gradient-free optimizers (Bayesian optimization, evolutionary algorithms) require explicitly parameterized design spaces and fail on ill-defined problems where valid design transformations are hard to enumerate. LLMs can dynamically interpret such spaces, but prior LLM-based optimization work lacks the systematic exploration/exploitation discipline of classical methods, making search trajectories ad hoc.

Method¶

AUTO runs an $N$-iteration optimization loop with two LLM agents whose context windows are fully reset between iterations to avoid drift:

Context curation: At each iteration $t$, a curated subset $C_t \subseteq D_t$ is constructed from the full design history $D_t$ by mixing $P$ highest-scoring, $Q$ lowest-scoring, and $R$ most-recent designs. This mixture replicates the role of a surrogate model in Bayesian optimization.
Strategist (Base LLM 1): Receives $C_t$ plus problem context, sketches, and hints; selects one of three strategies — refine (improve a near-breakthrough design), combine (merge complementary patterns), or innovate (escape plateaus) — and writes concrete implementation instructions. The first 10 iterations are forced to innovate.
Implementor (Base LLM 2): Generates CUDA code from the Strategist's instructions. Constraint failures (compilation, execution, correctness) trigger up to $K=4$ self-repair attempts within a persistent context window.
Evaluation: Valid designs are profiled with NVIDIA Nsight Systems (3 timed runs per input, averaged) and scored. Detailed per-input metrics are logged for the Strategist's future reference.

The search efficiency metric aligns Strategist decisions with Bayesian acquisition functions: $y_S^{(t)} = \text{exploitation}$ if $\varsigma_t \in \{\text{refine, combine}\}$, else exploration.

Key Contributions¶

A dual-agent harness (Strategist + Implementor) with explicit strategy selection, separating high-level planning from low-level code generation.
A context curation policy mixing best/worst/recent designs to balance surrogate-model richness against context-window constraints.
Quantitative alignment of LLM search behavior with Bayesian optimization acquisition functions, enabling search efficiency measurement.
Demonstration on GPU kernel optimization (Robertson kinetics ODE, dense GEMM) achieving near-expert CUDA code quality.
Cost analysis: ~$159/run vs. ~$480 for a median-wage developer at ~8 hours/run.

Results¶

Chemical kinetics (K1): Best solution (temperature 0.3, Python sketch A, iteration 23) matched and in some regimes exceeded the human baseline; the agent's register-usage strategy led the human implementors to subsequently improve their own code ("Human + K1").
Matrix multiplication (M2/M3): Approaches cuBLAS double-precision GEMM performance via loop tiling; best solution found at iteration 1 (M2) or 68 (M3). Advanced tensor-core paths (WMMA) failed due to API hallucination.
Search efficiency: 50–70% relative to Bayesian optimization across runs.
Compilation success: 56.94% (GEMM), 44.87% (kinetics) — consistent LLM hallucination of CUDA API calls.
t-SNE/consensus K-means over bag-of-words code vectors confirms intra-cluster implementation coherence and non-monotone performance trajectories.

Limitations¶

Low compilation success rates (44–57%) due to LLM hallucination of low-level CUDA/WMMA APIs.
No external documentation provided to agents; adding API docs is future work.
Once an early-iteration optimal design is found (e.g., M2 at iteration 1), AUTO cannot improve beyond it — the framework lacks a mechanism to signal early convergence.
Context window reset after each iteration prevents the Implementor from accumulating cross-iteration implementation knowledge.
Evaluated on only two GPU kernel types (a 3-species ODE and square GEMM); generalization to broader design domains is undemonstrated.
High per-run cost ($159) and wall-clock time (~8 h) limit rapid iteration.

Relevance to Harnesses / Meta-Harnesses¶

AUTO is a textbook meta-harness: it wraps two LLM agents inside a deterministic orchestration loop with explicit state management (design database $D_t$), context curation, constraint enforcement, evaluation, and logging — all components that define a harness architecture. The Strategist/Implementor split directly instantiates the planner-executor pattern common in agentic meta-harnesses, and the $P/Q/R$ context curation policy is a harness-level memory management strategy analogous to a replay buffer. The per-iteration context reset is an explicit harness decision to trade continuity for stability, a design tradeoff researchers building multi-agent harnesses routinely face. The cost/efficiency analysis against Bayesian optimization provides a rare empirical grounding for evaluating harness search strategies, making this directly relevant to anyone designing optimization-flavored meta-harnesses.