Optimas: An Intelligent Analytics-Informed Generative AI Framework for Performance Optimization¶

🕒 Published (v1): 2026-04-26 21:34 UTC · Source: Arxiv · link

Why this paper was selected

Performance-analytics-informed LLM framework for automated code optimization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Optimas is a fully automated, modular multi-agent framework that mines multimodal GPU performance diagnostics (Roofline, PC sampling, hardware counters) to construct evidence-grounded prompts for LLM-driven CUDA code optimization. Across 3,410 experiments on 10 benchmarks, it achieves 100% syntactically correct output and ≥8% average speedup, peaking at 79.09% on NVIDIA H100 GPUs. It introduces Evidence-Aligned Reasoning (EAR) metrics to verify that generated edits actually follow the provided diagnostic evidence rather than hallucinating generic changes.

Problem¶

LLMs applied to HPC code optimization produce syntactically valid but performance-neutral or regressive transformations when given code without runtime context—a "code-only" baseline yields 0% valid optimizations. Existing profiling tools identify bottlenecks but produce terabytes of raw telemetry unsuitable for direct LLM ingestion, and no systematic method existed to verify whether LLM edits are causally grounded in the profiler signals rather than coincidentally correct.

Method¶

Optimas pipelines four specialized agents:

Profiling Agent: invokes vendor tools (Nsight Compute) to collect raw JSON/CSV telemetry if not pre-existing.
Analysis Agent: condenses raw profiles via three saliency passes:
Roofline: classifies kernels as compute- or memory-bound using utilization ratios \(\rho \in [0,1]\) against a saturation threshold \(\tau_{sat} = 0.70\).
PC Sampling: aggregates stall events per source line, retaining only stalls contributing \(\geq\tau=30\%\) of stalled cycles; reduces a 990 MB trace to a <6 KB summary (99.9% reduction).
Hardware Counter Selection: solves a sparse selection problem \(\min \|t - Da\|_2^2\) s.t. \(\|a\|_0 \leq \kappa\) via randomized Ensemble OMP (Algorithm 1), selecting \(\kappa=5\) counters most correlated with runtime.
Prompt Construction Agent: combines annotated kernel source, diagnostic summaries, and explicit guardrails (no signature changes; must cite the diagnostic evidence motivating each edit) into a structured prompt; supports GPT-5, Gemini-2.5-pro, and Llama 3.1 via config.yml.
Evaluation Agent: compiles LLM output, runs each variant five times, validates bit-level correctness against a reference, and retries up to three times on compilation failure using error messages as feedback.

EAR metrics quantify: Evidence Coverage (did edits cite diagnostic items?), Localization Agreement (do edits land near high-stall lines?), and Directional Consistency (do post-optimization counter values move in the expected direction?).

Key Contributions¶

Multimodal data mining pipeline: distills multi-source HPC logs (Roofline + PC sampling + hardware counters) into token-budget-compliant prompts.
Evidence-Aligned Reasoning (EAR) metrics: five model-agnostic metrics verifying causal alignment between diagnostics and LLM-generated edits.
Large-scale evaluation: 3,410 controlled experiments across 10 benchmarks and 3 LLM backends on NVIDIA H100.
Backend-agnostic closed-loop framework: configurable via single YAML; modular enough to add new profilers or LLM backends without changing other components.
Optimization corpus: 573 records with aligned diagnostics, transformations, and measured outcomes for future retrieval-augmented optimization research.

Results¶

Correctness: 100% syntactically and semantically correct code across all 3,410 experiments.
Success rate: ≥98.82% of experiments show measurable performance improvement.
Speedup range: 8.02%–79.09% average gains depending on benchmark and diagnostic combination.
Best single result: Accuracy kernel with GPT-5 using PC+Roofline: 79.09% speedup.
Code-only baseline: 0% valid optimizations (no diagnostic context), confirming diagnostics are necessary.
Ablation: no single diagnostic source dominates; PC+Roofline or PC+IA combinations consistently outperform single-source prompts; Gemini-2.5-pro and LLaMA show more variance (e.g., Accuracy kernel degrades to ~0.25% with Gemini-2.5-pro).
SW4Lite (real DOE proxy app): multi-kernel optimization of top-5 kernels covering 80% of runtime achieved successfully.

Limitations¶

Evaluated exclusively on NVIDIA GPUs (H100); generalizability to AMD/Intel GPU architectures is unverified.
Hand-tuned expert baselines excluded from comparison due to unavailability; peak achievable speedup relative to human expert tuning is unknown.
Temperature tuning is manual (\(T=0.15\) fixed empirically); no automated diversity-adaptation in the current system.
eOMP counter selection is underdetermined (\(C \gg N\)); sparse selection of \(\kappa=5\) counters may miss multi-counter interactions.
Iterative retry limited to 3 compilation attempts; persistent failures require manual intervention via dashboard.
Dataset covers only CUDA kernels; CPU-bound or heterogeneous workloads are out of scope.

Relevance to Harnesses / Meta-Harnesses¶

Optimas is a textbook closed-loop meta-harness: it wraps profiling, analysis, LLM invocation, compilation, and validation into a single automated pipeline where each stage feeds structured context into the next. The pattern—extract salient signals from a noisy environment, compress them into LLM-consumable form, generate outputs, and verify them in a feedback loop with automated retry—is directly transferable to any domain where an LLM agent must act on rich, high-dimensional runtime state. The introduction of EAR metrics is particularly relevant: they provide an intrinsic quality signal for harness operators to detect when LLM outputs have drifted from evidence (hallucination detection within the loop), a challenge common to all production AI harnesses. The config.yml-driven, backend-agnostic, modular architecture also exemplifies the "plugin-able agent pipeline" design pattern increasingly important in meta-harness frameworks.