Skip to content

EffiSkill: Agent Skill Based Automated Code Efficiency Optimization

🕒 Published (v1): 2026-03-29 20:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EffiSkill is a two-stage framework that mines reusable "agent skills" from large-scale slow/fast code pairs and applies them to optimize unseen programs without any runtime execution feedback during inference. It introduces Operator Skills (concrete transformation mechanisms) and Meta Skills (procedural controllers for skill orchestration), achieving 3.69–12.52 percentage-point improvements over the strongest baseline on EffiBench-X.

Problem

Existing LLM-based code optimization methods (one-shot prompting, retrieval-augmented generation, search-based refinement) operate at the instance level: they either attempt a single rewrite or retrieve similar examples, but never distill recurring optimization patterns into transferable, reusable abstractions. This prevents generalization across tasks that share the same underlying transformation mechanism (e.g., DP state compression, BFS-to-formula replacement).

Method

Stage I — Offline Skill Mining. Given paired slow/fast solutions, an LLM generates structured optimization traces with four fields: ProblemBrief, SlowAudit, FastAudit, DeltaSummary. Each trace is abstracted into a compact signature (optimization type, complexity shift, trigger conditions, bottleneck category). Signatures are embedded in a hybrid TF-IDF + dense (all-MiniLM-L6-v2) space and clustered via KMeans with \(k\) estimated by maximizing cosine-silhouette score over candidates near \(\sqrt{n}\). Each cluster is distilled into an Operator Skill card (applicability signals, transformation steps, complexity effects, pitfalls). Highly similar cards (TF-IDF cosine \(\geq 0.8\)) are merged. A small set of Meta Skills is then induced from the operator library; these encode diagnosis procedures, retrieval heuristics, composition logic, and execution-free candidate assessment.

Stage II — Skill-Guided Optimization. For a new program: (1) Diagnose produces a structured OptimizationBrief; (2) Retrieve pulls 3 candidate Operator Skill sets from the registry; (3) Compose uses Meta Skills to assemble 2–3 optimization plans per skill bundle; (4) Generate produces plan-guided code candidates. No execution signal is used inside the loop; \(k=8\) candidates per task are ranked by runtime on a public 20% test split, with final metrics on the private 80%.

Key Contributions

  • Introduces Operator Skills and Meta Skills as a two-level abstraction for distilling recurring code-optimization patterns from slow/fast corpora.
  • Proposes the EffiSkill two-stage pipeline (offline skill mining + execution-free skill-guided optimization).
  • Delivers a portable, plug-and-play skill library compatible with broader agent workflows.
  • Demonstrates consistent gains on EffiBench-X across Python and C++ with two LLM backbones (GPT-5-mini, Qwen3-Coder-30B-A3B-Instruct).

Results

  • OPT@8 on Python, GPT-5-mini: EffiSkill improves over strongest baseline (SBLLM) by 4.98 pp.
  • OPT@8 on C++, GPT-5-mini: +12.03 pp over strongest baseline.
  • OPT@8 on Python, Qwen3-Coder-30B: +8.67 pp over strongest baseline.
  • OPT@8 on C++, Qwen3-Coder-30B: +12.36 pp over strongest baseline.
  • Overall improvement range over strongest baseline: 3.69–12.52 pp across all model/language settings.
  • Mining corpus: 900 Python + 900 C++ slow/fast pairs; evaluation on EffiBench-X (623 tasks, Python and C++ subsets); strict train/test separation verified.

Limitations

  • Skill mining requires a large corpus of paired slow/fast solutions with measurable efficiency differences (\(r \geq 2\)); such corpora are currently limited to competitive-programming domains.
  • The execution-free setting precludes iterative refinement based on actual runtime signals, which may be a bottleneck when transformation effects are subtle or context-dependent.
  • Meta Skill construction is manual/LLM-induced from the mined operator library rather than learned end-to-end; the quality of Meta Skills depends on the quality and coverage of Operator Skills.
  • Evaluation is limited to Python and C++ subsets of EffiBench-X; generalization to other languages or industrial codebases is not empirically validated.
  • The merge threshold (TF-IDF cosine 0.8) and cluster-count heuristic (\(\sqrt{n}\)) are engineering choices with limited sensitivity analysis reported.

Relevance to Harnesses / Meta-Harnesses

EffiSkill is a concrete instantiation of a meta-harness pattern: it separates an offline knowledge-construction phase (mining a structured registry of skills) from a runtime orchestration phase (retrieve → compose plans → generate), with the Meta Skills acting as a programmatic harness layer that governs how lower-level Operator Skills are selected and sequenced. The two-level skill hierarchy—Operator Skills as domain primitives, Meta Skills as procedural controllers—mirrors the harness/sub-agent decomposition found in agentic meta-harness design. The explicit portability design (plug-and-play skill library reusable across downstream agent workflows) is directly relevant to anyone building meta-harnesses that must coordinate heterogeneous tool-using agents, as it demonstrates how offline-distilled procedural knowledge can be injected into a live agent loop without requiring execution feedback.