Skip to content

Crucible: Quantifying the Potential of Control Algorithms through LLM Agents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Crucible is an LLM-agent framework that quantifies the "Tuning Potential" of control algorithms—how much performance a domain expert can extract through targeted tuning—using multi-level expert simulation and a formalized similarity-weighted metric. It demonstrates that logic-level LLM modifications expand the optimization space beyond what Bayesian hyperparameter search alone can reach. Applied to ABR streaming and distributed scheduling, Crucible-tuned heuristics surpass RL baselines in real deployments.

Problem

Control algorithm evaluation focuses on default/ideal performance, ignoring Tuning Potential: how much a deployed algorithm can improve when a domain expert tailors it to a specific environment. No systematic, quantifiable framework exists to measure this, making it impossible to compare algorithms on practical adaptability or to treat tunability as an explicit design objective.

Method

Crucible combines two components:

  1. LLM-driven multi-level expert simulation agent: A Claude-3.7-Sonnet-backed agent receives domain knowledge (task description, objectives, environment overview) via system prompt. It reads algorithm source code and execution logs (state–action–result triplets), proposes logic-level code modifications, and maintains a history of (rationale, action, result) triplets for iterative reflection. Bayesian optimization is wrapped as a callable tool for parameter-space exploitation. Developer capability levels are simulated by varying the number of Bayesian iterations (0/10/20) and LLM reflection steps (1/2/3)—not by changing prompts.

  2. Formalized Tuning Potential metric: A set of probe algorithms runs across a test environment suite to produce normalized performance characteristic vectors. Environment similarity is RMSE-based. Potential P is the similarity-weighted average performance gain after Crucible tuning relative to the algorithm's ideal environment, aggregated across all test environments (Eq. 5).

Key Contributions

  • Formalizes Tuning Potential as a distinct, measurable algorithm property distinct from default performance or parameter sensitivity.
  • Proposes Crucible, the first system to quantify Tuning Potential via an LLM agent that performs both logic-level code edits and tool-assisted parameter search.
  • Introduces an environment-similarity metric derived from probe-algorithm performance profiles, enabling cross-domain potential comparisons.
  • Identifies representational capacity (breadth of control state) and comprehensibility (structural transparency) as the two primary determinants of Tuning Potential.
  • Shows potential analysis can guide algorithm redesign (BBA→BBA_C: +4% post-tuning QoE) and validates findings in real-world Dash.js deployment.

Results

  • Cart-Pole: Bang-bang controller goes from score 34 to 500 (matches DQN ceiling) after a single LLM logic modification; PID goes from 34 to 500 after one Bayesian + one LLM step.
  • ABR (simulated): Crucible achieves up to 44.1% improvement over Bayesian-only baseline on Puffer dataset; with 3 LLM iterations + 20 Bayesian iterations, average improvement over initial reaches 67.85%.
  • ABR (real-world Dash.js): Tuned HYB and BBA each reach QoE 1.72, surpassing RL-based Pensieve (1.66); MPC improves from 1.72→1.79.
  • Scheduling (Spark/TPC-H): FIFO tuned by Crucible achieves optimal cumulative waiting time; simple SJF after Crucible outperforms the more complex Multi-level Feedback algorithm.
  • Multi-LLM robustness: GPT-4o-mini and Claude 3.5 Sonnet produce consistent relative rankings; Claude 3.7 Sonnet achieves highest HYB score (1.12 vs. 1.03/1.04 for others).
  • Pure LLM-only runs (no Bayesian): ~60% of scenarios show no gain; adding Bayesian reduces ineffective scenarios to ~20%.

Limitations

  • LLM variability: different model versions produce different absolute scores, creating reproducibility concerns for the potential metric itself.
  • Cannot modify black-box algorithms (e.g., RL-trained Pensieve, neural nets) directly; decision-tree surrogates are used as a workaround, leaving the black-box case open.
  • High-dimensional input state spaces (DAG scheduling) substantially reduce LLM comprehension and optimization effectiveness compared to simpler state spaces (ABR buffer+bandwidth).
  • Potential metric depends on choice of probe algorithms and test environment suite, which could bias comparisons if not carefully selected.

Relevance to Agentic AI / LLM Agents

Crucible is a direct instantiation of an LLM agent acting as a software-engineering expert: it reads code, reasons about logic, proposes targeted edits, calls optimization tools, and iterates on feedback—a closed-loop agentic loop applied to systems programming rather than open-ended tasks. The paper's differential capability simulation (varying tool-call budgets and reflection steps rather than prompts) is a novel design pattern for modeling agent expertise levels. The synergy between LLM logic-exploration and classical optimization tool-exploitation (Bayesian search) illustrates a reusable hybrid agentic architecture. For researchers tracking LLM agents, this work provides concrete evidence that agents can outperform domain-specific automated search on code-modification tasks, and surfaces comprehensibility of the target artifact as a key bottleneck for agent effectiveness.