Bian Que: An Agentic Framework with Flexible Skill Arrangement for Online System Operations¶

🕒 Published (v1): 2026-04-29 15:35 UTC · Source: Arxiv · link

Why this paper was selected

Flexible skill arrangement in agentic framework for large-scale system operations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Bian Que is a deployed LLM-based Operations & Maintenance (O&M) framework for large-scale online engine systems (search, recommendation, advertising) that replaces ad-hoc agent orchestration with a structured Skill Arrangement mechanism—each Skill explicitly declaring the data queries and reasoning prompts needed for a specific business-module context. The key insight is that the O&M bottleneck is not LLM reasoning quality but contextual signal selection; Bian Que addresses this with auto-generated, human-correctable Skills and a dual feedback loop that co-evolves both Skills and a domain knowledge base.

Problem¶

Existing LLM O&M agents treat alert triggering as the sole entry point and feed all available signals (metrics, logs, change events) indiscriminately into the model, causing context dilution and hallucination. Manually curating the mapping from operational event to the relevant subset of data and knowledge is intractable at the scale of dozens of daily releases across hundreds of service modules. There is also no mechanism for the system to learn incrementally from practitioner corrections without requiring code changes.

Method¶

Bian Que formalizes O&M into three canonical patterns ("lines of defense"): release interception, proactive inspection, and alert root cause analysis. Each is handled by a dedicated agent type; agents share one LLM backbone (Qwen3.5-35B-FP8) but differ in scenario prompt $p_{agent}$.

The core abstraction is the Skill, a structured YAML document:

\[\text{Skill} = \langle \textit{LoadDataSchema},\ \textit{Prompt},\ \textit{Meta} \rangle\]

LoadDataSchema is a JSON spec declaring which time-series metrics, logs, change events, and knowledge-base entries to retrieve (mandatory vs. optional). Prompt is a multi-step reasoning template for the retrieved context. An agent execution composes one agent with $\geq 1$ matched Skills via keyword matching on event metadata:

\[o = f_{\text{LLM}}\!\left(\bigcup_i D_{s_i}(e),\ \bigcup_i K_{s_i}(e),\ p_{agent},\ \{p_{s_i}\}\right)\]

Skill Lifecycle: New Skills are LLM-generated from a source descriptor + capability descriptor seed, validated by executing LoadDataSchema live (up to 3 retries). Practitioners correct Skills in natural language; the framework routes corrections to either LoadDataSchema (retrieval gap) or Prompt (reasoning gap).

Unified Self-Evolving Feedback: Every practitioner feedback signal $f$ triggers two parallel pathways: $$f \longrightarrow \begin{cases} \textit{MemoryWrite}(c) \to \textit{KnowledgeDistill}(c) \\ \textit{SkillRefine}(\{s_i\}, f) \end{cases}$$ The knowledge base uses three complementary indices: KV (definitive facts keyed by business+scenario), KKV (relational facts, e.g., module A's impact on module B), and a vector index for fuzzy retrieval. Daily consolidation deduplicates and prunes contradictions.

Key Contributions¶

Unified operational paradigm: Three-pattern abstraction (release interception, proactive inspection, alert RCA) covering the full O&M lifecycle, not just post-alert diagnosis.
Flexible Skill Arrangement: Structured Skill documents that are LLM-generable, LLM-updatable, and human-correctable via natural language—no code changes required for updates.
Unified self-evolving mechanism: Single feedback signal drives co-evolution of domain knowledge base and Skill configurations via two parallel pathways.
Industrial deployment evaluation: Six-month production deployment on KuaiShou's e-commerce search engine serving hundreds of millions of DAU, plus an offline benchmark of 104 SRE-annotated events.

Results¶

Online production (6-month post-deployment vs. pre-deployment baseline): - Fired alerts reduced by 75% (100% → 25% relative) - Non-actionable alert ratio: 80% → 15%; absolute non-actionable volume reduced by ~95% (compound: $0.25 \times 0.15/0.80 \approx 0.047$) - RCA accuracy: 80% (from no automated RCA baseline) - 95% of alerts resolved within 5 minutes - MTTR reduced by >50%

Offline benchmark (104 events, pass@k for Skill generation): - pass@1 = 78.8%, pass@5 = 94.2% (automated generation only) - After human-in-the-loop correction on 6 residual failures: end-to-end pass@3 = 99.0%

Ablation (pass@5 drops vs. full framework): - STATIC (no Flexible Skill): −10.5 pp overall (−13.7 pp for alert scenario) - NO KNOW (no knowledge retrieval): results reported but text truncated before precise numbers

Limitations¶

RCA accuracy ceiling at 80%: error analysis attributes most failures to insufficient domain knowledge for newly deployed services; the self-feedback mechanism is designed to address this over time but has not yet closed the gap.
Co-evolution not quantitatively measured: the compounding effect between knowledge base and Skill quality is only qualitatively observed; controlled measurement requires a longer deployment horizon than the current six months.
Inspection Skills plateau at 98.3%: the residual 1.7% involve higher problem complexity that iterative domain-knowledge refinement alone cannot resolve.
Keyword-based Skill matching: MATCH uses keyword matching against event metadata, which may fail for novel or ambiguous events not covered by existing tag vocabularies.
Scale dependency: the framework reuses existing industrial data infrastructure; portability to organizations without mature observability stacks is untested.
LLM scale requirement: model-agnostic only above ~30B parameters; smaller models are not viable for the operational reasoning tasks.

Relevance to Harnesses / Meta-Harnesses¶

Bian Que is a production meta-harness whose Skill abstraction directly instantiates the harness pattern: each Skill is a parameterized, self-describing execution unit that declares its own data retrieval schema and reasoning prompt, analogous to a harness task template. The auto-generation and iterative refinement of Skills via LLM + human feedback is a concrete mechanism for self-modifying meta-harnesses—the harness generates and updates its own sub-harness configurations rather than requiring static hand-coding. The dual feedback pathway (knowledge distillation + Skill refinement) demonstrates a closed-loop meta-harness architecture where execution traces directly drive harness evolution, a design pattern relevant to any harness system that must adapt to shifting task distributions or new data sources without full redeployment.