Skip to content

KPI2KVI: A Multi Agent Workflow for Calculating Key Value Indicators from Service Descriptions

🕒 Published (v1): 2026-03-31 09:10 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KPI2KVI is a nine-stage, deterministic multi-agent workflow that transforms a natural-language service description into bounded, traceable Key Value Indicator (KVI) estimates by orchestrating LLM-based agents that elicit context, select KVI categories from a taxonomy, generate service-specific KPIs, collect values, and compute interval-valued KVI outputs \(\{min, exact, max\}\) with explicit rationales. It targets the 6G/telecoms domain where stakeholder value assessment is currently manual and inconsistent. Simulations show it outperforms monolithic LLM and SLM baselines in stability, success rate, and taxonomy-grounded category accuracy.

Problem

Existing KVI assessment approaches either (a) provide governance/concept frameworks without specifying end-to-end computation pipelines, or (b) embed KVI calculations in optimization but assume a small, pre-fixed KVI set with non-trivial inputs already available. Neither handles: dynamic KVI scope selection for new heterogeneous services, missing/uncertain KPI evidence, end-to-end traceability from service description to computed indicators, or user-facing rationales linking results to underlying evidence.

Method

KPI2KVI implements a deterministic nine-stage pipeline alternating conversational and structured steps:

  1. Inspector — guided interview eliciting service intent, stakeholders, and value impacts.
  2. KVI Category Extractor — maps the interview transcript against a controlled KVI taxonomy to produce ranked candidate categories.
  3. KVI Category Evaluator — conversational human-in-the-loop loop to add/remove/replace categories.
  4. KVI Category Finalizer — consolidates a binding finalized category contract (fixes scope before evidence collection).
  5. KPI Generator — proposes a compact, service-specific KPI evidence plan (name, unit, description per KPI).
  6. KPI Collector — interactive dialogue collecting KPI values; missing values are delegated to system estimation and flagged as assumptions.
  7. KPI Structurer — converts free-form collection into a machine-readable table with provenance flags.
  8. KVI Calculator — per-KVI computation loop producing \(\{exact, min, max, rationale\}\) using a step-by-step CoT reasoning chain against the structured KPI table.
  9. KVI Advisor — post-hoc advisory queries answering user questions with citations to stored artifacts.

The orchestrator maintains a shared key–value memory of structured artifacts (interview transcript, finalized category list, KPI table, per-KVI results) passed selectively into each agent's prompt. The architecture is a FastAPI SSE backend with a React frontend; agents are auto-discovered from a plugin-style registry. LLM calls route through OpenRouter/PydanticAI.

Key Contributions

  • A fully specified, end-to-end KVI computation pipeline covering scope selection, evidence collection, interval computation, and post-hoc advising — the first of its kind in the literature.
  • A deterministic staged state machine with explicit structured artifact contracts between stages, enabling auditability and cross-agent context without prompt bloat.
  • Support for uncertain/missing evidence via delegated estimation with provenance flags, producing interval-valued \(\{min, exact, max\}\) KVI outputs rather than point estimates.
  • Comparative simulation against four variants (monolithic LLM, agentic SLM ± taxonomy ± CoT) across three axes: calculation complexity, KVI scope size, and taxonomy quality \(q \in [0,1]\).

Results

  • KVI instability vs. complexity (Fig. 3A): KPI2KVI (full) maintains the lowest run-to-run variance as formula depth and KPIs-per-KVI increase; SLM baselines without CoT degrade significantly; the monolithic LLM (DeepSeek-R1) is more stable than SLM variants but less stable than the staged pipeline.
  • End-to-end success rate vs. KVI scope (Fig. 3B): KPI2KVI stays robust longest as the number of requested KVI categories grows; monolithic LLM degrades more gracefully than SLM baselines but fails earlier than KPI2KVI.
  • Category selection F1 vs. taxonomy quality \(q\) (Fig. 3C): KPI2KVI achieves the best F1 across all \(q\) values by combining taxonomy grounding with the inspector/evaluator refinement loop; the agentic SLM with taxonomy benefits from high \(q\) but is misled at low \(q\); no-taxonomy SLM is insensitive to \(q\).
  • Each experimental condition averaged over 10 runs with different seeds and prompt paraphrases; shaded regions show variance.

Limitations

  • Simulations use synthetic/representative service cases; no real-world deployment or user study is reported.
  • The KVI taxonomy is fixed and inherited from a prior work [23]; KPI2KVI cannot yet extend or update it autonomously.
  • In-memory session state with TTL is not durable across long-horizon sessions; artifacts are lost when a session expires.
  • RAG integration for reuse of past artifacts across sessions/services is listed as future work only.
  • Fine-tuning SLMs to remove dependency on explicit taxonomy and CoT prompting is deferred to future work.
  • No quantitative ablation of the human-in-the-loop refinement loop's contribution is provided separately from the full system.

Relevance to Harnesses / Meta-Harnesses

KPI2KVI is a textbook domain-specific multi-agent harness: a deterministic orchestrator routes turns through a fixed nine-stage pipeline, manages a shared structured artifact store, and chains specialized agents whose outputs feed downstream agents as typed contracts — the same architectural pattern seen in general-purpose meta-harnesses. The explicit staged state machine with artifact contracts and the plugin-style agent registry are transferable harness design patterns: they enforce sequencing, prevent context bleed between agents, and make intermediate state auditable. The comparative evaluation against a monolithic LLM baseline provides direct empirical evidence that staged orchestration with shared memory outperforms single-prompt approaches in stability and coverage — a core claim relevant to meta-harness design. Future-work integration of RAG for cross-session artifact reuse points toward persistent-memory harness architectures.