Skip to content

Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization

🕒 Published (v1): 2026-06-16 13:34 UTC · Source: Arxiv · link

Why this paper was selected

LLM-orchestrated meta-harness spanning ingest→MLOps→drift; self-composable pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a trustworthy, self-composable Big-Data-as-a-Service (BDaaS) framework in which a central LLM orchestration layer dynamically composes and coordinates eight specialized agents covering the full ML lifecycle—from data ingestion through post-deployment drift recovery. The key claim is that multi-agent decomposition with shared artifact governance outperforms single-agent LLM, AutoML-only, and manual ML baselines on both predictive quality and lifecycle-level reliability metrics. Evaluation is prototype-scale on controlled synthetic tabular benchmarks, not real production data.

Problem

Existing LLM-based data science agents and AutoML systems address isolated pipeline stages (e.g., cleaning, feature engineering, or model selection), but lack lifecycle-level orchestration that spans ingestion through post-deployment drift recovery. Production BDaaS environments additionally require artifact governance, reproducibility, human oversight, and adaptive feedback loops that current single-stage or single-agent approaches do not provide.

Method

The framework is structured as four stacked layers:

  1. User Interaction Layer — accepts high-level dataset uploads, task definitions, constraints, and deployment goals, converting them to structured task specifications.
  2. LLM Orchestration Layer — a central orchestrator performs task decomposition, agent selection, execution ordering, output validation, and dynamic workflow recomposition based on dataset characteristics and operational constraints.
  3. Multi-Agent Execution Layer — eight specialized agents: Data Ingestion, Data Cleaning, Feature Engineering, AutoML Training, Model Evaluation, MLOps Deployment, Monitoring & Drift Detection, and Human Oversight. Each returns structured outputs to the orchestrator for validation before the next stage proceeds.
  4. Shared Artifact Governance Layer — a unified repository storing raw/cleaned data, schemas, feature configs, trained models, evaluation reports, deployment artifacts, and version histories to support reproducibility, traceability, and rollback.

Drift-aware feedback: the Monitoring & Drift Detection Agent computes statistical comparisons of incoming feature distributions against training baselines; on detection, the orchestrator triggers retraining, threshold recalibration, redeployment, or rollback. Human-in-the-loop checkpoints gate critical operations (cleaning decisions, model approval, deployment release, drift response).

Key Contributions

  • Unified LLM-orchestrated multi-agent architecture for self-composable, end-to-end BDaaS lifecycle automation.
  • Functional decomposition of eight specialized agents spanning data engineering, AutoML, MLOps deployment, and post-deployment monitoring.
  • Trustworthiness mechanisms: validation checkpoints at each stage boundary, human-in-the-loop control, shared artifact governance, reproducibility metadata, and drift-aware feedback loops.
  • Prototype evaluation protocol assessing not only predictive performance but also workflow completion, artifact traceability, deployment readiness, and drift recovery—metrics absent from standard AutoML benchmarks.

Results

All numbers are from a local prototype on controlled synthetic tabular benchmarks (not real production data).

Predictive performance (classification, avg. F1): - Proposed multi-agent BDaaS: 0.662 - Single-agent LLM: 0.652 - AutoML-only: 0.644 - Manual ML: 0.563

Regression (Cloud Workload Demand, RMSE): - Proposed: 2.809 vs. AutoML-only: 3.279 (~14.3% improvement); \(R^2 = 0.694\) vs. 0.583

Workflow automation / trustworthiness (Table III): - Proposed: 100% lifecycle completion, 0 failed stages, 1 human intervention, 100% artifact traceability, 100% deployment readiness, 100% reproducibility - Single-agent LLM: 77.8% completion, 2 failed stages, 78% traceability, 67% deployment readiness - AutoML-only: 66.7% completion, 44% deployment readiness - Manual ML: 55.6% completion, 33% deployment readiness

Drift recovery (Sensor Fault stream, Table IV): - Proposed: 0 detection delay, 0 false alarms, recovery within 1 monitoring window (F1 restored from 0.495 to 0.667) - Single-agent LLM: 0 delay, recovery in 2 windows - AutoML-only: 3-window delay, recovery in 3 windows - Manual ML: 4-window delay, recovery in 3 windows

Limitations

  • Entire evaluation is conducted on a controlled local prototype with synthetically generated tabular benchmarks; no real enterprise or cloud production datasets were used.
  • Dataset scale is small (1,800–2,600 samples, 12–16 features), which limits generalizability to large-scale BDaaS workloads.
  • All baselines use comparable lightweight models, making predictive gains moderate and potentially attributable to preprocessing differences rather than orchestration architecture.
  • Computational cost, end-to-end latency, LLM API cost, and human review quality under realistic operational constraints are not measured.
  • The 100% traceability/completion scores are trivially achievable on a controlled prototype; real system faults (network, schema drift at ingestion, LLM API failures) are not stress-tested.
  • No ablation study isolates the contribution of individual agents or governance components.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of the LLM-as-orchestrator / meta-harness pattern: the LLM orchestration layer functions as a meta-harness that dynamically selects, sequences, and validates a pool of specialized sub-agents rather than executing any domain logic itself. The shared artifact governance layer is architecturally analogous to a harness context store—it maintains cross-agent state, enables rollback, and enforces reproducibility invariants that individual agents cannot provide in isolation. The drift-aware feedback loop demonstrates how a meta-harness can close the control loop post-deployment, extending the harness metaphor from pipeline construction to lifecycle management. For researchers building harness frameworks, this paper provides a concrete design for human-in-the-loop checkpoints, validation gates between agent handoffs, and structured output contracts that the orchestrator enforces.