Skip to content

Trustworthy AI in the Agentic Lakehouse: from Concurrency to Governance

🕒 Published (v1): 2025-11-20 14:21 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This position paper argues that trustworthy agentic access to enterprise lakehouses requires solving infrastructure concurrency first, and that governance follows naturally. The authors draw an analogy to Multi-Version Concurrency Control (MVCC) in databases, explain why a direct transplant fails in distributed, heterogeneous lakehouses, and propose Bauplan — an agent-first lakehouse with Git-like data branching, containerized FaaS compute isolation, and a single unified run API.

Problem

Traditional lakehouses are designed for small human teams: they lack multi-table transactional guarantees across heterogeneous runtimes (SQL + Python), expose wide API surfaces that are hard to govern, and have no principled mechanism to prevent an agent from leaving the lake in an inconsistent state (e.g., a partial pipeline write that corrupts downstream JOINs). MVCC from monolithic databases (e.g., Postgres) cannot be directly transplanted because lakehouses decouple storage from compute and support heterogeneous runtimes, breaking the uniform-runtime and co-located-control assumptions that make MVCC work.

Method

Bauplan re-implements MVCC primitives natively for the lakehouse across three axes:

  1. Data isolation via Git-like branching: Every write is an immutable snapshot; a pipeline run automatically opens a temporary branch, executes all DAG nodes, and atomically merges to main on success (or leaves main untouched on failure). This extends transactional semantics across multi-table, multi-commit pipelines rather than per-table.

  2. Compute isolation via FaaS: Each DAG node runs in a containerized, network-isolated Function-as-a-Service (FaaS) environment with a declaratively pinned Python version and pip dependencies (e.g., @bauplan.python('3.11', pip={'polars': '0.88'})). The platform owns the runtime; the agent supplies only code.

  3. Declarative I/O abstraction: Functions accept and return typed bauplan.Model table references — not raw S3 paths — so the platform mediates all I/O, enabling transactional coupling of data and compute and reducing the governable API surface to a single bauplan.run(pipeline) call.

Governance is derived as a corollary: the narrow, declarative API surface enables RBAC on data operations, decorator-based package whitelisting for compute governance, and branch-then-merge as a human-in-the-loop review gate.

A reference self-healing pipeline implementation uses a ReAct loop that operates entirely on a data branch; a platform-registered verifier checks acceptance criteria before an engineer approves the final merge to main.

Key Contributions

  • Formal identification of why MVCC cannot be directly transplanted to lakehouses (storage-compute decoupling, heterogeneous runtimes, multi-table pipeline writes)
  • Agent-first lakehouse design (Bauplan) with three unified primitives: immutable branching, FaaS compute isolation, and declarative I/O
  • Proof that solving concurrency/correctness yields governance as a structural consequence, not a bolt-on
  • Open-source reference implementation of a self-healing agentic pipeline with human-in-the-loop branch-merge review

Results

This is a position paper with no empirical benchmarks. The only concrete evidence cited is: - Bauplan's copy-on-write branching is described as efficient "even on hundreds of tables and billions of rows" (no numbers provided) - The self-healing pipeline implementation is open-sourced at github.com/BauplanLabs/the-agentic-lakehouse as a working reference

Limitations

  • No quantitative evaluation: no latency, throughput, or scalability benchmarks comparing Bauplan to Airflow, Databricks, or other lakehouses
  • Position paper framing: claims about correctness and governance are argued by analogy and design, not empirically validated
  • FaaS model introduces its own overhead (container cold starts, serialization) that is acknowledged but not measured
  • The merge-conflict resolution strategy for concurrent agent writes to the same table is not detailed
  • Assumes agents interact only through the Bauplan API; governance argument breaks if agents can bypass the API surface

Relevance to Harnesses / Meta-Harnesses

Bauplan is essentially a meta-harness for agentic data pipelines: it provides the execution substrate — branching, sandboxing, atomic commit/rollback, and a single run API — that a higher-level agent orchestrator (e.g., a ReAct loop) sits on top of. The self-healing pipeline example directly instantiates the harness pattern: an outer reasoning loop drives tool calls, the harness enforces correctness invariants, and a verifier step acts as an automated acceptance gate before human review. The branch-then-merge flow is structurally identical to how code review harnesses (CI/CD) gate software deployments, applied here to data assets — a design principle directly transferable to any meta-harness that manages agentic writes to shared state.