GitLake: Git-for-data for the agentic lakehouse

🕒 Published (v1): 2026-07-09 10:03 UTC · Source: Arxiv · link

Why this paper was selected

GitLake: Git-for-data agentic lakehouse; branching/versioning primitives for agent workflows

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GitLake ports Git primitives (commits, branches, merges) to the OLAP lakehouse layer, letting AI agents operate on isolated data branches while humans gate publication to production via merge review. It lifts single-table Apache Iceberg snapshots into lakehouse-wide transactional units, making multi-table pipeline failures atomic and reversible. The system is in production at Bauplan, handling hundreds of thousands of branches per week.

Problem

Traditional OLAP/lakehouse systems lack affordances for safe agentic data workflows: a failing multi-table pipeline can leave the lakehouse in a globally inconsistent state (some tables updated, others not), there is no built-in rollback or point-in-time audit, and no principled human review boundary before agent-generated changes reach production. Existing primitives such as Snowflake zero-copy clone and Databricks shallow copy are ~100× slower than what agentic workloads require.

Method

GitLake lifts single-table Iceberg snapshot mechanics to lakehouse-wide Git abstractions:

1. Commits: each table write atomically updates an Iceberg snapshot and records a catalog-level commit that maps every table to its current snapshot, with a hash, parent pointer, and metadata — enabling time-travel via ref=<hash>.
2. Branches: a branch is a movable pointer to a HEAD commit. Creating a branch is a metadata-only no-op (\(p_{95} \approx 80\text{ ms}\)). Writes to feature diverge from main without touching production.
3. Merges: merging is a control-plane-only catalog update — no Parquet files are moved. The merge into main acts as the human review gate for agent output.
4. Transactional runs: the run() API auto-opens an ephemeral branch, materializes pipeline writes there, then atomically merges on success or abandons the branch on failure, preventing partial-write inconsistency (MVCC-style isolation). The aborted branch remains reachable for debugging.

Copy-on-write storage ensures branched appends add new Parquet files only for new rows. The CLI is implemented in Rust with Python bindings; both surfaces share identical types and error taxonomies. Core abstractions were stress-tested with a lightweight Alloy formal model to surface counterexamples.

Key Contributions

• Git-for-data abstraction layer (commits, branches, merges, reverts) built on top of Apache Iceberg single-table ACID snapshots, generalized to lakehouse-wide consistency.
• Transactional pipeline execution via ephemeral branches coupled to run(), providing all-or-nothing publication across multi-table DAGs.
• A natural human-in-the-loop review boundary: agents iterate freely on branches; production changes require an inspectable, approved merge.
• Production validation at scale: millions of jobs, hundreds of thousands of branches per week; branch creation \(p_{95} \approx 80\text{ ms}\), up to 25× faster catalog CRUD than Nessie, and ~100× faster branching than Snowflake/Databricks equivalents.
• Open-source Alloy formal model revealing correctness tensions (e.g., nested-branch consistency counterexample) published for community use.

Results

• Branch creation \(p_{95} \approx 80\text{ ms}\) at hundreds of thousands of new branches per week in production.
• ~100× faster branch creation than Snowflake zero-copy clone and Databricks shallow copy (per OlapBranchBench).
• Up to 25× faster standard CRUD-like catalog requests versus Nessie.
• Merge conflict rate: ~10 conflicts per 100k merge attempts in production traces.
• Millions of jobs run across hundreds of thousands of data branches total (Bauplan production).

Limitations

• Alloy modeling identified an unresolved correctness counterexample: after a failed run, another agent can branch off the aborted commit and merge back to main, producing an inconsistent state (Figure 4); resolution is left to future work.
• Human review becomes the bottleneck as agent-generated exploration outpaces verification throughput; cross-branch querying is proposed but not yet implemented.
• LLMs still struggle to fully internalize lakehouse nuances (e.g., optimal use of nested branches for durable execution), requiring purpose-built skill augmentation.
• No support for multi-language pipeline transactions in industry lakehouses (Snowflake, Databricks) without GitLake; interoperability with those platforms is not addressed.
• The paper is a workshop paper with preliminary Alloy results; formal correctness proofs are incomplete.

Relevance to Agentic AI / LLM Agents

GitLake directly addresses the infrastructure gap that makes AI agents unsafe in data environments — the absence of isolation, rollback, and human oversight primitives — by providing a Git-like harness native to the lakehouse. The branch-as-sandbox pattern is architecturally analogous to how coding agents use Git worktrees: agents can explore freely without corrupting shared state, and merges encode the human-in-the-loop approval step. The production evidence (millions of agentic jobs) makes this one of the first large-scale demonstrations that Git-for-data abstractions are viable at agentic concurrency and throughput. For researchers tracking agentic AI, this work provides a concrete template for building safe, auditable, reversible agent action spaces over structured data, complementing analogous work on code agents and tool-use safety.