Databricks Lakehouse
Platform Architecture

The Legacy Failure Modes

Traditional Hadoop/HDFS-based data lakes suffered from the "swamp" problem: schema drift, failed job debris, and the complete absence of isolation guarantees. Reading a partition mid-write on HDFS returned partial data. Hive's per-partition, client-side locking via ZooKeeper created deadlock vectors at scale and didn't enforce atomicity across multi-partition writes.

Object stores like S3, GCS, and ADLS compounded this: they offer eventual consistency per object (S3 only became strongly consistent in December 2020 per AWS documentation), lack native rename atomicity, and the LIST operation is O(n) with high latency — a fundamental architectural impedance mismatch with the "directory-as-partition" model Hive depended on.

The result: production pipelines at petabyte scale required elaborate workarounds — two-phase writes, external transaction coordinators, and manual repair scripts — all outside the engine, un-auditable, and brittle.

Checkpoint Mechanics

Every 10 commits (configurable via delta.checkpointInterval), Delta materializes the current table state into a Parquet checkpoint file. This collapses all prior add/remove actions into a single file. Reading a checkpoint + subsequent JSON entries is O(1) in the number of historical commits — critical at petabyte scale where transaction logs can have millions of entries. The _last_checkpoint JSON file stores a pointer to the latest checkpoint version number so clients skip full log replay.

Optimistic Concurrency Control (OCC)

Writers read the current version, compute changes, then attempt to atomically write version N+1. The write succeeds only if no conflicting transaction committed between read and write. Conflict detection is operation-aware: two concurrent blind appends to different partitions never conflict; a MERGE and a concurrent schema change will. The WriteSerializable isolation level (default) detects read-write conflicts; Serializable additionally prevents phantom reads — at higher abort rates.

Z-Ordering (Multi-Dimensional Clustering)

Z-ordering interleaves bits of multiple column values using the Z-curve (Morton code) space-filling curve. Running OPTIMIZE ... ZORDER BY (col_a, col_b) physically co-locates rows with similar values across both columns into the same Parquet files. This maximizes the effectiveness of min/max column statistics for multi-column predicate pushdown, reducing files scanned from O(n) to O(log n) for selective point lookups. The compaction rewrites files to the target file size (default 1GB) while sorting by Z-value.

Data Skipping via Column Statistics

Every add action in the Delta Log embeds per-file column statistics: minValues, maxValues, and nullCount for the first 32 columns (configurable via delta.dataSkippingNumIndexedCols). At query time, the Delta reader evaluates the query predicate against the file-level stats — this is pure metadata I/O, no file data is read. Files with non-overlapping ranges are entirely skipped. On a 10TB table with good clustering, this can reduce physical I/O by 99%+ for selective queries.

Bloom Filters (Delta 2.0+)

For high-cardinality string columns where min/max is useless (e.g., UUID lookups), Delta supports Bloom filters at the column chunk level, stored in Parquet page footers. Configured per-column via delta.bloomFilter.enabled and delta.bloomFilter.fpp (false positive probability, default 1%). The Bloom filter acts as a probabilistic skip layer: a definitive not-in-file signal prevents reading data pages entirely. False positives result in unnecessary file reads but never incorrect results.

The JVM Execution Overhead Stack

Apache Spark's core execution model is Tungsten — an off-heap memory manager and whole-stage code generation (WSCG) framework introduced in Spark 2.0. WSCG compiles query plans into JVM bytecode via Janino, eliminating per-row virtual dispatch overhead. However, fundamental JVM constraints remain:

CPU Cache Locality Strategy

Photon's in-memory columnar format aligns each column's data array to 64-byte cache line boundaries (matching x86-64 hardware). When scanning a single column for filtering or aggregation, each cache-line load brings 8 × int64 or 16 × int32 values into L1 cache simultaneously. The access pattern is purely sequential — maximizing hardware prefetcher effectiveness. Compare this to row-format access where a 100-column table requires touching 100 cache lines to retrieve 100 values from the same conceptual column, even though only one column is needed.

JVM Interoperability Layer

Photon is not a separate engine — it runs within the existing Spark executor JVM process as a native shared library (libphoton.so). The Spark driver generates a logical plan (analyzed, optimized via Catalyst) which is then handed to Photon's C++ physical planner via JNI. Photon replaces individual Spark plan nodes (scans, filters, aggregations, joins, exchanges) with C++ equivalents where supported. Unsupported operators fall back transparently to Spark's Java operators — this is the mixed-mode execution model. Data handoff between Photon and Spark code paths uses the shared off-heap Arrow columnar format, avoiding serialization cost.

✅ Photon-Accelerated Operators

• All scan operators: Parquet / Delta scan with predicate pushdown, column pruning, and dictionary encoding decoding
• Hash aggregation (GROUP BY) and sort-based aggregation — SIMD-optimized hash tables
• Broadcast hash join and sort-merge join — columnar build + probe phases
• All arithmetic expressions, comparisons, boolean logic, conditional (CASE WHEN)
• String functions: LIKE, SUBSTR, UPPER/LOWER, REGEXP_LIKE (PCRE via RE2)
• Window functions: RANK, ROW_NUMBER, LEAD/LAG, SUM OVER
• Data type casts, date/timestamp arithmetic

❌ JVM Fallback (Not Photon-Accelerated)

• Python/Scala/Java UDFs — cross-JVM boundary; serialization cost eliminates vectorization benefit
• Python UDAFs (User-Defined Aggregate Functions)
• Pandas UDFs (Arrow-based) — partially optimized but still bridges JVM/Python
• Certain complex LATERAL VIEW explode operations on nested structs/maps
• ML inference operators (mlflow.pyfunc model scoring) — Python GIL-bound
• RDD API operations — Photon is SQL/DataFrame-layer only
• Some geospatial functions, custom encoders

The Pre-Unity Governance Chaos

Before Unity Catalog, Databricks workspaces were governance silos. Each workspace had its own Hive Metastore (HMS) — a MySQL/PostgreSQL-backed database storing table/partition/column metadata. This created:

Identity Federation Model

Unity Catalog federates identities from enterprise IdPs (Okta, Azure AD, Google Workspace) via SCIM 2.0 provisioning. Users and service principals are assigned account-level identities, independent of workspace. Group membership is hierarchical: account_group → workspace_group → user. Privilege evaluation is RBAC + ABAC hybrid: privileges are granted on securable objects (CATALOG, SCHEMA, TABLE, COLUMN, ROW FILTER) and evaluated against the requesting principal's full group membership chain at query time — not cached, always-fresh evaluation.

The Service Principal model uses OAuth 2.0 M2M tokens — a Spark job authenticates as a service principal, Unity Catalog verifies the JWT at the data plane token endpoint, and the executor's S3/ADLS credential is a short-lived Credential Vending token (not a long-lived IAM key) — the principal never holds cloud credentials, only Unity Catalog's token service does.

Fine-Grained Access Control (FGAC)

Unity Catalog enforces FGAC at three granularities:

-- 1. Column Masking: applies a SQL expression to mask PII
CREATE OR REPLACE FUNCTION prod.default.mask_email(email STRING)
RETURN CASE
  WHEN IS_MEMBER('pii_readers') THEN email
  ELSE CONCAT(LEFT(email, 2), '***@***')
END;

ALTER TABLE prod.customers.users
ALTER COLUMN email
SET MASK prod.default.mask_email;

-- 2. Row-Level Security via ROW FILTER function
CREATE OR REPLACE FUNCTION row_filter_by_region(region STRING)
RETURN IS_MEMBER('global_reader') OR region = CURRENT_USER_REGION();

ALTER TABLE prod.sales.orders
SET ROW FILTER row_filter_by_region ON (region);

Unity Catalog captures lineage at three levels from Spark query plans automatically (no annotation required):

1. TABLE Which tables were read/written by which job/notebook, at what time, by which principal. Stored in the lineage graph service (not HMS).
2. COLUMN Which source column contributed to which target column — tracked via Catalyst's ExprId to column mapping. A revenue = price * quantity expression creates lineage edges: price → revenue, quantity → revenue.
3. PROCESS The notebook/job/DLT pipeline and the specific code cell or task that triggered each data movement — actionable for impact analysis and audit.

The Imperative Pipeline Tax

Traditional Spark + Airflow pipelines require engineers to explicitly manage:

• Execution order: DAG definition in Airflow with explicit upstream/downstream dependencies, retry logic, and SLA callbacks
• State management: Structured Streaming checkpoints, watermarks, trigger intervals — all require manual configuration and operational knowledge
• Error recovery: Failed tasks require manual inspection of executor logs, checkpoint repair, and re-run from partial state
• Cluster lifecycle: Cluster provisioning, autoscaling policies, spot instance interruption handling — all outside the pipeline code
• Data quality: Custom assertion logic scattered across PySpark code — not standardized, not observable, no centralized reporting

The DLT Declarative Model

In DLT, you declare what tables should contain, not how to compute them. DLT infers dependencies from the Python/SQL code, builds an execution DAG, manages cluster lifecycle, handles retries, and tracks data quality — all automatically.

import dlt
from pyspark.sql import functions as F

# Bronze: raw ingest — DLT handles streaming state
@dlt.table(
  name="raw_events",
  comment="Raw Kafka ingest — append-only",
  table_properties={"quality": "bronze"}
)
def raw_events():
  return spark.readStream.format("kafka") \
    .option("kafka.bootstrap.servers", ...) \
    .load()

# Silver: transformations + EXPECTATIONS (data quality)
@dlt.expect_or_drop("valid_user",    "user_id IS NOT NULL")
@dlt.expect_or_drop("valid_revenue", "revenue >= 0")
@dlt.expect(       "fresh_event",   "event_ts > current_timestamp() - INTERVAL 7 DAYS")
@dlt.table(name="silver_events")
def silver_events():
  # DLT resolves LIVE.raw_events dependency automatically
  return dlt.read_stream("raw_events") \
    .withColumn("revenue", F.col("payload.revenue").cast("double"))

# Gold: aggregation — automatically batch (no stream needed)
@dlt.table(name="gold_daily_revenue")
def gold_daily_revenue():
  return dlt.read("silver_events") \
    .groupBy("event_date").agg(F.sum("revenue"))

Enhanced Autoscaling (EAS)

Standard Spark autoscaling reacts to executor idle time — it is reactive and has a minimum 60-second scale-down lag. DLT's Enhanced Autoscaling is predictive and cost-aware:

EAS monitors the input backlog of each streaming source (Kafka consumer lag, Auto Loader file queue depth, Delta table new file count). It uses this backlog metric as a leading indicator — scaling UP before the cluster is fully saturated, preventing pipeline lag accumulation. It scales DOWN after observing sustained low backlog, accounting for the cost of adding a node vs. the value of backlog clearance speed.

The autoscaler also respects the cluster_log_conf minimum cluster size policy — it never scales below the defined minimum for SLA guarantees. In practice, EAS reduces cluster idle time by 40–60% vs. standard autoscaling in bursty, event-driven pipelines.

Data Quality: Expectations Framework

DLT Expectations are SQL predicates attached to table definitions with three enforcement modes:

Expectation results are persisted in the pipeline's system.events table — queriable via SQL for SLA dashboards, alerting, and historical quality trend analysis.

The Databricks Terraform Provider (databricks/databricks) covers the full control-plane API surface: clusters, jobs, SQL warehouses, Unity Catalog objects (catalogs, schemas, grants), secret scopes, MLflow experiments, and workspace configurations. This enables GitOps workflows for data platform infrastructure.

# Serverless SQL Warehouse — no cluster management needed
resource "databricks_sql_endpoint" "analytics" {
  name             = "prod-analytics-warehouse"
  cluster_size     = "Medium"     # 8 DBUs/hour
  warehouse_type   = "PRO"
  serverless       = true         # Instant startup, ~3s cold start
  auto_stop_mins   = 10
  enable_photon    = true
  
  channel { name = "CHANNEL_NAME_CURRENT" }
  tags { custom_tags { key = "env" value = "prod" } }
}

# Unity Catalog: fine-grained privilege as code
resource "databricks_grants" "finance_schema" {
  schema = "prod.finance"
  
  grant { principal  = "analysts@corp.com"
          privileges = ["USE_SCHEMA", "SELECT"] }
  grant { principal  = "dbt-service-principal"
          privileges = ["CREATE_TABLE", "MODIFY"] }
}

# DLT Pipeline as code
resource "databricks_pipeline" "ingestion" {
  name        = "prod-customer-cdc-pipeline"
  target      = "prod.silver"
  continuous  = true           # 24/7 streaming mode
  
  cluster { label = "default" autoscale { min_workers = 2 max_workers = 20 } }
  library { notebook { path = "/pipelines/customer_cdc" } }
  
  configuration = {
    "delta.enableDeletionVectors" = "true"
    "pipelines.clusterShutdown.delay" = "60s"
  }
}

CI/CD Integration with Git

Databricks Asset Bundles (DAB) — introduced in DBR 13.0 — define the full deployment spec (jobs, pipelines, notebooks, SQL files) in a databricks.yml YAML file, committed alongside code. This replaces the previous Databricks CLI + JSON workflow with a structured, environment-aware deployment model.

# databricks.yml — Asset Bundle
bundle:
  name: customer-platform

resources:
  jobs:
    gold_aggregation:
      name: gold-daily-aggregation
      job_clusters:
        - job_cluster_key: default
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            node_type_id: "i3.2xlarge"
            autoscale: { min_workers: 4, max_workers: 40 }
            enable_photon: true
      tasks:
        - task_key: transform
          dbt_task:
            project_directory: ./dbt
            commands: ["dbt run --select gold_models"]

targets:
  dev:
    workspace: { host: "https://dev.azuredatabricks.net" }
  prod:
    mode: production
    workspace: { host: "https://prod.azuredatabricks.net" }

Architecture of Serverless SQL

Traditional "Classic" SQL Warehouses required pre-provisioning a cluster (2–10 minute startup). Serverless SQL Warehouses eliminate cluster management — compute is pooled in Databricks' own cloud accounts, and warehouse "instances" are micro-provisioned per-query in ~3 seconds. The key architectural components:

• Compute Pool: Pre-warmed Photon executors in Databricks-managed VMs. The workspace connects to this pool via a control plane API — no cluster in customer's VPC
• Query Router: Incoming JDBC/ODBC/REST queries are routed to an available compute slot with query-level isolation guarantees
• Result Cache: Query results are cached at the warehouse level in object storage — identical query reruns return cached results in <100ms
• Predictive I/O: Databricks caches frequently-accessed Delta files in a local SSD cache tier (Delta Cache) on executor nodes — transparent to the user

BI Integration via JDBC/ODBC + Arrow

Databricks exposes a JDBC/ODBC endpoint compatible with all major BI tools. The Databricks SQL Connector uses Arrow Flight SQL for result transmission — replacing JDBC's row-serialized protocol with columnar Arrow batches. For a 1M-row result set, Arrow Flight transfers ~5× faster than JDBC.

Protocol Architecture

The sharing server (Databricks-managed or self-hosted open-source) exposes a REST API. When a recipient requests a share, the server:

1. Validates the recipient's bearer token (generated per-recipient, scoped to specific shares)
2. Evaluates Unity Catalog row/column filters against the recipient's identity
3. Queries the Delta Log to identify current data files for the requested snapshot version
4. Generates pre-signed S3/ADLS URLs (15-minute TTL) for each Parquet data file
5. Returns a response containing: JSON metadata (schema, partition info) + list of pre-signed URLs

The recipient's engine (Spark, Pandas, Power BI, DuckDB) downloads the Parquet files directly from object storage using the pre-signed URLs — zero data copy through Databricks infrastructure. Bandwidth charges are borne by the object storage provider (the provider's account), not piped through Databricks control plane.