Everything About
Databricks

Isolate the data plane in a private VNet/VPC. Use VPC peering or Private Link to connect to source systems.

• Enable VPC/VNet injection (no public IPs on workers)
• Configure NAT Gateway for outbound traffic only
• Use Private Link for Databricks control plane communication
• Place clusters in same AZ as primary object storage bucket
• Dedicated subnets per workspace (at least /24 for PB workloads)

# Terraform: Databricks on Azure (private)
resource "azurerm_databricks_workspace" "prod" {
  name                        = "dbx-prod"
  resource_group_name         = var.rg
  location                    = "eastus2"
  sku                         = "premium"
  public_network_access_enabled = false
  custom_parameters {
    virtual_network_id        = var.vnet_id
    public_subnet_name        = "dbx-public"
    private_subnet_name       = "dbx-private"
    no_public_ip              = true
  }
}

Organise your Lakehouse storage using Medallion pattern with separate containers per zone.

• Separate storage accounts per environment (dev/staging/prod)
• Enable hierarchical namespace (ADLS Gen2 / S3 with table format)
• Set lifecycle policies: archive Bronze after 90 days to cold tier
• Enable versioning on Gold layer containers
• Use customer-managed keys (CMK) for encryption at rest

# Recommended folder structure
abfss://bronze@prod.dfs.core.windows.net/
  ├── source_system_a/
  ├── source_system_b/
abfss://silver@prod.dfs.core.windows.net/
  ├── domain_users/
  ├── domain_orders/
abfss://gold@prod.dfs.core.windows.net/
  ├── mart_finance/
  ├── mart_operations/

For PB workloads, use instance fleets with autoscaling and spot/preemptible instances on workers.

# Recommended cluster config (JSON)
{
  "cluster_name": "prod-etl-large",
  "spark_version": "14.3.x-scala2.12",
  "node_type_id": "Standard_E64ds_v4",
  "driver_node_type_id": "Standard_E32ds_v4",
  "autoscale": {
    "min_workers": 4,
    "max_workers": 200
  },
  "enable_elastic_disk": true,
  "spot_bid_max_price": 100,
  "spark_conf": {
    "spark.databricks.delta.optimizeWrite.enabled": "true",
    "spark.databricks.delta.autoCompact.enabled": "true",
    "spark.sql.shuffle.partitions": "auto",
    "spark.databricks.photon.enabled": "true"
  }
}

Unity Catalog is mandatory for production. It provides fine-grained access control, lineage, and auditing.

• Create one metastore per cloud region
• Use 3-level namespace: catalog.schema.table
• Apply row-level filters and column masking for PII
• Enable audit logging to SIEM (Splunk, Sentinel)
• Use Service Principals for all automated jobs (no personal tokens)

-- Unity Catalog: Fine-grained permissions
GRANT SELECT ON TABLE prod_catalog.gold.orders
  TO `data-analysts@company.com`;

CREATE ROW FILTER orders_region_filter
  ON TABLE prod_catalog.gold.orders
  USING (region = current_user_region());

ALTER TABLE prod_catalog.gold.orders
  SET ROW FILTER orders_region_filter ();

At petabyte scale, proper Delta table tuning is critical for performance and cost.

-- 1. OPTIMIZE + Z-ORDER (run weekly on Gold)
OPTIMIZE prod_catalog.gold.events
  ZORDER BY (event_date, user_id);

-- 2. Liquid Clustering (Databricks 13+) 
--    replaces static partitioning
ALTER TABLE prod_catalog.silver.events
  CLUSTER BY (event_date, region);

-- 3. VACUUM old files (30-day retention)
VACUUM prod_catalog.gold.events
  RETAIN 720 HOURS;

-- 4. Bloom filter for high-cardinality lookups
CREATE BLOOMFILTER INDEX
  ON TABLE prod_catalog.gold.events
  FOR COLUMNS (user_id OPTIONS (fpp=0.1));

Monitor costs, performance, and data quality proactively — before they become incidents.

• Enable Cluster Usage & DBU dashboards in Databricks Admin Console
• Export cluster events to Spark UI / Log Analytics
• Set up DLT data quality expectations with @dlt.expect_or_fail
• Budget alerts via Databricks Cost Management or AWS Cost Explorer
• Use Spot interruption handling with autoscaling retry policies
• Enable predictive autoscaling (Databricks Enhanced Autoscaling)

# DLT Data Quality Expectations
import dlt

@dlt.table
@dlt.expect_or_drop("valid_order_id",
    "order_id IS NOT NULL")
@dlt.expect_or_fail("positive_amount",
    "amount > 0")
def silver_orders():
    return dlt.read("bronze_orders") \
        .filter("status != 'CANCELLED'")