AWS Databricks Enterprise Automation – Workspaces, Isolation & RBAC

AWS Databricks Enterprise Automation

Workspaces, Environment Isolation, Unity Catalog & RBAC – Fully Automated

---

Why Enterprise Automation Is Mandatory

In enterprise environments, Databricks must be deployed with:

• Strict environment isolation (Dev / QA / Prod)
• Centralized identity and access management
• Fine-grained data access controls
• Auditable and repeatable infrastructure

Manual workspace creation or UI-based permission management does not scale and introduces security risk. This blog shows how to automate everything on AWS.

---

High-Level Architecture

AWS Account │ ├── Databricks Account (Control Plane) │ ├── Unity Catalog Metastore (Single, Central) │ ├── SCIM Groups (Synced from IdP) │ │ │ ├── Workspace: Dev │ │ ├── VPC + Subnets │ │ ├── S3 Bucket (Dev Only) │ │ └── Cluster Policies (Small / Auto-Terminate) │ │ │ └── Workspace: Prod │ ├── VPC + Subnets │ ├── S3 Bucket (Prod Only) │ └── Cluster Policies (Restricted / Large)

---

Technology Stack Used

Component	Purpose
Terraform	Workspace, network, storage, cluster policy automation
Databricks REST API / SDK	Unity Catalog, RBAC, grants
AWS S3	Managed storage for Unity Catalog
AWS IAM	Secure access to data storage
SCIM Groups	User → group → permission mapping

---

Step 0 – Prerequisites (One-Time Setup)

AWS Side

• Create dedicated S3 buckets per environment
• Create IAM roles with least privilege access
• Enable VPC endpoints for S3 (no public internet)

Databricks Account

• Databricks Enterprise (Premium) account
• Account-level admin access
• Unity Catalog enabled

---

Step 1 – Automated Workspace Creation (Terraform)

Provider Configuration

provider "databricks" {
  host  = var.databricks_account_host
  token = var.databricks_account_token
}

Storage Configuration

resource "databricks_mws_storage_configs" "dev_storage" {
  account_id   = var.account_id
  name         = "dev-storage"
  bucket_name  = "dbx-dev-bucket"
  iam_role_arn = var.dev_iam_role
}

Workspace Creation

resource "databricks_mws_workspaces" "dev" {
  account_id  = var.account_id
  workspace_name = "dbx-dev"
  region      = "us-east-1"
  storage_configuration_id =
    databricks_mws_storage_configs.dev_storage.id
  sku = "premium"
}

Each workspace is fully isolated at the network, storage, and compute layer.

---

Step 2 – Unity Catalog Metastore Automation

Create Metastore

resource "databricks_metastore" "main" {
  name          = "enterprise-metastore"
  storage_root  = "s3://databricks-uc-root/"
  region        = "us-east-1"
}

Attach Metastore to Workspaces

resource "databricks_metastore_assignment" "dev" {
  workspace_id = databricks_mws_workspaces.dev.workspace_id
  metastore_id = databricks_metastore.main.id
}

---

Step 3 – Catalog, Schema & Table Creation (Python)

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

w.catalogs.create(
    name="prod_catalog",
    comment="Production data"
)

w.schemas.create(
    name="sales",
    catalog_name="prod_catalog"
)

Table Creation

spark.sql("""
CREATE TABLE prod_catalog.sales.customers (
  id STRING,
  name STRING,
  country STRING
) USING DELTA
""")

---

Step 4 – RBAC (User A vs User B Example)

Groups

• group_prod_engineers
• group_dev_engineers

Grant Permissions

w.grants.update(
  securable_type="table",
  securable_name="prod_catalog.sales.customers",
  changes=[
    {"principal": "group_prod_engineers", "privileges": ["SELECT", "MODIFY"]},
    {"principal": "group_dev_engineers", "privileges": ["SELECT"]}
  ]
)

Result

• User A (Prod group): Read + Write
• User B (Dev group): Read-only

RBAC is enforced at query time, not at notebook level.

---

Step 5 – Cluster Isolation with Policies

resource "databricks_cluster_policy" "prod_policy" {
  name = "prod-policy"
  definition = jsonencode({
    node_type_id = {
      type  = "fixed"
      value = "i3.2xlarge"
    }
    autotermination_minutes = {
      type  = "fixed"
      value = 60
    }
  })
}

Attach this policy only to group_prod_engineers.

---

Step 6 – CI/CD Automation Flow

Git Commit ↓ Terraform Apply ↓ Workspace + Storage + Policies ↓ Python SDK ↓ Catalogs + Schemas + RBAC

---

What This Enables Next

• Safe cross-workspace data sharing
• Read-only Prod access from Dev
• Strong audit and compliance posture
• Zero-touch onboarding for new teams

---

Enterprise Outcome

This setup gives you:

• Environment isolation at every layer
• Identity-driven access control
• Full automation and repeatability
• Security that auditors trust

---

Next Blog

Step 3 – Advanced Unity Catalog Patterns:
External Locations, Row-Level Security, Dynamic Views, and Cross-Account Sharing.