Databricks Architecture Matrix (Serverless on AWS)
This document shows where major Databricks components live when using Serverless on AWS, including Disaster Recovery strategies and Terraform resources used for automation.
Control Plane Components
| Component | Purpose | Where It Runs | Plane | DR Best Practice | Terraform Resource |
|---|---|---|---|---|---|
| Workspace | Main analytics workspace | Databricks SaaS | Control Plane | Create secondary workspace in another region | databricks_mws_workspaces |
| Users | User identity | Databricks account | Control Plane | Use centralized IdP | databricks_user |
| Groups | Access management | Databricks account | Control Plane | Manage via SCIM | databricks_group |
| Group Membership | User-group association | Databricks account | Control Plane | Recreate from IaC | databricks_group_member |
| Notebook Source Code | Notebook scripts | Workspace storage | Control Plane | Store notebooks in Git | databricks_notebook |
| Repos (Git Integration) | Source code integration | Workspace metadata | Control Plane | Keep Git remote as source of truth | databricks_repo |
| Job Scheduler | Pipeline scheduling | Databricks control services | Control Plane | Define jobs as code | databricks_job |
| Cluster Configuration | Compute definition | Databricks control services | Control Plane | Recreate clusters via IaC | databricks_cluster |
| SQL Warehouse | Serverless SQL endpoint | Databricks control services | Control Plane | Recreate warehouse in DR region | databricks_sql_endpoint |
| Unity Catalog Metastore | Metadata store | Databricks metadata service | Control Plane | Replicate configuration | databricks_metastore |
| Unity Catalog Catalog | Top level data container | Databricks governance service | Control Plane | Recreate catalogs | databricks_catalog |
| Unity Catalog Schema | Database layer | Databricks governance service | Control Plane | Recreate schema structure | databricks_schema |
| Permissions | Access control policies | Databricks governance service | Control Plane | Store as code | databricks_grants |
| Model Registry | ML model version tracking | Databricks metadata services | Control Plane | Replicate model metadata | databricks_mlflow_model |
| Feature Store Metadata | ML feature definitions | Databricks metadata services | Control Plane | Store definitions in Git | databricks_feature_table |
Data Plane Components (AWS)
| Component | Purpose | Where It Runs | Plane | DR Best Practice | Terraform Resource |
|---|---|---|---|---|---|
| S3 Data Lake | Primary storage | AWS S3 | Data Plane | Enable cross-region replication | aws_s3_bucket |
| Delta Tables | Structured data storage | S3 | Data Plane | Replicate bucket | aws_s3_bucket |
| DBFS Root Storage | Databricks filesystem | S3 | Data Plane | Enable bucket versioning | aws_s3_bucket |
| MLflow Artifact Storage | Stores ML models | S3 | Data Plane | Replicate artifact bucket | aws_s3_bucket |
| Streaming Checkpoints | Streaming progress tracking | S3 | Data Plane | Replicate checkpoint folders | aws_s3_bucket |
| Feature Store Data | ML training features | S3 | Data Plane | Enable replication | aws_s3_bucket |
| Execution Logs | Spark logs | S3 | Data Plane | Central logging system | aws_s3_bucket |
| Serverless Spark Compute | Job execution | AWS compute | Data Plane | Use multi-region workspace | N/A (managed by Databricks) |
| Temporary Spark Shuffle Data | Intermediate processing | Compute disk | Data Plane | No DR required | N/A |
Architecture Flow
Databricks Control Plane
Workspace
Unity Catalog
Metastore
Jobs
Clusters
SQL Warehouses
|
| Secure API
v
AWS Data Plane
Serverless Spark Compute
Delta Tables
ML Models
Streaming Checkpoints
|
v
Amazon S3 Data Lake