Databricks Roles Full Reference Matri

Databricks Roles – Full Reference Matrix

This table includes Workspace Roles, Account Roles, and Unity Catalog Roles with exact capabilities.

Role	Category	Capabilities / Permissions	Notes
Workspace Admin	Workspace	Manage users and groups Assign workspace roles Create/manage clusters Restart/terminate all clusters Create/manage jobs and workflows Create SQL warehouses Manage secrets, libraries, instance profiles Access DBFS (read/write) Run notebooks and jobs	Full control of workspace; does NOT grant automatic data access in Unity Catalog
User	Workspace	Create/edit/run own notebooks Create/run jobs Create clusters (if allowed by cluster policies) Access DBFS (read/write) Use SQL warehouses (if permitted)	Cannot manage other users or workspace settings
Can Manage / Job Creator	Workspace	Create/manage own jobs and clusters Run notebooks Upload files to DBFS	Limited admin; cannot manage other users or workspace-wide settings
Viewer	Workspace	Read-only access to notebooks, dashboards View clusters and jobs Read access to DBFS (if allowed)	No write permissions
Account Admin	Account	Create and delete workspaces Assign workspace admins Manage metastore assignments Access account-wide audit logs Manage billing / usage	Full control over account; workspace-level roles must still be respected
Billing / Support Roles	Account	View usage and billing Access technical support	Cannot manage workspace or data; read-only account permissions
Metastore Admin	Unity Catalog	Create catalogs and schemas Create storage credentials and external locations Assign catalog-level permissions Grant/revoke data access	Full control over UC metadata; does NOT give workspace admin rights
Catalog Owner	Unity Catalog	Manage catalog and contained schemas Grant/revoke access at catalog level	Limited to one catalog; cannot manage other catalogs
Schema Owner	Unity Catalog	Manage schema and contained tables/views Grant/revoke access at schema level	Cannot manage catalog-level permissions
Volume Owner	Unity Catalog	Manage managed volumes (file storage) Grant/revoke access to volumes	Access to volume paths only
Data Access Roles (SELECT / MODIFY / USAGE)	Unity Catalog	Read/write/query specific tables, views, volumes Can be granted granular privileges via grants	Applied per-object; separate from workspace admin rights

Databricks on AWS – Least Privilege Permission Matrix

This matrix describes the minimal AWS and Databricks permissions required to create or manage common platform resources when using Databricks on AWS. The goal is to follow enterprise least-privilege security principles.

Resource	Primary Owner Role	Required AWS Permissions	Required Databricks Permissions	Purpose	Security / Least Privilege Notes
Workspace	Platform Admin	iam:CreateRole, iam:AttachRolePolicy, ec2:CreateVpc, s3:CreateBucket	Account Admin	Create Databricks workspace	Automate using Terraform and restrict to platform team
Cross Account IAM Role	AWS Cloud Admin	iam:CreateRole, iam:PutRolePolicy, sts:AssumeRole	None	Allows Databricks control plane access	Trust only Databricks account
Root Storage (DBFS)	AWS Cloud Admin	s3:CreateBucket, s3:PutBucketPolicy	None	Workspace default storage	Enable encryption and versioning
Unity Catalog Metastore	Data Platform Admin	s3:GetObject, s3:PutObject, s3:ListBucket	Metastore Admin	Central governance metadata store	Dedicated metastore bucket
Metastore Assignment	Platform Admin	None	Account Admin	Attach metastore to workspace	Single metastore per region recommended
Storage Credential	Data Platform Admin	iam:PassRole, sts:AssumeRole	CREATE STORAGE CREDENTIAL	Connect Unity Catalog to S3	IAM role should allow only specific S3 path
External Location	Data Governance Admin	s3:GetObject, s3:PutObject	CREATE EXTERNAL LOCATION	Expose S3 path to Unity Catalog	Use path-level permissions
Catalog	Data Governance Admin	Access to storage location	CREATE CATALOG	Top governance layer	One catalog per domain recommended
Schema	Data Owner	None	CREATE SCHEMA	Database container	Grant schema-level privileges
Delta Table	Data Engineer	S3 read/write	CREATE TABLE	Structured table storage	Use Unity Catalog governance
External Table	Data Engineer	S3 read	CREATE TABLE	Reference external dataset	Avoid direct S3 access
Notebook	Data Engineer / Analyst	None	Workspace Editor	Analytics code	Store production code in Git
Git Repo Integration	Developer	None	Workspace Editor	Version control integration	Use GitHub / GitLab PAT
Job / Workflow	Data Engineer	None	CREATE JOB	Automated pipelines	Define jobs as code
Cluster	Platform Admin	ec2:RunInstances, iam:PassRole	CREATE CLUSTER	Compute resource	Restrict using cluster policies
SQL Warehouse	Data Engineer	None	CREATE SQL WAREHOUSE	Serverless SQL analytics	Limit compute size via policies
Cluster Policy	Platform Admin	None	CREATE CLUSTER POLICY	Restrict compute usage	Important governance control
Feature Store Table	ML Engineer	S3 read/write	CREATE TABLE	Machine learning features	Stored as Delta tables
ML Model Registry	ML Engineer	S3 artifact storage	CREATE MODEL	Track ML model versions	Store artifacts in secure bucket
Streaming Checkpoints	Data Engineer	s3:PutObject, s3:GetObject	Job permission	Streaming progress tracking	Separate checkpoint directory
Unity Catalog Volume	Data Platform Admin	S3 access	CREATE VOLUME	File storage governance	Alternative to DBFS
Audit Logs	Security Team	S3 write	Account Admin	Security auditing	Send logs to SIEM
PrivateLink Networking	AWS Cloud Admin	ec2:CreateVpcEndpoint	Account Admin	Private connectivity	Required for highly secure environments
DBFS File Upload	User	s3:PutObject	Workspace User	Temporary file storage	Avoid for production data

Databricks Architecture Matrix with DR and Terraform Resources

Databricks Architecture Matrix (Serverless on AWS)

This document shows where major Databricks components live when using Serverless on AWS, including Disaster Recovery strategies and Terraform resources used for automation.

Control Plane Components

Component	Purpose	Where It Runs	Plane	DR Best Practice	Terraform Resource
Workspace	Main analytics workspace	Databricks SaaS	Control Plane	Create secondary workspace in another region	databricks_mws_workspaces
Users	User identity	Databricks account	Control Plane	Use centralized IdP	databricks_user
Groups	Access management	Databricks account	Control Plane	Manage via SCIM	databricks_group
Group Membership	User-group association	Databricks account	Control Plane	Recreate from IaC	databricks_group_member
Notebook Source Code	Notebook scripts	Workspace storage	Control Plane	Store notebooks in Git	databricks_notebook
Repos (Git Integration)	Source code integration	Workspace metadata	Control Plane	Keep Git remote as source of truth	databricks_repo
Job Scheduler	Pipeline scheduling	Databricks control services	Control Plane	Define jobs as code	databricks_job
Cluster Configuration	Compute definition	Databricks control services	Control Plane	Recreate clusters via IaC	databricks_cluster
SQL Warehouse	Serverless SQL endpoint	Databricks control services	Control Plane	Recreate warehouse in DR region	databricks_sql_endpoint
Unity Catalog Metastore	Metadata store	Databricks metadata service	Control Plane	Replicate configuration	databricks_metastore
Unity Catalog Catalog	Top level data container	Databricks governance service	Control Plane	Recreate catalogs	databricks_catalog
Unity Catalog Schema	Database layer	Databricks governance service	Control Plane	Recreate schema structure	databricks_schema
Permissions	Access control policies	Databricks governance service	Control Plane	Store as code	databricks_grants
Model Registry	ML model version tracking	Databricks metadata services	Control Plane	Replicate model metadata	databricks_mlflow_model
Feature Store Metadata	ML feature definitions	Databricks metadata services	Control Plane	Store definitions in Git	databricks_feature_table

Data Plane Components (AWS)

Component	Purpose	Where It Runs	Plane	DR Best Practice	Terraform Resource
S3 Data Lake	Primary storage	AWS S3	Data Plane	Enable cross-region replication	aws_s3_bucket
Delta Tables	Structured data storage	S3	Data Plane	Replicate bucket	aws_s3_bucket
DBFS Root Storage	Databricks filesystem	S3	Data Plane	Enable bucket versioning	aws_s3_bucket
MLflow Artifact Storage	Stores ML models	S3	Data Plane	Replicate artifact bucket	aws_s3_bucket
Streaming Checkpoints	Streaming progress tracking	S3	Data Plane	Replicate checkpoint folders	aws_s3_bucket
Feature Store Data	ML training features	S3	Data Plane	Enable replication	aws_s3_bucket
Execution Logs	Spark logs	S3	Data Plane	Central logging system	aws_s3_bucket
Serverless Spark Compute	Job execution	AWS compute	Data Plane	Use multi-region workspace	N/A (managed by Databricks)
Temporary Spark Shuffle Data	Intermediate processing	Compute disk	Data Plane	No DR required	N/A

Architecture Flow

Databricks Control Plane Workspace Unity Catalog Metastore Jobs Clusters SQL Warehouses | | Secure API v AWS Data Plane Serverless Spark Compute Delta Tables ML Models Streaming Checkpoints | v Amazon S3 Data Lake

Databricks Architecture Matrix with DR Best Practices

Databricks Architecture Matrix (Serverless on AWS) with DR Best Practices

This document explains where major Databricks components reside when running Databricks Serverless on AWS and the recommended Disaster Recovery (DR) strategy for each component.

Control Plane Components

Component	Purpose	Where It Runs	Plane	DR Best Practice
Workspace UI	User interface for notebooks and jobs	Databricks SaaS	Control Plane	Create secondary workspace in another region
Workspace APIs	Automation APIs	Databricks SaaS	Control Plane	Automate infrastructure using Terraform
Users & Groups	User identity management	Databricks account services	Control Plane	Use centralized IdP like Okta/Azure AD
Authentication / SSO	Login via external identity provider	Databricks account services	Control Plane	Configure SSO redundancy at IdP level
Permissions / RBAC	Access control policies	Databricks control services	Control Plane	Store policies as code using Terraform
Notebook Source Code	Notebook scripts	Workspace storage	Control Plane	Sync notebooks with Git repositories
Notebook Outputs	Charts and query results	Workspace storage	Control Plane	Do not rely on outputs; regenerate from data
Workspace Files	Files uploaded to workspace	Workspace storage	Control Plane	Store important files in S3 or Git
Repos (Git Integration)	Git source control integration	Workspace metadata	Control Plane	Maintain source code in GitHub/GitLab
Job Scheduler	Schedules workflows	Databricks orchestration service	Control Plane	Define jobs using Infrastructure-as-Code
Workflows	Pipeline orchestration	Databricks orchestration service	Control Plane	Export workflows via API and Terraform
SQL Query Planner	SQL optimization engine	Databricks query services	Control Plane	No DR needed (managed by Databricks)
SQL Warehouse Management	Serverless SQL management	Databricks control services	Control Plane	Recreate warehouses in secondary region
Unity Catalog	Central governance system	Databricks governance service	Control Plane	Replicate catalogs configuration using scripts
Metastore	Metadata storage	Databricks metadata services	Control Plane	Export metadata periodically
Data Lineage	Tracks data relationships	Databricks governance services	Control Plane	Export lineage metadata via APIs
Audit Logs	Security logs	Databricks governance services	Control Plane	Send logs to centralized SIEM storage
Cluster Management	Compute lifecycle management	Databricks control services	Control Plane	Recreate clusters via automation
Feature Store Metadata	Feature definitions	Databricks metadata services	Control Plane	Backup definitions in Git
Model Registry Metadata	ML model tracking	Databricks metadata services	Control Plane	Replicate registry configuration
Lakehouse Monitoring Metadata	Dataset monitoring metrics	Databricks monitoring services	Control Plane	Export monitoring metrics
Vector Search Metadata	Vector index configuration	Databricks control services	Control Plane	Recreate vector indexes from embeddings

Data Plane Components (Customer AWS)

Component	Purpose	Where It Runs	Plane	DR Best Practice
Serverless Spark Compute	Executes jobs	AWS compute	Data Plane	Deploy in multi-region workspace
SQL Warehouse Compute	SQL query execution	AWS compute	Data Plane	Provision warehouses in secondary region
Delta Table Data	Table storage	S3	Data Plane	Enable S3 cross-region replication
Managed Tables	Managed table storage	S3	Data Plane	Use versioned S3 buckets
External Tables	External dataset storage	S3	Data Plane	Replicate underlying S3 storage
DBFS Root	Databricks filesystem	S3	Data Plane	Enable bucket replication
Unity Catalog Managed Storage	Catalog table storage	S3	Data Plane	Cross-region replication
Unity Catalog Volumes	Governed file storage	S3	Data Plane	Replicate S3 buckets
MLflow Model Artifacts	ML models	S3	Data Plane	Replicate artifact bucket
Feature Store Data	ML feature datasets	S3	Data Plane	S3 replication and versioning
Vector Search Index Data	Embedding storage	S3	Data Plane	Rebuild indexes from replicated embeddings
Streaming Checkpoints	Streaming progress tracking	S3	Data Plane	Replicate checkpoint directories
Temporary Spark Shuffle Data	Intermediate processing	Compute disk	Data Plane	No DR required (recomputed)
Job Execution Logs	Spark logs	S3	Data Plane	Send logs to centralized logging system
ML Training Data	Training datasets	S3	Data Plane	Multi-region S3 replication
Delta Transaction Logs	Table version metadata	S3	Data Plane	Protect using S3 versioning

Architecture Flow

Databricks Control Plane (Managed by Databricks) Workspace UI Authentication Unity Catalog Metastore Metadata Query Planner Job Scheduler | | Secure API v AWS Data Plane (Customer Account) Serverless Spark Compute SQL Warehouses Delta Tables ML Models Spark Temp Storage | v Amazon S3 Data Lake

Databricks Architecture Matrix

Databricks Architecture Matrix (Serverless on AWS)

This document explains where major Databricks components reside when running Databricks Serverless on AWS. Databricks architecture is divided into two planes:

• Control Plane – Managed by Databricks
• Data Plane – Runs in the customer AWS account

Control Plane Components

Component	Purpose / What It Does	Where It Runs or Is Stored	Plane
Workspace UI	Web interface to access notebooks, jobs, dashboards	Databricks SaaS infrastructure	Control Plane
Workspace APIs	REST APIs for automation, Terraform, CLI	Databricks SaaS	Control Plane
Users & Groups	Identity and user management	Databricks account services	Control Plane
Authentication / SSO	Integrates with IdP such as Okta or Azure AD	Databricks account services	Control Plane
Permissions / RBAC	Access control policies	Databricks control services	Control Plane
Notebook Source Code	Python / SQL / Scala notebooks	Workspace storage	Control Plane
Notebook Outputs	Charts and result previews	Workspace storage	Control Plane
Workspace Files	Files uploaded to workspace	Workspace storage	Control Plane
Repos (Git Integration)	GitHub / Git integration	Workspace metadata	Control Plane
Job Scheduler	Schedules pipelines and jobs	Databricks orchestration service	Control Plane
Workflows	Pipeline orchestration	Databricks orchestration service	Control Plane
SQL Query Planner	Optimizes SQL queries	Databricks query services	Control Plane
SQL Warehouse Management	Manages serverless SQL endpoints	Databricks control services	Control Plane
Unity Catalog	Central governance system	Databricks governance service	Control Plane
Metastore	Stores catalog and table metadata	Databricks metadata services	Control Plane
Data Lineage	Tracks data dependencies	Databricks governance services	Control Plane
Audit Logs	Security and governance logs	Databricks governance services	Control Plane
Cluster Management	Manages compute lifecycle	Databricks control services	Control Plane
Feature Store Metadata	ML feature definitions	Databricks metadata services	Control Plane
Model Registry Metadata	Tracks ML model versions	Databricks metadata services	Control Plane
Lakehouse Monitoring Metadata	Tracks dataset quality	Databricks monitoring services	Control Plane
Vector Search Metadata	Vector index configuration	Databricks control services	Control Plane

Data Plane Components (Customer AWS Account)

Component	Purpose / What It Does	Where It Runs or Is Stored	Plane
Serverless Spark Compute	Runs notebooks and jobs	AWS compute instances	Data Plane
Databricks SQL Warehouse Compute	Executes SQL queries	AWS compute instances	Data Plane
Delta Table Data	Actual table data	S3	Data Plane
Managed Tables	Databricks managed tables	S3	Data Plane
External Tables	Tables referencing external datasets	S3	Data Plane
DBFS Root	Databricks File System root	S3 bucket	Data Plane
Unity Catalog Managed Storage	Table storage governed by Unity Catalog	S3	Data Plane
Unity Catalog Volumes	File governance storage	S3	Data Plane
MLflow Model Artifacts	ML models and artifacts	S3	Data Plane
Feature Store Data	ML feature datasets	S3	Data Plane
Vector Search Index Data	Vector embeddings	S3	Data Plane
Streaming Checkpoints	Streaming job progress	S3	Data Plane
Temporary Spark Shuffle Data	Intermediate processing data	Local disk / S3	Data Plane
Job Execution Logs	Spark execution logs	S3	Data Plane
ML Training Data	Training datasets	S3	Data Plane
Delta Transaction Logs	Table versioning metadata	S3	Data Plane

Architecture Flow

Databricks Control Plane (Managed by Databricks) Workspace UI Authentication Unity Catalog Metastore Metadata Query Planner Job Scheduler Notebook Code | | Secure API v AWS Data Plane (Customer AWS Account) Serverless Spark Compute SQL Warehouses Delta Tables DBFS Storage ML Models Spark Temp Storage | v Amazon S3 (Customer Data Lake)

Key Architecture Rule

Type	Location
Metadata	Databricks Control Plane
Data	Customer AWS S3
Compute	Customer AWS
Governance Policies	Unity Catalog (Control Plane)