Databricks REST API – Complete Enterprise Automation Guide (Python + AWS)
Databricks REST API – Complete Enterprise Automation Guide
This guide documents almost all commonly used Databricks REST API endpoints
with working Python examples for enterprise automation on AWS.
0️⃣ Authentication & Base Configuration
Account-Level APIs
Base URL: https://accounts.cloud.databricks.com
Auth: Account PAT
Workspace-Level APIs
Base URL: https://dbc-xxxx.region.databricks.com
Auth: Workspace PAT
import requests
ACCOUNT_ID = "xxxx"
ACCOUNT_HOST = "https://accounts.cloud.databricks.com"
WORKSPACE_HOST = "https://dbc-xxxx.us-east-1.databricks.com"
ACCOUNT_HEADERS = {
"Authorization": "Bearer ACCOUNT_TOKEN",
"Content-Type": "application/json"
}
WORKSPACE_HEADERS = {
"Authorization": "Bearer WORKSPACE_TOKEN",
"Content-Type": "application/json"
}
1️⃣ Identity & SCIM APIs
| Endpoint | Purpose |
|---|
| POST /scim/v2/Users | Create user |
| GET /scim/v2/Users | List users |
| POST /scim/v2/Groups | Create group |
| PATCH /scim/v2/Groups/{id} | Add/remove members |
Create User
url = f"{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}/scim/v2/Users"
payload = {
"userName": "alice@company.com",
"displayName": "Alice",
"active": true
}
requests.post(url, headers=ACCOUNT_HEADERS, json=payload).raise_for_status()
Create Group
url = f"{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}/scim/v2/Groups"
payload = {"displayName": "data-engineers"}
group = requests.post(url, headers=ACCOUNT_HEADERS, json=payload).json()
2️⃣ Workspace (Account-Level) APIs
| Endpoint | Description |
|---|
| POST /workspaces | Create workspace |
| GET /workspaces | List workspaces |
| POST /permissionassignments | Assign groups to workspace |
Create Workspace
url = f"{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}/workspaces"
payload = {
"workspace_name": "prod",
"aws_region": "us-east-1",
"credentials_id": "cred-123",
"storage_configuration_id": "storage-123",
"network_id": "network-123"
}
workspace = requests.post(url, headers=ACCOUNT_HEADERS, json=payload).json()
3️⃣ Cluster APIs
| Endpoint | Description |
|---|
| POST /clusters/create | Create cluster |
| GET /clusters/list | List clusters |
| POST /clusters/start | Start cluster |
| POST /clusters/delete | Delete cluster |
Create Cluster
url = f"{WORKSPACE_HOST}/api/2.0/clusters/create"
payload = {
"cluster_name": "engineering",
"spark_version": "13.3.x-scala2.12",
"node_type_id": "m5.xlarge",
"num_workers": 2
}
cluster = requests.post(url, headers=WORKSPACE_HEADERS, json=payload).json()
Set Cluster Permissions
url = f"{WORKSPACE_HOST}/api/2.0/permissions/clusters/{cluster['cluster_id']}"
payload = {
"access_control_list": [
{
"group_name": "data-engineers",
"permission_level": "CAN_ATTACH_TO"
}
]
}
requests.patch(url, headers=WORKSPACE_HEADERS, json=payload)
4️⃣ Jobs API
| Endpoint | Purpose |
|---|
| POST /jobs/create | Create job |
| POST /jobs/run-now | Run job |
| GET /jobs/list | List jobs |
Create Job
url = f"{WORKSPACE_HOST}/api/2.0/jobs/create"
payload = {
"name": "etl-job",
"new_cluster": {
"spark_version": "13.3.x-scala2.12",
"node_type_id": "m5.large",
"num_workers": 2
},
"notebook_task": {
"notebook_path": "/Shared/etl"
}
}
job = requests.post(url, headers=WORKSPACE_HEADERS, json=payload).json()
5️⃣ SQL & Warehouses API
| Endpoint | Description |
|---|
| POST /sql/warehouses | Create SQL warehouse |
| POST /sql/statements | Execute SQL |
Execute SQL
url = f"{WORKSPACE_HOST}/api/2.0/sql/statements"
payload = {
"statement": "SELECT current_user(), current_date()",
"warehouse_id": "wh-123"
}
requests.post(url, headers=WORKSPACE_HEADERS, json=payload)
6️⃣ DBFS & Workspace APIs
| Endpoint | Description |
|---|
| POST /dbfs/put | Upload file |
| GET /workspace/list | List notebooks |
| POST /workspace/import | Import notebook |
Upload File to DBFS
url = f"{WORKSPACE_HOST}/api/2.0/dbfs/put"
payload = {
"path": "/tmp/data.txt",
"contents": "SGVsbG8="
}
requests.post(url, headers=WORKSPACE_HEADERS, json=payload)
7️⃣ Unity Catalog APIs (Most Used)
| Endpoint | Description |
|---|
| POST /unity-catalog/catalogs | Create catalog |
| POST /unity-catalog/schemas | Create schema |
| POST /unity-catalog/tables | Create table |
| PATCH /unity-catalog/permissions | Grant access |
Create Catalog
url = f"{WORKSPACE_HOST}/api/2.1/unity-catalog/catalogs"
payload = {"name": "finance"}
requests.post(url, headers=WORKSPACE_HEADERS, json=payload)
Grant Table Access
url = f"{WORKSPACE_HOST}/api/2.1/unity-catalog/permissions/table/finance.payments.txns"
payload = {
"changes": [{
"principal": "data-scientists",
"add": ["SELECT"]
}]
}
requests.patch(url, headers=WORKSPACE_HEADERS, json=payload)
8️⃣ Tokens, Secrets, Repos
| Endpoint | Use |
|---|
| POST /token/create | Create PAT |
| POST /secrets/scopes/create | Create secret scope |
| POST /repos | Create repo |
9️⃣ Enterprise Best Practices
- Terraform for bootstrap & security
- Python APIs for day-2 operations
- Unity Catalog for ALL data access
- No IAM-based data access
This API-first approach is used by regulated banks, fintech, and large enterprises.
Next Topics You Can Publish
- Databricks CI/CD pipelines
- API error handling & retries
- Zero-trust data architecture
- Cross-account Unity Catalog sharing