Loading Enterprise DataOps Playbook...
Production-grade configs for ADF, Databricks, Airflow, dbt, SQL, and DevOps. Defend end-to-end big data architectures with battle-tested talking points and code templates.
HTTPS REST API via Azure Private Link (VNet Peered)
Connect ADF to Azure Databricks using ADF's System-Assigned or User-Assigned Managed Identity. Eliminates static Personal Access Tokens (PATs) and handles automatic credential rotation via Microsoft Entra ID.
{
"name": "LS_AzureDatabricks_ManagedIdentity",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"description": "ADF to Databricks job execution via Managed Identity",
"type": "AzureDatabricks",
"typeProperties": {
"domain": "https://adb-1234567890123456.7.azuredatabricks.net",
"authentication": "MSI",
"workspaceResourceId": "/subscriptions/sub-id/resourceGroups/rg-data-prod/providers/Microsoft.Databricks/workspaces/adb-workspace-prod",
"newClusterNodeType": "Standard_E4ds_v5",
"newClusterNumOfWorkers": "2:8",
"newClusterSparkVersion": "14.3.x-scala2.12",
"newClusterInitScripts": [],
"newClusterSparkConf": {
"spark.databricks.delta.preview.enabled": "true"
}
},
"connectVia": {
"referenceName": "AutoResolveIntegrationRuntime-ManagedVNet",
"type": "IntegrationRuntimeReference"
}
}
}Zero secrets required in Key Vault for this link. Entra ID generates short-lived OAuth tokens under the hood automatically.
“Always emphasize: 'In enterprise production, we mandate Managed Identity or Entra Workload Identity over PAT tokens. This removes credential sprawl, enforces role-based access control inside Databricks, and guarantees compliance with zero-trust networking through Managed Private Endpoints.'”
HTTPS REST (DFS endpoint) & Azure Key Vault REST API
HTTPS REST API 2.1 via DatabricksHook
SQL over HTTPS (HTTP Path / Thrift) with Delta format
T-SQL TDS over Port 1433 with Entra ID
OpenID Connect (OIDC) Workload Identity Federation