'Azure DevOps CD Pipeline to Deploy Library to Databricks DBFS 403 Forbidden Error

I'm following the tutorial Continuous integration and delivery on Azure Databricks using Azure DevOps to automate the process to deploy and install library on an Azure Databricks cluster. However, I'm stucked in the step "Deploy the library to DBFS" using task Databricks files to DBFS in Databricks Script Deployment Task extension by Data Thirst.

It continuously gives me this error:

##[error]The remote server returned an error: (403) Forbidden.

The configuration of this task is shown below: configuration of Databricks files to DBFS

I've checked with my token that it works fine when I try to upload the libraries manually through Databricks CLI. Thus, the problem shouldn't be due to the permission of the token.

Can anyone suggest any solution to this? Or is there any alternative way to deploy libraries to clusters on Azure Databricks via the release CD pipelines on Azure DevOps?

azureazure-devopsazure-pipelinesazure-databricks


Solution 1:[1]

Did you check your Azure Region in Databricks? If you don't use the same Azure Region in Azure Devops, you will get 403 error.

Solution 2:[2]

After trying multiple times, it turns out if you don't use the extension and use Databricks CLI in the pipeline to directly upload files, the uploading will work smoothly. Hope this helps if someone got the same problem.

Solution 3:[3]

I also faced similar problem while using the Databricks Script Deployment Task created by Data Thirst. Then switched to DevOps for Azure Databricks created by Microsoft DevLabs. Below are the steps I used to work with Databricks CLI to achieve what I wanted to do as part of Azure Release Pipeline:

  1. First, added Use Python version task. Referred to Python 3.7
  2. Then, added Configure Databricks CLI. Provided workspace URL, e.g. adb-1234567890123456.12.azuredatabricks.net, and provided the personal access token by referring to a secret variable
  3. Added a Command Line Script task, and added Databricks CLI scripts as inline code. Moreover, added --profile AZDO along with the scripts as this profile is configured in the previous step. E.g., dbfs cp $(System.DefaultWorkingDirectory)/abcd dbfs:/mytempfiles --recursive --overwrite --profile AZDO

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Cristian
Solution 2 Selena Li
Solution 3 Vivek Atal

Related Questions