Databricks Instance Profile Creation Failure - "AWS error: You are not authorized to perform this operation"

Question

I'm trying to create a databricks instance profile for use with a previously provisioned workspace and getting the following error when running terraform apply:

2022-01-25T09:32:31.063-0800 [DEBUG] provider.terraform-provider-databricks_v0.4.4: 400 Bad Request {
  "error_code": "DRY_RUN_FAILED",
  "message": "Verification of the instance profile failed. AWS error: You are not authorized to perform this o... (616 more bytes)"
}: timestamp=2022-01-25T09:32:31.062-0800
2022-01-25T09:32:31.063-0800 [WARN]  provider.terraform-provider-databricks_v0.4.4: /api/2.0/instance-profiles/add:400 - Verification of the instance profile failed. AWS error: You are not authorized to perform this operation. Encoded authorization failure message: 5AzyUESoYe18kM...

This is what I see when I decode the Encoded authorization failure message:

{
  "allowed": false,
  "explicitDeny": false,
  "matchedStatements": {
    "items": []
  },
  "failures": {
    "items": []
  },
  "context": {
    "principal": {
      "id": "AROA4A2DDDVLP3F64BTD7:databricks",
      "arn": "arn:aws:sts::<AWS Account ID>:assumed-role/<AWS Account alias>-crossaccount/databricks"
    },
    "action": "iam:PassRole",
    "resource": "arn:aws:iam::<AWS Account ID>:role/databricks-shared-ec2-role-for-s3",
    "conditions": {
      "items": [
        {
          "key": "aws:Region",
          "values": {
            "items": [
              {
                "value": "us-east-1"
              }
            ]
          }
        },
        {
          "key": "aws:Service",
          "values": {
            "items": [
              {
                "value": "ec2"
              }
            ]
          }
        },
        {
          "key": "aws:Resource",
          "values": {
            "items": [
              {
                "value": "role/databricks-shared-ec2-role-for-s3"
              }
            ]
          }
        },
        {
          "key": "iam:RoleName",
          "values": {
            "items": [
              {
                "value": "databricks-shared-ec2-role-for-s3"
              }
            ]
          }
        },
        {
          "key": "aws:Type",
          "values": {
            "items": [
              {
                "value": "role"
              }
            ]
          }
        },
        {
          "key": "aws:Account",
          "values": {
            "items": [
              {
                "value": "<AWS Account ID>"
              }
            ]
          }
        },
        {
          "key": "aws:ARN",
          "values": {
            "items": [
              {
                "value": "arn:aws:iam::<AWS Account ID>:role/databricks-shared-ec2-role-for-s3"
              }
            ]
          }
        }
      ]
    }
  }
}

I'm trying to follow the databricks documentation.

Here's the relevant terraform code fragment:

data "aws_iam_policy_document" "instance-assume-role-policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "role_for_s3_access" {
  name               = "databricks-shared-ec2-role-for-s3"
  description        = "Role for shared access for Databricks"
  assume_role_policy = data.aws_iam_policy_document.instance-assume-role-policy.json
}

data "aws_iam_policy_document" "pass_role_for_s3_access" {
  statement {
    effect    = "Allow"
    actions   = ["iam:PassRole"]
    resources = [aws_iam_role.role_for_s3_access.arn]
  }
}

resource "aws_iam_policy" "pass_role_for_s3_access" {
  name   = "shared-pass-role-for-s3-access"
  path   = "/"
  policy = data.aws_iam_policy_document.pass_role_for_s3_access.json
}
resource "aws_iam_role_policy_attachment" "pass_role_for_s3_access" {
  policy_arn = aws_iam_policy.pass_role_for_s3_access.arn
  role       = aws_iam_role.role_for_s3_access.id
}

resource "aws_iam_instance_profile" "read" {
  name = "sophi-aux_read_instance_profile"
  role = aws_iam_role.role_for_s3_access.name
}

resource "time_sleep" "wait" {
  depends_on = [aws_iam_instance_profile.read]
  create_duration = "10s"
}

resource "databricks_instance_profile" "read" {
  instance_profile_arn = aws_iam_instance_profile.read.arn
}

Any inputs will be greatly appreciated.

Answer 1

Your code looks correct to me.

It sounds like the EC2 role being used by Databricks doesn't have permissions to create an instance profile and/or role.

This is a permission you'll have to explicitly add to the EC2 role on the AWS side by allowing the CreateInstanceProfile and CreateRole actions.