Terraform in aws multi account env created by AWS Control Tower

Question

I have just moved to a multi account set up using Control Tower and am having a 'mare using Terraform to deploy resources in different accounts.

My (simplified) account structure is:

|--Master
   |--management (backends etc)
   |--images     (s3, ecr)
   |--dev
   |--test

As a simplified experiment I am trying to create an ecr in the images account. So I think I need to create a policy to enable role switching and provide permissions within the target account. For now I am being heavy handed and just trying to switch to Admin access. The AWSAdministratorAccess role is created by Control Tower on configuration.

provider "aws" {
   region  = "us-west-2"
   version = "~> 3.1"
}

data "aws_iam_group" "admins" { // want to attach policy to admins to switch role
   group_name = "administrators"
}


// Images account
resource "aws_iam_policy" "images-admin" {
  name        = "Assume-Role-Images_Admin"
  description = "Allow assuming AWSAdministratorAccess role on Images account"
  policy      = <<EOP
  {
     "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::<Images_Account_ID>:role/AWSAdministratorAccess"
          }
        ]
      }
    EOP
   }

 resource "aws_iam_group_policy_attachment" "assume-role-images-admin" {
  group      = data.aws_iam_group.admins.group_name
  policy_arn = aws_iam_policy.images-admin.arn
 }

Having deployed this stack I then attempt to deploy another stack which creates a resource in the images account.

provider "aws" {
  region  = var.region
  version = "~>3.1"
}

provider "aws" { 
  alias   = "images"
  region  = var.region
  version = "~> 3.1"
  assume_role {
     role_arn = "arn:aws:iam::<Images_Account_ID>:role/AWSAdministratorAccess"
  }
}

resource "aws_ecr_repository" "boot-images" {
  provider             = aws.images
  name                 = "boot-images"
}

On deployment I got:

> Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::*********:role/AWSAdministratorAccess) cannot be assumed.

There are a number of possible causes of this - the most common are:
  * The credentials used in order to assume the role are invalid
  * The credentials do not have appropriate permission to assume the role
  * The role ARN is not valid

Error: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

First one: the creds provided are from the master account which always worked in a single account environment

Second: that's what I think has been achieved by attaching the policy

Third: less sure on this but AWSAdministratorAccess defo exists in the account, I think the arn format is correct and while AWS Single Sign On refers to it as a Permission Set the console also describes this as a role.

I found Deploying to multiple AWS accounts with Terraform? which was helpful but I am missing something here.

I am also at a loss of how to extend this idea to deploying an s3 remote backend into my "management" account.

Terraform version 0.12.29

Answer 1

Turns out there were a couple of issues here:

1. Profile

The credentials profile was incorrect. Setting the correct creds in Env Vars let me run a simple test when just using the creds file failed. There is still an issue here I don't understand as updating the creds file also failed but I have a system that works.

2. AWS created roles

While my assumption was correct that the Permission Sets are defined as Roles, they have a trust relationship which was not extended to my Master Admin User (my bad) AND it cannot be amended as it was created by AWS automatically and it is locked down.

3. Manually grant permissions

So while I can grant permissions to a Group to assume a role programatically via Terraform I need to manually create a role in the Target account which extends trust and hence permissions to the Master account.

Answer 2

In my own experience and considering you already have a working AWS infrastructure, I'd rule out and move away from Control Tower and look into doing same things with CloudFormation StackSets. They let you target OUs or individual accounts.

Control Tower has been recommended to me several times, but having an AWS ecosystem of more >25 accounts with production workloads, I am very reluctant to event try. It's great to start from scratch I guess, but not when you already have a decent amount of workloads and accounts in AWS.