AWS Codepipeline Multiple Output Artifacts

Question

Still fairly new to AWS Codepipeline and I am trying to pass an output artifact out into the next stage of my build. In this particular case, I want to do two artifacts for the builds phase which is the only part I am focusing on right now. I have included my codepipeline code for reference:

resource "aws_codepipeline" "cp_plan_pipeline" {

  name     = "${local.name_prefix_H}-${var.cp_name}"
  role_arn = aws_iam_role.cp_service_role.arn

  artifact_store {
    type     = var.cp_artifact_type
    location = module.S3.bucket_name
  }

  stage {
    name = "Clone"

      action {
        name                = "Source"
        category            = "Source"
        owner               = "AWS"
        provider            = "CodeCommit"
        input_artifacts     = [] 
        version             = "1"
        output_artifacts    = ["CodeWorkspace"]
        
        configuration = {
          RepositoryName        = var.cp_repo_name
          BranchName            = var.cp_branch_name
          PollForSourceChanges  = var.cp_poll_sources
          OutputArtifactFormat  = var.cp_ouput_format
        }

        run_order = "1"
      }
  }

  stage {
    name = "Plan"

      action {
          name              = "Terraform-Plan"
          category          = "Build"
          owner             = "AWS"
          provider          = "CodeBuild"
          version           = "1"
          input_artifacts   = ["CodeWorkspace"]
          output_artifacts  = ["CodeSource","TerraformPlanFile"]
        
          configuration = {
            ProjectName          = var.cp_plan_project_name
            EnvironmentVariables = jsonencode([
              {
                name  = "PIPELINE_EXECUTION_ID"
                value = "#{codepipeline.PipelineExecutionId}"
                type  = "PLAINTEXT"
              }
            ])
          }
      }
  }

  stage {
      name = "Test"

      action {
        name              = "Testing"
        category          = "Test"
        owner             = "AWS"
        provider          = "CodeBuild"
        input_artifacts   = ["CodeSource"]
        output_artifacts  = ["TestOutput"]
        version           = "1"

        configuration = {
          ProjectName = var.cp_test_project_name
          EnvironmentVariables = jsonencode([
              {
                name  = "PIPELINE_EXECUTION_ID"
                value = "#{codepipeline.PipelineExecutionId}"
                type  = "PLAINTEXT"
              }
          ])
        }
      }
  }

  stage {
    name = "Manual-Approval"

    action {
      run_order = 1
      name                = "AWS-Admin-Approval"
      category            = "Approval"
      owner               = "AWS"
      provider            = "Manual"
      version             = "1"
      input_artifacts     = []
      output_artifacts    = []

      configuration  = {
          CustomData      = "Please verify the terraform plan output on the Plan stage and only approve this step if you see expected changes!"
      }
    }
  }


  stage {
    name = "Deploy"

    action {
      run_order           = 1
      name                = "Terraform-Apply"
      category            = "Build"
      owner               = "AWS"
      provider            = "CodeBuild"
      input_artifacts     = ["TerraformPlanFile"]
      output_artifacts    = []
      version             = "1"

      configuration = {
        ProjectName          = var.cp_apply_project_name
        PrimarySource        = "CodeWorkspace"
        EnvironmentVariables = jsonencode([
          {
            name  = "PIPELINE_EXECUTION_ID"
            value = "#{codepipeline.PipelineExecutionId}"
            type  = "PLAINTEXT"
          }
        ])
      }
    }
    }

}

So the pipeline works fine as far as the deployment of the pipeline goes. My buildspec works great up until the point where I get to the artifact stage of the buildspec where i get the following error: Phase context status code: CLIENT_ERROR Message: no definition for secondary artifact CodeSource in buildspec My issue is the buildspec and trying to output the second source so i can upload it for the second phase. Please note some parts of the buildspec such as variables have been scrubbed. Buildspec below:

version: 0.2
env:
  variables:
    TF_VERSION: "1.0.7"
    PY_VERSION: "3.9.6"
    GIT_VERSION: "2.9.5"
    PACKER_VERSION: "1.7.8"
    JQ_VERSION: "1.6"
    TFLINT_VERSION: "0.34.0"
    PERMISSION_SETS_DIR: "CodeSource"
    
phases:
  install:
    commands:
      # iNSTALL/UPDATE SSH CLIENT
      - echo UPDATING SSH CLIENT
      - "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"

      # iNSTALL JQ JSON PARSER
      - curl -s -qL -o jq https://github.com/stedolan/jq/releases/download/jq-${JQ_VERSION}/jq-linux64
      - chmod +x ./jq
      - cp jq /usr/bin
      
      # INSTALL TERRAFORM
      - echo STARTING TERRAFORM INSTALLATION
      - curl -s -qL -o terraform.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip
      - unzip terraform -d /usr/bin/
      - "chmod +x /usr/bin/terraform"
      - "/usr/bin/terraform --version"

      # INSTALL Packer
      - echo STARTING PACKER INSTALLATION
      - curl -s -qL -o packer.zip https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip
      - unzip packer -d /usr/bin/
      - "chmod +x /usr/bin/packer"
      - "/usr/bin/packer --version"

      # INSTALL PYTHON
      - echo "STARTING PYTHON INSTALLATION"
      - "curl -s -qL -o python.tgz https://www.python.org/ftp/python/${PY_VERSION}/Python-${PY_VERSION}.tgz"
      - "tar xf python.tgz -C /usr/bin/"
      - "python --version"
      - python -m pip install -U pip
      
      # ADD PYTHON MODULES
      - echo "INSTALL PYTHON MODULES"
      - pip install git-remote-codecommit

  pre_build:
    commands:
      # Adds a private SSH key to allow us to clone or npm install Git repositories
      - rootpath=$(pwd)
      - eval $(ssh-agent -s)
      - mkdir -p ~/.ssh

      # Configure SSH Key
      - echo "$ssh_key" > ~/.ssh/cc_rsa
      - cd ~/.ssh/
      - cat cc_rsa
      - |
        echo "Multiline command"
        cat > ~/.ssh/config <<EOL
        Host idt-codecommit
            Hostname git-codecommit.us-east-1.amazonaws.com
            User ${cc_user}
            IdentityFile ~/.ssh/cc_rsa
        EOL
      - cat ~/.ssh/config
      
      # Configure SSH Permissions
      - echo +++++++++CONFIG SSH+++++++++
      - chmod 700 ~/.ssh
      - chmod 600 ~/.ssh/config
      - chmod 600 ~/.ssh/cc_rsa
      - ssh-keyscan -t rsa1,rsa,dsa git-codecommit.us-east-1.amazonaws.com >> ~/.ssh/known_hosts

      # Clone directories
      - echo +++++++++CLONE DIRECTORIES+++++++++
      - mkdir -p ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}
      - cd ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}
      - git clone codecommit://sourcerepo local_primary_repo
      - git clone -b development ssh://sourcerepo2/v1/repos/sourcerepo2
      
      # GET AWS ACCOUNT VARIABLES
      - aws_region=$AWS_DEFAULT_REGION
      - awsaccountnumber=$(aws sts get-caller-identity --query "Account")
      - awsaccountname=$(aws iam list-account-aliases | jq -r '.AccountAliases | .[]')
      - |
            if [[ "$awsaccountname" == *"prod"* ]]; then
              appenv="prod"
            else
              if [[ "$awsaccountname" == *"test"* ]]; then
                appenv="test"
              else
                if [[ "$awsaccountname" == *"dev"* ]]; then
                  appenv="dev"
                else
                  if [[ "$awsaccountname" == *"sbx"* ]]; then
                    appenv="sbx"
                  else
                    appenv="other"
                  fi
                fi
              fi
            fi

  build:
    on-failure: ABORT
    commands: 
      # Import S3 Folder Location Variable
      - cd ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}/local_primary_repo
      - CODEBUILD_GIT_BRANCH=`git symbolic-ref HEAD --short 2>/dev/null`

      # CHECK TERRAFORM CODE
      - cd ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}/local_primary_repo
      - echo "yes" | terraform init 
      - terraform validate > ${bucketconfig}-tfvalidateexport.txt
      - terraform validate
      - terraform plan -out=tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID}
      - cp ${bucketconfig}-tfvalidateexport.txt ${CODEBUILD_SRC_DIR}/
      - cp tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID} ${CODEBUILD_SRC_DIR}/
  
  post_build:
    commands:
      - echo "Terraform plan completed on `date`"

artifacts:
  files:
    - tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID}
  name: TerraformPlanFile
  secondary-artifacts:
    artifact_1:  
      files:
        - '**/*'
      name: CodeSource
      base-directory: ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}

Links or advice of what i am doing wrong would be very helpful. Thank you!!

Answer 1

First of I wanted to thank Antonio González for his post. It was exactly what I needed for the solution. I would like to recap the solution here for any other folks looking for an answer to the same issue. I have a working understanding that will help beginners. When Outputting multiple artifacts you have to use the secondary artifacts: action within the phase of the artifacts. See the example below as you follow along:

artifacts:
  base-directory: ${CODEBUILD_SRC_DIR}/
  files:
    - '**/*'
  
  secondary-artifacts:
    ARTIFACT1:  
      base-directory: DIRECTORY_FOR_ARTIFACT_1/
      files:
        - FILES_YOU_LOOKING_TO_OUTPUT
      name: ARTIFACT_1_NAME_ANYTHING_YOU_WANT

    ARTIFACT2:  
      base-directory: DIRECTORY_FOR_ARTIFACT_2/
      files:
        - FILES_YOU_LOOKING_TO_OUTPUT
      name: ARTIFACT_2_NAME_ANYTHING_YOU_WANT

So this is the basic setup for multiple artifacts, if you wanted to do another artifact you would do a third. Now the key to this setup is mapping the secondary-artifacts headings to the outputs that you defined in your codepipeline output. Recall my pipeline build phase:

stage { name = "Plan"

  action {
      name              = "Terraform-Plan"
      category          = "Build"
      owner             = "AWS"
      provider          = "CodeBuild"
      version           = "1"
      input_artifacts   = ["CodeWorkspace"]
      output_artifacts  = ["CodeSource","TerraformPlanFile"]

The output artifacts MUST be associated with the secondary-artifacts heading as shown below:

  secondary-artifacts:
      TerraformPlanFile:
          base-directory: ${CODEBUILD_SRC_DIR}/
          files:
            - tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID}
          name: TerraformPlanFile
    
       CodeSource:  
          base-directory: ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}/
          files:
            - '**/*'
          name: CodeSource

Notice how the secondary artifact heading matches the output_artifacts. By ensuring that those heading match with the headings you ensure that the files passed are outputted correctly.