How to use the AWS Python SDK while connecting via SSO credentials

Question

I am attempting to create a python script to connect to and interact with my AWS account. I was reading up on it here https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html

and I see that it reads your credentials from ~/.aws/credentials (on a Linux machine). I however and not connecting with an IAM user but SSO user. Thus, the profile connection data I use is located at ~/.aws/sso/cache directory.

Inside that directory, I see two json files. One has the following keys:

• startUrl
• region
• accessToken
• expiresAt

the second has the following keys:

• clientId
• clientSecret
• expiresAt

I don't see anywhere in the docs about how to tell it to use my SSO user.

Thus, when I try to run my script, I get error such as

botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeSecurityGroups operation: AWS was not able to validate the provided access credentials

even though I can run the same command fine from the command prompt.

Answer 1

This was fixed in boto3 1.14.

So given you have a profile like this in your ~/.aws/config:

[profile sso_profile]
sso_start_url = <sso-url>
sso_region = <sso-region>
sso_account_id = <account-id>
sso_role_name = <role>
region = <default region>
output = <default output (json or text)>

And then login with $ aws sso login --profile sso_profile

You will be able to create a session:

import boto3
boto3.setup_default_session(profile_name='sso_profile')
client = boto3.client('<whatever service you want>')

Answer 2

Your current .aws/sso/cache folder structure looks like this:

$ ls
botocore-client-XXXXXXXX.json       cXXXXXXXXXXXXXXXXXXX.json

The 2 json files contain 3 different parameters that are useful.

botocore-client-XXXXXXXX.json -> clientId and clientSecret
cXXXXXXXXXXXXXXXXXXX.json -> accessToken

Using the access token in cXXXXXXXXXXXXXXXXXXX.json you can call get-role-credentials. The output from this command can be used to create a new session.

Your Python file should look something like this:

import json
import os
import boto3

dir = os.path.expanduser('~/.aws/sso/cache')

json_files = [pos_json for pos_json in os.listdir(dir) if pos_json.endswith('.json')]

for json_file in json_files :
    path = dir + '/' + json_file
    with open(path) as file :
        data = json.load(file)
        if 'accessToken' in data:
            accessToken = data['accessToken']

client = boto3.client('sso',region_name='us-east-1')
response = client.get_role_credentials(
    roleName='string',
    accountId='string',
    accessToken=accessToken
)

session = boto3.Session(aws_access_key_id=response['roleCredentials']['accessKeyId'], aws_secret_access_key=response['roleCredentials']['secretAccessKey'], aws_session_token=response['roleCredentials']['sessionToken'], region_name='us-east-1')

Answer 3

A well-formed boto3-based script should transparently authenticate based on profile name. It is not recommended to handle the cached files or keys or tokens yourself, since the official code methods might change in the future. To see the state of your profile(s), run aws configure list --examples:

$ aws configure list --profile=sso

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      sso           manual    --profile

The SSO session associated with this profile has expired or is otherwise invalid.
To refresh this SSO session run aws sso login with the corresponding profile.

$ aws configure list --profile=old

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      old           manual    --profile
access_key     ****************3DSx shared-credentials-file    
secret_key     ****************sX64 shared-credentials-file    
    region                us-west-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

Answer 4

So here's the long and hairy answer tested on boto3==1.21.39:

It's an eight-step process where:

1. register the client using sso-oidc.register_client
2. start the device authorization flow using sso-oidc.start_device_authorization
3. redirect the user to the sso login page using webbrowser.open
4. poll sso-oidc.create_token until the user completes the signin
5. list and present the account roles to the user using sso.list_account_roles
6. get role credentials using sso.get_role_credentials
7. create a new boto3 session with the session credentials from (6)
8. eat a cookie

Step 8 is really key and should not be overlooked as part of any successful authorization flow.

In the sample below the account_id should be the account id of the account you are trying to get credentials for. And the start_url should be the url that aws generates for you to start the sso flow (in the AWS SSO management console, under Settings).

from time import time, sleep
import webbrowser
from boto3.session import Session
session = Session()
account_id = '1234567890'
start_url = 'https://d-0987654321.awsapps.com/start'
region = 'us-east-1' 
sso_oidc = session.client('sso-oidc')
client_creds = sso_oidc.register_client(
    clientName='myapp',
    clientType='public',
)
device_authorization = sso_oidc.start_device_authorization(
    clientId=client_creds['clientId'],
    clientSecret=client_creds['clientSecret'],
    startUrl=start_url,
)
url = device_authorization['verificationUriComplete']
device_code = device_authorization['deviceCode']
expires_in = device_authorization['expiresIn']
interval = device_authorization['interval']
webbrowser.open(url, autoraise=True)
for n in range(1, expires_in // interval + 1):
    sleep(interval)
    try:
        token = sso_oidc.create_token(
            grantType='urn:ietf:params:oauth:grant-type:device_code',
            deviceCode=device_code,
            clientId=client_creds['clientId'],
            clientSecret=client_creds['clientSecret'],
        )
        break
    except sso_oidc.exceptions.AuthorizationPendingException:
        pass
 
access_token = token['accessToken']
sso = session.client('sso')
account_roles = sso.list_account_roles(
    accessToken=access_token,
    accountId=account_id,
)
roles = account_roles['roleList']
# simplifying here for illustrative purposes
role = roles[0]
role_creds = sso.get_role_credentials(
    roleName=role['roleName'],
    accountId=account_id,
    accessToken=access_token,
)
session = Session(
    region_name=region,
    aws_access_key_id=role_creds['accessKeyId'],
    aws_secret_access_key=role_creds['secretAccessKey'],
    aws_session_token=role_creds['sessionToken'],
)

Answer 5

What works for me is the following:

import boto 3


session = boto3.Session(profile_name="sso_profile_name")
session.resource("whatever")

using boto3==1.20.18.

This would work if you had previously configured SSO for aws ie. aws configure sso.

Interestingly enough, I don't have to go through this if I use ipython, I just aws sso login beforehand and then call boto3.Session(). I am trying to figure out whether there is something wrong with my approach - I fully agree with what was said above with respect to transparency and although it is a working solution, I am not in love with it.

---

EDIT: there was something wrong and here is how I fixed it:

1. run aws configure sso (as above);
2. install aws-vault - it basically replaces aws sso login --profile <profile-name>;
3. run aws-vault exec <profile-name> to create a sub-shell with AWS credentials exported to environment variables.

Doing so, it is possible to run any boto3 command both interactively (eg. iPython) and from a script, as in my case. Therefore, the snippet above simply becomes:

import boto 3


session = boto3.Session()
session.resource("whatever")

Here for further details on AWS vault.