The security token included in the request is expired

Question

I have a script that pulls a lot of metrics from Cloudwatch for our own internal reports.

The script iterates all of the EC2 instances in a specific region and ask for 5 cloudwatch metrics (all the statistics available) for the past 2 weeks (each time 5 days back in 5 minutes interval which is exactly the 1440 quota). I'm using an assumed session:

session = Session(aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY, region_name=regionName)
sts = session.client('sts')
response = sts.assume_role(
    RoleArn=arn, # External role arn
    RoleSessionName='role-name',
    ExternalId='<some-id-here>',
)
tempAccessKeyId = response['Credentials']['AccessKeyId']
tempSecretAccessKey = response['Credentials']['SecretAccessKey']
tempSessionToken = response['Credentials']['SessionToken']
assumedSession = Session(
    aws_access_key_id=tempAccessKeyId,
    aws_secret_access_key=tempSecretAccessKey,
    aws_session_token=tempSessionToken,
    region_name=regionName)

While running the script I got this exception:

botocore.exceptions.ClientError: An error occurred (ExpiredToken) when calling the GetMetricStatistics operation: The security token included in the request is expired

Is there a way to make sure the token doesn't expire while running the script? I'm using boto3.

Answer 1

The assume_role method you are using returns temporary security credentials. The following is taken from the official documentation:

The temporary security credentials are valid for the duration that you specified when calling AssumeRole , which can be from 900 seconds (15 minutes) to 3600 seconds (1 hour). The default is 1 hour.

Since you are not using the DurationSeconds keyword argument, the returned credentials are valid for the default 1 hour. You must make sure to get new credentials in order to make requests after 1 hour. See the following from the Temporary Security Credentials official documentation:

When (or even before) the temporary security credentials expire, the user can request new credentials, as long as the user requesting them still has permissions to do so.

Answer 2

In my case the issue was that, I had credentials in my .aws/configure and was trying to configure from that but what I didn't realize is I had another pair of credentials AWS_SESSION_TOKEN AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY set in environmental variables.

you can do this ( which will remove credentials from environment ).

unset AWS_SESSION_TOKEN AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

Now you will have only one set of access keys i.e in .aws/configure and I was able to make connection sucessfully.

aws configure
aws sts get-caller-identity

if you are using profile other than default, use --profile flag in the above command.

Answer 3

I found AWS Premium Support answer for the issue very concise and relevant

Also important to note that error sometimes comes up when server time is drifted from the correct time (10-15 minutes will cause the error)

Answer 4

I also face this error and even I checked my .aws/credential file. It contains credentials but still I will recommend following step:

before doing anything, first thing you must do run following command

aws sts get-caller-identity

if you have more than one profile in aws credential then use this one

you can check your profile in following file .aws/credential. If you have only [default] then no need to mention profile

aws --profile NAME_OF_YOUR_PROFILE  sts get-caller-identity

Now the question is that above command will solve the problem?

No, But at least it will let you know that your credential are correct or not. I was facing same error and when I run above command it give me following error

An error occurred (SignatureDoesNotMatch) when calling the GetCallerIdentity operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

So at least I find out that I was using wrong credential. I just replace credential and my problem get solved.

Answer 5

Instance time sync issue can also cause this which blocks IAM token refresh, I had tried all the solutions but nothing worked and finally found that its the time difference with instance and AWS services. Installing chrony can help in automatic time sync.