kubectl error You must be logged in to the server (Unauthorized) when accessing EKS cluster

Question

I have been trying to follow the getting started guide to EKS. When I tried to call kubectl get service I got the message: error: You must be logged in to the server (Unauthorized) Here is what I did:
1. Created the EKS cluster.
2. Created the config file as follows:

apiVersion: v1
clusters:
- cluster:
    server: https://*********.yl4.us-west-2.eks.amazonaws.com
    certificate-authority-data: *********
  name: *********
contexts:
- context:
    cluster: *********
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"
        - "*********"
        - "-r"
        - "arn:aws:iam::*****:role/******"

3. Downloaded and installed latest aws cli
4. Ran aws configure and set the credentials for my IAM user and the region as us-west-2
5. Added a policy to the IAM user for sts:AssumeRole for the EKS role and set it up as a trusted relationship
6. Setup kubectl to use the config file

I can get a token when I run heptio-authenticator-aws token -r arn:aws:iam::**********:role/********* -i my-cluster-ame However when I try to access the cluster I keep receiving error: You must be logged in to the server (Unauthorized)

Any idea how to fix this issue?

Answer 1

When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl.

eks-docs

So to add access to other aws users, first you must edit ConfigMap to add an IAM user or role to an Amazon EKS cluster.

You can edit the ConfigMap file by executing: kubectl edit -n kube-system configmap/aws-auth, after which you will be granted with editor with which you map new users.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers: |
    - userarn: arn:aws:iam::111122223333:user/ops-user
      username: ops-user
      groups:
        - system:masters
  mapAccounts: |
    - "111122223333"

Pay close attention to the mapUsers where you're adding ops-user together with mapAccounts label which maps the AWS user account with a username on Kubernetes cluster.

However, no permissions are provided in RBAC by this action alone; you must still create role bindings in your cluster to provide these entities permissions.

As the amazon documentation(iam-docs) states you need to create a role binding on the kubernetes cluster for the user specified in the ConfigMap. You can do that by executing following command (kub-docs):

kubectl create clusterrolebinding ops-user-cluster-admin-binding --clusterrole=cluster-admin --user=ops-user

which grants the cluster-admin ClusterRole to a user named ops-user across the entire cluster.

Answer 2

I am sure issue is resolved but I will be putting more information here so if any other people are still facing the issue related to any of the below setup then they can use the steps below.

When we create the EKS cluster by any method via CloudFormation/CLI/EKSCTL the IAM role/user who created the cluster will automatically binded to the default kubernetes RBAC API group system:masters (https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) and in this way creator of the cluster will get the admin access to the cluster. Although we can always give the access to other IAM user/role using the aws-auth file but for that we must have to use the IAM user/role who created the cluster.

To verify the role/user for the EKS cluster we can search for the CreateCluster" Api call on cloudtrail and it will tell us the creator of the cluster in the sessionIssuer section for field arn (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).

When we create the cluster using the IAM role or IAM user, setting up the access for the EKS cluster will become little tricky when we created the cluster using the role compare to user.

I will put the steps we can follow for each different method while setting up the access to EKS cluster.

Scenario-1: Cluster was Created using the IAM user (For example "eks-user")

---

Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity

$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

Scenario-2: Cluster was Created using the IAM Role (For example "eks-role")

---

Mainly there are four different way to setup the access via cli when cluster was created via IAM role.

1. Setting up the role directly in kubeconfig file.

In this case we do not have to make any assume role api call via cli manually, before running kubectl command because that will be automatically done by aws/aws-iam-authenticator set in the kube config file.

Lets say now we are trying to setup the access for the user eks-user the first make sure that user does have permission to assume the role eks-role

Add the assume role permission to the eks-user

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::xxxxxxxxxxx:role/eks-role"
        }
    ]
}

Edit the trust relationship on the role so that it will allow the eks-user to assume the role.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity. Important thing to remember it should show us the IAM user ARN not the IAM assumed ROLE ARN.

$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name --role-arn arn:aws:iam::xxxxxxxxxxx:user/eks-role

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      - --role
      - arn:aws:iam::xxxxxxx:role/eks-role
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

2. If you have setup the AWS profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) on CLI and if you want to use that with the kube config.

Confirm that profile is set properly so that it can use the credentials for the eks-user

 $ cat ~/.aws/config
[default]
output = json
region = us-east-1
[eks]
output = json
region = us-east-1
[profile adminrole]
role_arn = arn:aws:iam::############:role/eks-role
source_profile = eks

 $ cat ~/.aws/credentials
[default]
aws_access_key_id = xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[eks]
aws_access_key_id =  xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Once this profile configuration is done please confirm that profile configuration is fine by running the command aws sts get-caller-identity --profile eks

$ aws sts get-caller-identity --profile eks
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command with the profile and please make sure we are not using the role here.

aws eks update-kubeconfig --name devel --profile eks

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws
      env:
      - name: AWS_PROFILE
        value: eks

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

3. Assume the role by any other way, For example we can attach the IAM role to the instance directly.

If role is directly attached to the instance profile then we can follow the similar steps as we followed while setting up the access for IAM user in Scenario-1

Verify that we have attached the correct role to EC2 instance and as this instance profile will come into least precedence, this step will also verify that there are no any other credentials setup on the instnace.

[ec2-user@ip-xx-xxx-xx-252 ~]$ aws sts get-caller-identity
{
    "Account": "xxxxxxxxxxxx",
    "UserId": "xxxxxxxxxxxxxxxxxxxxx:i-xxxxxxxxxxx",
    "Arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/eks-role/i-xxxxxxxxxxx"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

4. Manually assuming the IAM role via aws sts assume-role command.

Assume the role eks-role manually by running the cli command.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxxxxxx:role/eks-role --role-session-name test

{
    "AssumedRoleUser": {
        "AssumedRoleId": "xxxxxxxxxxxxxxxxxxxx:test",
        "Arn": "arn:aws:sts::xxxxxxxxxxx:assumed-role/eks-role/test"
    },
    "Credentials": {
        "SecretAccessKey": "xxxxxxxxxx",
        "SessionToken": xxxxxxxxxxx",
        "Expiration": "xxxxxxxxx",
        "AccessKeyId": "xxxxxxxxxx"
    }
}

After that set the required environment variable using the value from above output so that we can use the correct credentials generated from the session.

export AWS_ACCESS_KEY_ID=xxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx
export AWS_SESSION_TOKEN=xxxxxxxxxx

After that verify that we assumed the IAM role by running the command aws sts get-caller-identity.

aws sts get-caller-identity

{
    "Account": "xxxxxxxxxx",
    "UserId": "xxxxxxxxxx:test",
    "Arn": "arn:aws:sts::xxxxxxxxxx:assumed-role/eks-role/test"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

---

NOTE:

I have try to cover major use case here but there might be other use case too where we need to setup the access to the cluster.

Also the above tests are mainly aiming at the first time setup of the EKS cluster and none of the above method is touching the aws-auth configmap yet. But once you have given access to other IAM user/role to EKS cluster via aws-auth (https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) file you can use the same set of commands for those users too as mentioned in above answer.

Update: If you are using the SSO then setup will be preety much same but one thing we have to consider is either in case of SSO or while using the role directly if we are trying to update path based role in ConfigMap then we have to get rid of the paths in role for example instead arn:aws:iam::xxxxxxxxxxx:role/path-1/subpath-1/eks-role of this use arn:aws:iam::xxxxxxxxxxx:role/eks-role so basically we are getting rid of the /path-1/subpath-1 becuase when we run kubectl command it will first make AssumeRole api call and if we see assumed role ARN then it will not contians the path so if we include the path then it EKS will deny those requests.

Answer 3

If you are using eksctl to manage your aws eks deployments you can add the user to the config map with one command:

eksctl create iamidentitymapping --cluster <cluster-name> --arn arn:aws:iam::<id>:user/<user-name> --group system:masters --username ops-user

Answer 4

I commented out the last two lines of the config file

# - "-r"
# - "arn:aws:iam::**********:role/**********"

and it worked though I have no idea why

Answer 5

You need to create the cluster under the same IAM profile that you are accessing it from via AWS cli.

Said in another way, inside ~/.aws/credentials, the profile that is accessing kubectl must match exactly the same IAM that was used to create the cluster.

My recommendation is to use AWS cli to create your clusters as creating from the GUI may be more confusing than helpful. The Getting Started guide is your best bet to get up and running.

Answer 6

Also, make sure your users are in the aws-auth k8s ConfigMap:

https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Answer 7

I had the same problem . It's likely that you are using a root account. It appears root accounts are blocked from assuming the required roles. This error can sometimes be cloaked if you are using expired keys.

Answer 8

I had the same problem, my AWS credentials for CLI change frequently. These steps fixed the problem:

export AWS_ACCESS_KEY_ID="***************"
export AWS_SECRET_ACCESS_KEY="*************"
export AWS_SESSION_TOKEN="************************"

Answer 9

I just debugged this issue. I have a question. Are you running this on a corporate wifi network? If yes, could you create an EC2 instance and then test if you are able to do kubectl get svc?

Also, try if this command works kubectl get svc --insecure-skip-tls-verify

Answer 10

The issue for me was that I had set up environment variables with different, invalid, AWS credentials (maybe a long time ago, and forgot). I realised this after running aws configure list and seeing that the credentials are different from what I expected with aws configure list --profile default. Finding and deleting those invalid environment variables fixed the issue, now I can run kubectl get svc.

Answer 11

If you have created the EKS cluster with kops, Then all you need to do is update your kubecfig file with following kops command

kops export kubecfg --admin

Answer 12

This happens also to me with local environment on minikube, independently of EKS. My problem is related to this issue: https://github.com/kubernetes/kubernetes/issues/76774

The solution i adopted is to remove the cache directories of kubectl: rm -rf ~/.kube/{cache,http-cache}. I guess is the only workaround at the time of writing.

Answer 13

In my case it is the AWS profile issue, be sure to use aws sts get-caller-identity to verify the IAM user.

Answer 14

The issue is with the policy added for the roles created. We must have AWSEKSCNI policy.

Better to create eks using the command:

eksctl create cluster --name ekscluster --version 1.19 --with-oidc
--vpc-public-subnets=subnet-08c6b0b0166abc1d1,subnet-02822a142bb5a802a
--vpc-private-subnets=subnet-09bbf4871902ee64c,subnet-0926c224909b5a811

This will create and assign the policy automatically using cloudformation.

Answer 15

If you have exhausted all of the above solutions and are still getting the same error. Make sure kubectl is actually using the AWS credentials you think you are.

I made a very stupid mistake. I have environment variables setup for different accounts and kubectl seems to always pick up the one from AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. So precedence of options on this page may help https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

Answer 16

I got this error when I created the eks cluster using the root from the eks console. I recreated the eks cluster using an IAM user and use the access keys to update the aws configure. It worked. Now you can add additional IAM users to issue kubectl commands.

Answer 17

I was trying to create an EKS cluster with a private endpoint. Read this thread several times and the thing that worked for me:

• Created a user: admin (with access to the console)
• Logged into the console using that user
• Created the cluster using the admin user
• Created my kube/config file using aws eks (aws eks --region update-kubeconfig --name )
• Done (kubectl get ns)

The last 2 commands were executed from an ec2 instance part of the same VPC.

Answer 18

I faced the same issue. I tried to configure AWS CLI directly with access key and secret key and it worked.

It should be bug to not able to assume role. Try setup cli and test.

Answer 19

In my case the scenario was, that I replaced a person who worked as a devops and created all the infrastructure on aws. He had his own IAM user (so yes, no root access case, however could be also applicable) with all AWS permission and I had my own IAM user also with all available AWS permissions. So, he left the company and that was the problem, since I didn't get any access to cluster which by defauly is shared to the creator only ... and in fact all of my approaches to get the access to cluster were failed, even despite a fact that I had all permissions. The good thing was, guy whom I replaced was still had his IAM user available (not removed). And what I did, I simply generated new AWS access pair under his account and set them as my default aws credentials on my ubuntu host (from which I've tried to access the cluster). Important part was to make sure that after running aws sts get-caller-identity command it meant to be HIS account to appear on the output. In that case I've been able to run all the kubectl commands that I wanted without error You must be logged in to the server (Unauthorized) message. So the solution is - be lucky to find cluster creator credentials and use them! (sounds like a crime, however ...)

Answer 20

I had the same issue. Refer the answers:

1. Cannot create namespaces in AWS Elastic Kubernetes Service - Forbidden
2. User cannot log into EKS Cluster using kubectl

The correct way is to:

1. Create an IAM Group with the following permissions.
   1. AmazonEKSClusterPolicy
   2. AmazonEKSWorkerNodePolicy
   3. AmazonEC2ContainerRegistryReadOnly
   4. AmazonEKS_CNI_Policy
   5. AmazonElasticContainerRegistryPublicReadOnly
   6. EC2InstanceProfileForImageBuilderECRContainerBuilds
   7. AmazonElasticContainerRegistryPublicFullAccess
   8. AWSAppRunnerServicePolicyForECRAccess
   9. AmazonElasticContainerRegistryPublicPowerUser
   10. SecretsManagerReadWrite

2. Create an IAM user (AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID will be provided) and add the user to the IAM Group created above.

3. Next, login to AWS Console as the IAM user and create the EKS Cluster.

4. Next, use the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID to setup the AWS CLI in local machine.

5. Next, run the following commands in the local machine:
   1. aws sts get-caller-identity
   2. aws eks describe-cluster --name [cluster-name] --region [aws-region] --query cluster.status (To check the status of the Cluster)
   3. aws eks update-kubeconfig --name [cluster-name] --region [aws-region]

6. After this, you will be able to run the kubectl commands.

If you need to add additional users to the EKS Cluster, create the additional IAM user, add the user to the IAM Group in AWS. Next, log into the EKS cluster as the original IAM user and run: kubectl edit -n kube-system configmap/aws-auth.

Add the following block of code to the existing configuration:

(Make sure to add changes to the ARN and username)

mapUsers: |
- userarn: arn:aws:iam::111122223333:user/admin
  username: admin
  groups:
    - system:masters

Refer the following link to understand this better: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Hope this become useful. Happy learning!

Answer 21

I followed these docs. It took some time to understand things. But finally implemented the things easily with full understanding.

https://kubernetes.io/docs/reference/access-authn-authz/rbac/ https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Answer 22

When invoking eksctl with an assumed role through sso, the following steps got me access to the cluster.

1. eksctl.exe create cluster --name test --without-nodegroup --profile <your_profile>.
2. Update the kubeconfig :

eksctl utils write-kubeconfig --cluster=test --profile<your_profile>.

3. Enable Cloudwatch to get the iam role arn:

eksctl utils update-cluster-logging --enable-types=all --cluster=test --approve --profile=<your_profile>.

4. run "kubectl get pods -A" . server responds with "error: You must be logged in to the server (Unauthorized)" message.
5. Switch to the console and get the role arn from the cloudwatch group audit log.
6. Update the aws-auth config map by mapping the derived arn to system:masters group. The username is users:name field in your current kubeconfig context as extracted it below

user_id=` k config view --minify -o json | jq -r '.users[0].name'` 
eksctl create iamidentitymapping --cluster test --group system:masters --username ${user_id} --arn arn:aws:iam::<account_id>:role/<assumed_role> --profile <your_profile>

Answer 23

In addition to the great answers that have already been given, I would like to add a good way to troubleshoot issues. In my case, I was trying to run kubectl in an ECS task as part of an AWS pipeline, and kubectl version was failing with the "You must be logged in to the server" message.

What was happening was that the ECS task was assuming a service role, and then aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_DEFAULT_REGION was being run. It turns out the service role, CodeBuildServiceRole, was not mapped to an RBAC user via a clusterrolebinding in the EKS cluster, and the aws-iam-authenticator EKS service was denying access to the AWS service account (or something like that).

What helped me figure out what was going on in this case was to go to CloudWatch ? Log Insights and run the following query against the /aws/eks/<cluster-name>/cluster log group:

fields @timestamp, @message
| sort @timestamp desc
| filter @logStream like /authenticator/
| filter @message like /access denied/
| limit 50

It was clear from the log output that the authenticator service was expecting an ARN in configmap/aws-auth that was lowercase: arn:aws:iam::123456789:role/codebuildservicerole. Once I fixed the case of the ARN in the configmap, kubectl version started working in the ECS task.

$ kubectl edit -n kube-system configmap/aws-auth

Answer 24

For me adding the user in a single line like below worked

kubectl edit -n kube-system configmap/aws-auth

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::******:role/eksctl-atoa-microservices-nodegro-NodeInstanceRole-346C48Q1W7OB
      username: system:node:{{EC2PrivateDNSName}}
  mapUsers: "- groups:\n  - system:masters\n  userarn: arn:aws:iam::*****:user/<username>
    \ \n  username: <username>\n"