'Authorization errors when starting a Loki pod

This morning I discovered that Loki stopped working in the EKS cluster

In the loki pod logs I see the following:

level=error ts=2022-04-07T10:44:43.298418416Z caller=table_manager.go:233 msg="error syncing tables" err="WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.us-east-2.amazonaws.com/\": dial tcp XXX.XXX.XXX.XXX:443: i/o timeout"
  • ServiceAccount exists and has not changed
  • IAM Role exists and has not changed
  • Dashboards related to Loki and prometheus stopped working (prometheus works with thanos)

Example values ​​file - deployment happens via flux:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: loki
  namespace: loki
spec:
  values:
    extraArgs:
      target: all,table-manager
    serviceAccount: 
      create: true
      name: lokiaccess
      annotations: 
        eks.amazonaws.com/role-arn: arn:aws:iam::<PRIVATE_ID>:role/LokiAccess
        eks.amazonaws.com/sts-regional-endpoints: "true"
    config: 
      storage_config: 
        boltdb_shipper:
          shared_store: s3
        aws: 
          s3: s3://us-east-2/<PRIVATE_STORAGE>
          dynamodb: 
            dynamodb_url: dynamodb://us-east-2
      schema_config:
        configs:
          - from: "2022-04-04"
            store: aws
            object_store: s3
            schema: v11
            index:
              prefix: loki_
              period: 24h
 


Solution 1:[1]

As a result, the problem was in the IAM role for AWS. I changed the settings and added

                "arn:aws:dynamodb:us-east-2:${data.aws_caller_identity.current.account_id}:table/loki_/index/*",
            "arn:aws:dynamodb:us-east-2:${data.aws_caller_identity.current.account_id}:table/loki_",
            "arn:aws:s3:::${var.aws_s3_loki_storage}/*",
            "arn:aws:s3:::${var.aws_s3_loki_storage}"

To our terraform script

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Andrew