'Terminate istio sidecar istio-proxy for a kubernetes job / cronjob

We recently started using istio Istio to establish a service-mesh within out Kubernetes landscape.

We now have the problem that jobs and cronjobs do not terminate and keep running forever if we inject the istio istio-proxy sidecar container into them. The istio-proxy should be injected though to establish proper mTLS connections to the services the job needs to talk to and comply with our security regulations.

I also noticed the open issues within Istio (istio/issues/6324) and kubernetes (kubernetes/issues/25908), but both do not seem to provide a valid solution anytime soon.

At first a pre-stop hook seemed suitable to solve this issue, but there is some confusion about this conecpt itself: kubernetes/issues/55807

lifecycle:
  preStop:
    exec:
      command: 
        ...

Bottomline: Those hooks will not be executed if the the container successfully completed.

There are also some relatively new projects on GitHub trying to solve this with a dedicated controller (which I think is the most preferrable approach), but to our team they do not feel mature enough to put them right away into production:

In the meantime, we ourselves ended up with the following workaround that execs into the sidecar and sends a SIGTERM signal, but only if the main container finished successfully:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: terminate-sidecar-example-service-account
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: terminate-sidecar-example-role
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: terminate-sidecar-example-rolebinding
subjects:
  - kind: ServiceAccount
    name: terminate-sidecar-example-service-account
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: terminate-sidecar-example-role
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: terminate-sidecar-example-cronjob
  labels:
    app: terminate-sidecar-example
spec:
  schedule: "30 2 * * *"
  jobTemplate:
    metadata:
      labels:
        app: terminate-sidecar-example
    spec:
      template:
        metadata:
          labels:
            app: terminate-sidecar-example
          annotations:
            sidecar.istio.io/inject: "true"
        spec:
          serviceAccountName: terminate-sidecar-example-service-account
          containers:
          - name: ****
            image: ****
            command:
              - "/bin/ash"
              - "-c"
            args:
              - node index.js && kubectl exec -n ${POD_NAMESPACE} ${POD_NAME} -c istio-proxy -- bash -c "sleep 5 && /bin/kill -s TERM 1 &"
            env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace

So, the ultimate question to all of you is: Do you know of any better workaround, solution, controller, ... that would be less hacky / more suitable to terminate the istio-proxy container once the main container finished its work?

kubernetesistio


Solution 1:[1]

- command:
  - /bin/sh
  - -c
  - |
    until curl -fsI http://localhost:15021/healthz/ready; do echo \"Waiting for Sidecar...\"; sleep 3; done;
    echo \"Sidecar available. Running the command...\";
    <YOUR_COMMAND>;
    x=$(echo $?); curl -fsI -X POST http://localhost:15020/quitquitquit && exit $x

Update: sleep loop can be omitted if holdApplicationUntilProxyStarts is set to true (globally or as an annotation) starting with istio 1.7

Solution 2:[2]

This was not a misconfiguration, this was a bug in upstream Kubernetes. As of September of 2019, this has been resolved by Istio by introducing a /quitquitquit endpoint to the Pilot agent.

Unfortunately, Kubernetes has not been so steadfast in solving this issue themselves. So it still does exist in some facets. However, the /quitquitquit endpoint in Istio should have resolved the problem for this specific use case.

Solution 3:[3]

I have found a work around by editing the configmap of istio-sidecar-injector as per the link Istio documentation

https://istio.io/docs/setup/additional-setup/sidecar-injection/

apiVersion: v1 kind: ConfigMap metadata: name: istio-sidecar-injector data: config: |- policy: enabled neverInjectSelector: - matchExpressions: - {key: job-name, operator: Exists} But with this changes in our cronjob sidecar will not inject and istio policy will not apply on the cronjob job, and in our case we dont want any policy to be enforced by istio

Note :- job-name is by default label gets added in the pod creation

Solution 4:[4]

For those for whom curl is a luxury my wget version of the Dimitri's code:

command:
  - /bin/sh
  - -c
  - |
    until wget -q --spider http://127.0.0.1:15021/healthz/ready 2>/dev/null; do echo "Waiting for Istio sidecar..."; sleep 3; done;
    echo \"Sidecar available. Running...\";
    <COMMAND>;
    x=$?; wget -q --post-data='' -S -O /dev/null http://127.0.0.1:15020/quitquitquit && exit $x

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 TJ Zimmerman
Solution 3 Ankit Saxena
Solution 4

Related Questions