Inject cronjob once namespace created

Question

This mainly for GKE cluster with preemptible nodes. Google Cloud would shutdown any node without warning, which leaves lots of pods with Terminated and / or NodeShutdown state. My current UAT environment is a GKE cluster with hundreds of tenants, where each tenant occupies one namespace. Delete all pods with Terminated and / or NodeShutdown state (means across namespace) would take a very time and not really practical as single run, so I think it might be better for each namespace running its own 'janitor' Cronjob. Now, I could create Cronjob manually with existing namespaces, but I can't stop my colleagues to create other namespaces for their tenants. So I start to think of if I could do similar thing just like Istio auto sidecar injection.

Just curious anybody know what documents / github repo I should dig from here. Or, if what I thought might be totally wrong, please point a correct way to solve this problem.

Answer 1

so I think it might be better for each namespace running its own 'janitor' Cronjob. Now, I could create Cronjob manually with existing namespaces, but I can't stop my colleagues to create other namespaces for their tenants. So I start to think of if I could do similar thing just like Istio auto sidecar injection.

You can write your own CRD or controller to inject Job or Cronjob to the newly created namespace by default.

Anyone who creates a new namespace, it will have this cronjob(janitor) running for cleaning up.

i was looking into this CRD or controller daemonjob which will run on each of Node as JOB. Maybe you could take as base and update it further to implement your scenario.

https://github.com/AmitKumarDas/metac/blob/master/examples/daemonjob/sync.py