'Correct way to update rules and configuration for a Prometheus installation on a Kubernetes cluster that was setup by prometheus-operator helm chart?
I have configured Prometheus via helm chart https://github.com/helm/charts/tree/master/stable/prometheus-operator I need to update Prometheus rules and configuration on the fly (hot-reload) but for some reason, it's not working.
I tried to edit the cofigmap for Prometheus rules but the changes are reverted as soon as I save the changes. I have tried deleting the Prometheus configmap, it get's regenerated instantly. I have also attempted to delete Prometheus pod itself but it also gets recreated instantly.
If the custom controller and CRDs are the reason behind this(i.e. rejecting the changes), what's the correct approach to do this?
Solution 1:[1]
Solution 2:[2]
Not sure if this is relevant to this deployment mode. But we use the prometheus operator and there is a CRD for prometheus you can edit like this with the various settings in it:
kubectl edit prometheus -n monitoring prom-operator-kube-prometh-prometheus
This assumes the namespace is monitoring and the name is prom-operator-kube-prometh-prometheus. You can also list them though with get prometheus or whatever to help track it down.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | evgenyl |
| Solution 2 | John Humphreys |