The etcdserver throws warnings of request 'took to long'

Question

I have an MKE cluster with three manager node. The etcd is giving the following warnings

2021-12-06 10:03:10.624622 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:8" took too long (157.50303ms) to execute
2021-12-06 10:03:24.987502 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:442" took too long (127.110289ms) to execute
2021-12-06 10:03:26.302979 W | etcdserver: request "header:<ID:16726225580315543268 username:\"system:ucp:v2d7rih1ql0sztk06dnu37a58\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:352278653 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:218 >> failure:<>>" with result "size:22" took too long (101.390612ms) to execute
2021-12-06 10:03:26.498919 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:8" took too long (141.572923ms) to execute
2021-12-06 10:03:37.589988 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:296" took too long (141.997165ms) to execute
2021-12-06 10:04:03.211705 W | etcdserver: read-only range request "key:\"/registry/networkpolicies/\" range_end:\"/registry/networkpolicies0\" count_only:true " with result "range_response_count:0 size:8" took too long (108.336554ms) to execute
2021-12-06 10:04:08.707179 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:8" took too long (102.900861ms) to execute
2021-12-06 10:04:09.145791 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:296" took too long (253.282239ms) to execute
2021-12-06 10:04:09.834741 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:460" took too long (149.503672ms) to execute
2021-12-06 10:04:25.053962 W | etcdserver: read-only range request "key:\"/registry/persistentvolumeclaims/tme-ns-cms-prd/tme-pvc-tmeestore\" " with result "range_response_count:1 size:1201" took too long (209.981256ms) to execute
2021-12-06 10:05:30.778453 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:460" took too long (253.111751ms) to execute

We already increased the CPUs and memory on these three manager nodes, without any impact.

I suppose that disk IO is very important and can result in these warnings if we have to high latency?

Can network between the three etcd instances cause this too? Or are these requests pure locally ?

Answer 1

"took too long (108.336554ms) " is trigger by default 100ms. this is disk performance issue. if you use etcd v3.4.x,you can see param to tune the limit: config parameter experimental-apply-warning-duration

more info see here: https://github.com/etcd-io/etcd/issues/10860