kube proxy servcie dead unexpectly in kubenetes host machine

Question

My pod give me tips: could not resolve host:mybatis.org. I found the kubernete's(v1.15.2) pods connect to kube DNS server failed,and now I found the failed host's kube-proxy service was down :

[root@ops001 ~]# systemctl status kube-proxy
● kube-proxy.service - Kubernetes Kube-Proxy Server
   Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Sun 2020-03-01 17:10:36 CST; 56min ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 24908 ExecStart=/opt/k8s/bin/kube-proxy --config=/etc/kubernetes/kube-proxy-config.yaml --logtostderr=true --v=2 (code=killed, signal=PIPE)
 Main PID: 24908 (code=killed, signal=PIPE)

Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729068   24908 graceful_termination.go:175] Deleting rs: 10.254.81.136:11005/TCP/172.30.224.27:11005
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729482   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:53/UDP/172.30.224.22:53
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729526   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:53/UDP/172.30.224.22:53
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729677   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:9153/TCP/172.30.224.22:9153
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729707   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:9153/TCP/172.30.224.22:9153
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729859   24908 graceful_termination.go:161] Trying to delete rs: 10.254.230.119:11006/TCP/172.30.224.7:11006
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729893   24908 graceful_termination.go:175] Deleting rs: 10.254.230.119:11006/TCP/172.30.224.7:11006
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.730331   24908 graceful_termination.go:161] Trying to delete rs: 10.254.230.119:12001/TCP/172.30.224.7:12001
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.730364   24908 graceful_termination.go:175] Deleting rs: 10.254.230.119:12001/TCP/172.30.224.7:12001
Mar 01 17:05:23 ops001.example.com kube-proxy[24908]: I0301 17:05:23.357479   24908 proxier.go:762] Stale udp service kube-system/kube-dns:dns -> 10.254.0.2

and I could not found any error messagae tell me why the server is down.I am check the log:

    [root@ops001 ~]# journalctl -f -n 50 -u kube-proxy

Mar 01 16:59:46 ops001.example.com kube-proxy[24908]: I0301 16:59:45.908337   24908 graceful_termination.go:161] Trying to delete rs: 10.254.207.10:11002/TCP/172.30.224.25:11002
Mar 01 16:59:46 ops001.example.com kube-proxy[24908]: I0301 16:59:45.908372   24908 graceful_termination.go:175] Deleting rs: 10.254.207.10:11002/TCP/172.30.224.25:11002
Mar 01 16:59:46 ops001.example.com kube-proxy[24908]: I0301 16:59:45.908559   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:53/TCP/172.30.224.22:53
Mar 01 16:59:46 ops001.example.com kube-proxy[24908]: I0301 16:59:45.908588   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:53/TCP/172.30.224.22:53
Mar 01 16:59:46 ops001.example.com kube-proxy[24908]: I0301 16:59:45.909726   24908 graceful_termination.go:161] Trying to delete rs: 10.254.252.37:11001/TCP/172.30.224.5:11001
Mar 01 16:59:46 ops001.example.com kube-proxy[24908]: I0301 16:59:45.909758   24908 graceful_termination.go:175] Deleting rs: 10.254.252.37:11001/TCP/172.30.224.5:11001
Mar 01 16:59:53 ops001.example.com kube-proxy[24908]: I0301 16:59:53.550741   24908 proxier.go:762] Stale udp service kube-system/kube-dns:dns -> 10.254.0.2
Mar 01 17:01:47 ops001.example.com kube-proxy[24908]: I0301 17:01:11.420580   24908 trace.go:81] Trace[984210025]: "iptables save" (started: 2020-03-01 17:00:54.619332757 +0800 CST m=+1345.889061119) (total time: 6.798183074s):
Mar 01 17:02:44 ops001.example.com kube-proxy[24908]: Trace[984210025]: [6.798183074s] [6.798183074s] END
Mar 01 17:02:45 ops001.example.com kube-proxy[24908]: I0301 17:01:51.318346   24908 trace.go:81] Trace[383010503]: "iptables save" (started: 2020-03-01 17:01:16.218996646 +0800 CST m=+1367.488724918) (total time: 31.900207714s):
Mar 01 17:02:45 ops001.example.com kube-proxy[24908]: Trace[383010503]: [31.900207714s] [31.900207714s] END
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.124380   24908 graceful_termination.go:161] Trying to delete rs: 10.254.149.90:11007/TCP/172.30.224.28:11007
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.124435   24908 graceful_termination.go:175] Deleting rs: 10.254.149.90:11007/TCP/172.30.224.28:11007
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.125333   24908 graceful_termination.go:161] Trying to delete rs: 10.254.230.119:11006/TCP/172.30.224.7:11006
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.125364   24908 graceful_termination.go:175] Deleting rs: 10.254.230.119:11006/TCP/172.30.224.7:11006
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.125517   24908 graceful_termination.go:161] Trying to delete rs: 10.254.124.127:11016/TCP/172.30.224.6:11016

    -- Logs begin at Sat 2019-12-28 20:58:38 CST. --
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.126134   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:53/UDP/172.30.224.22:53
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.126164   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:53/UDP/172.30.224.22:53
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.148160   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:9153/TCP/172.30.224.22:9153
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.148243   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:9153/TCP/172.30.224.22:9153
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.148822   24908 graceful_termination.go:161] Trying to delete rs: 10.254.230.119:12001/TCP/172.30.224.7:12001
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.148864   24908 graceful_termination.go:175] Deleting rs: 10.254.230.119:12001/TCP/172.30.224.7:12001
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.149466   24908 graceful_termination.go:161] Trying to delete rs: 10.254.207.10:11002/TCP/172.30.224.25:11002
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.149510   24908 graceful_termination.go:175] Deleting rs: 10.254.207.10:11002/TCP/172.30.224.25:11002
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.149673   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:53/TCP/172.30.224.22:53
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.149707   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:53/TCP/172.30.224.22:53
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.151003   24908 graceful_termination.go:161] Trying to delete rs: 10.254.252.37:11001/TCP/172.30.224.5:11001
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.151076   24908 graceful_termination.go:175] Deleting rs: 10.254.252.37:11001/TCP/172.30.224.5:11001
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.152403   24908 graceful_termination.go:161] Trying to delete rs: 10.254.51.192:11017/TCP/172.30.224.3:11017
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.152435   24908 graceful_termination.go:175] Deleting rs: 10.254.51.192:11017/TCP/172.30.224.3:11017
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.152594   24908 graceful_termination.go:161] Trying to delete rs: 10.254.172.133:11014/TCP/172.30.224.24:11014
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.152623   24908 graceful_termination.go:175] Deleting rs: 10.254.172.133:11014/TCP/172.30.224.24:11014
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.152780   24908 graceful_termination.go:161] Trying to delete rs: 10.254.234.135:8088/TCP/172.30.224.29:8088
Mar 01 17:02:46 ops001.example.com kube-proxy[24908]: I0301 17:02:46.152808   24908 graceful_termination.go:175] Deleting rs: 10.254.234.135:8088/TCP/172.30.224.29:8088
Mar 01 17:02:53 ops001.example.com kube-proxy[24908]: I0301 17:02:53.468216   24908 proxier.go:762] Stale udp service kube-system/kube-dns:dns -> 10.254.0.2
Mar 01 17:03:45 ops001.example.com kube-proxy[24908]: I0301 17:03:38.441122   24908 trace.go:81] Trace[1879196474]: "iptables save" (started: 2020-03-01 17:03:25.839624749 +0800 CST m=+1497.109352986) (total time: 10.104547281s):
Mar 01 17:04:35 ops001.example.com kube-proxy[24908]: Trace[1879196474]: [10.104547281s] [10.104547281s] END
Mar 01 17:04:45 ops001.example.com kube-proxy[24908]: I0301 17:04:00.041791   24908 trace.go:81] Trace[577968152]: "iptables save" (started: 2020-03-01 17:03:38.738090492 +0800 CST m=+1510.007818688) (total time: 15.405784415s):
Mar 01 17:04:45 ops001.example.com kube-proxy[24908]: Trace[577968152]: [15.405784415s] [15.405784415s] END
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.724742   24908 graceful_termination.go:161] Trying to delete rs: 10.254.207.10:11002/TCP/172.30.224.25:11002
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.724794   24908 graceful_termination.go:175] Deleting rs: 10.254.207.10:11002/TCP/172.30.224.25:11002
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.724954   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:53/TCP/172.30.224.22:53
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.724985   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:53/TCP/172.30.224.22:53
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.726324   24908 graceful_termination.go:161] Trying to delete rs: 10.254.252.37:11001/TCP/172.30.224.5:11001
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.726378   24908 graceful_termination.go:175] Deleting rs: 10.254.252.37:11001/TCP/172.30.224.5:11001
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.727471   24908 graceful_termination.go:161] Trying to delete rs: 10.254.51.192:11017/TCP/172.30.224.3:11017
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.727520   24908 graceful_termination.go:175] Deleting rs: 10.254.51.192:11017/TCP/172.30.224.3:11017
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.727711   24908 graceful_termination.go:161] Trying to delete rs: 10.254.172.133:11014/TCP/172.30.224.24:11014
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.727742   24908 graceful_termination.go:175] Deleting rs: 10.254.172.133:11014/TCP/172.30.224.24:11014
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.727915   24908 graceful_termination.go:161] Trying to delete rs: 10.254.234.135:8088/TCP/172.30.224.29:8088
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.727946   24908 graceful_termination.go:175] Deleting rs: 10.254.234.135:8088/TCP/172.30.224.29:8088
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.728614   24908 graceful_termination.go:161] Trying to delete rs: 10.254.149.90:11007/TCP/172.30.224.28:11007
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.728647   24908 graceful_termination.go:175] Deleting rs: 10.254.149.90:11007/TCP/172.30.224.28:11007
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.728819   24908 graceful_termination.go:161] Trying to delete rs: 10.254.124.127:11016/TCP/172.30.224.6:11016
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.728848   24908 graceful_termination.go:175] Deleting rs: 10.254.124.127:11016/TCP/172.30.224.6:11016
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729003   24908 graceful_termination.go:161] Trying to delete rs: 10.254.81.136:11005/TCP/172.30.224.27:11005
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729068   24908 graceful_termination.go:175] Deleting rs: 10.254.81.136:11005/TCP/172.30.224.27:11005
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729482   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:53/UDP/172.30.224.22:53
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729526   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:53/UDP/172.30.224.22:53
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729677   24908 graceful_termination.go:161] Trying to delete rs: 10.254.0.2:9153/TCP/172.30.224.22:9153
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729707   24908 graceful_termination.go:175] Deleting rs: 10.254.0.2:9153/TCP/172.30.224.22:9153
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729859   24908 graceful_termination.go:161] Trying to delete rs: 10.254.230.119:11006/TCP/172.30.224.7:11006
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.729893   24908 graceful_termination.go:175] Deleting rs: 10.254.230.119:11006/TCP/172.30.224.7:11006
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.730331   24908 graceful_termination.go:161] Trying to delete rs: 10.254.230.119:12001/TCP/172.30.224.7:12001
Mar 01 17:04:47 ops001.example.com kube-proxy[24908]: I0301 17:04:47.730364   24908 graceful_termination.go:175] Deleting rs: 10.254.230.119:12001/TCP/172.30.224.7:12001
Mar 01 17:05:23 ops001.example.com kube-proxy[24908]: I0301 17:05:23.357479   24908 proxier.go:762] Stale udp service kube-system/kube-dns:dns -> 10.254.0.2

the question is: how to find why the kube-proxy servcie is down?

Answer 1

"code=killed, signal=PIPE" it maybe break by write pipe error signal.....

Add "RestartForceExitStatus=SIGPIPE" to service, force restart it !