'etcd.service failed because a timeout was exceeded
I am working through "learn kubernetes the hard way" and am at the "bootstrapping the etcd cluster" step: https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/07-bootstrapping-etcd.md
I have run into a command that causes a timeout:
somersbmatthews@controller-0:~$ { sudo systemctl daemon-reload; sudo systemctl enable etcd; sudo systemctl start etcd; }
Job for etcd.service failed because a timeout was exceeded.
See "systemctl status etcd.service" and "journalctl -xe" for details.
Here I follow the above recommendations:
This is the first thing the CLI asked me to check:
somersbmatthews@controller-0:~$ systemctl status etcd.service
● etcd.service - etcd
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: activating (start) since Wed 2020-12-02 03:15:05 UTC; 34s ago
Docs: https://github.com/coreos
Main PID: 49251 (etcd)
Tasks: 8 (limit: 9544)
Memory: 10.2M
CGroup: /system.slice/etcd.service
└─49251 /usr/local/bin/etcd --name controller-0 --cert-file=/etc/etcd/kubernetes.pem --key-file=/etc/etcd/kubernetes-key.pem --peer-cert-file>
Dec 02 03:15:38 controller-0 etcd[49251]: raft2020/12/02 03:15:38 INFO: f98dc20bce6225a0 is starting a new election at term 570
Dec 02 03:15:38 controller-0 etcd[49251]: raft2020/12/02 03:15:38 INFO: f98dc20bce6225a0 became candidate at term 571
Dec 02 03:15:38 controller-0 etcd[49251]: raft2020/12/02 03:15:38 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 571
Dec 02 03:15:38 controller-0 etcd[49251]: raft2020/12/02 03:15:38 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:15:38 controller-0 etcd[49251]: raft2020/12/02 03:15:38 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:15:39 controller-0 etcd[49251]: raft2020/12/02 03:15:39 INFO: f98dc20bce6225a0 is starting a new election at term 571
Dec 02 03:15:39 controller-0 etcd[49251]: raft2020/12/02 03:15:39 INFO: f98dc20bce6225a0 became candidate at term 572
Dec 02 03:15:39 controller-0 etcd[49251]: raft2020/12/02 03:15:39 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 572
Dec 02 03:15:39 controller-0 etcd[49251]: raft2020/12/02 03:15:39 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:15:39 controller-0 etcd[49251]: raft2020/12/02 03:15:39 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
The second thing the CLI asked me to check:
somersbmatthews@controller-0:~$ journalctl -xe
-- A stop job for unit etcd.service has finished.
--
-- The job identifier is 3597 and the job result is done.
Dec 02 03:05:32 controller-0 systemd[1]: Starting etcd...
-- Subject: A start job for unit etcd.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit etcd.service has begun execution.
--
-- The job identifier is 3597.
Dec 02 03:05:32 controller-0 etcd[48861]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Dec 02 03:05:32 controller-0 etcd[48861]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Dec 02 03:05:32 controller-0 etcd[48861]: etcd Version: 3.4.10
Dec 02 03:05:32 controller-0 etcd[48861]: Git SHA: 18dfb9cca
Dec 02 03:05:32 controller-0 etcd[48861]: Go Version: go1.12.17
Dec 02 03:05:32 controller-0 etcd[48861]: Go OS/Arch: linux/amd64
Dec 02 03:05:32 controller-0 etcd[48861]: setting maximum number of CPUs to 2, total number of available CPUs is 2
Dec 02 03:05:32 controller-0 etcd[48861]: the server is already initialized as member before, starting as etcd member...
Dec 02 03:05:32 controller-0 etcd[48861]: peerTLS: cert = /etc/etcd/kubernetes.pem, key = /etc/etcd/kubernetes-key.pem, trusted-ca = /etc/etcd/ca.pem, cli>
Dec 02 03:05:32 controller-0 etcd[48861]: name = controller-0
Dec 02 03:05:32 controller-0 etcd[48861]: data dir = /var/lib/etcd
Dec 02 03:05:32 controller-0 etcd[48861]: member dir = /var/lib/etcd/member
Dec 02 03:05:32 controller-0 etcd[48861]: heartbeat = 100ms
Dec 02 03:05:32 controller-0 etcd[48861]: election = 1000ms
Dec 02 03:05:32 controller-0 etcd[48861]: snapshot count = 100000
Dec 02 03:05:32 controller-0 etcd[48861]: advertise client URLs = https://10.240.0.10:2379
Dec 02 03:05:32 controller-0 etcd[48861]: initial advertise peer URLs = https://10.240.0.10:2380
Dec 02 03:05:32 controller-0 etcd[48861]: initial cluster =
Dec 02 03:05:32 controller-0 etcd[48861]: restarting member f98dc20bce6225a0 in cluster 3e7cc799faffb625 at commit index 3
Dec 02 03:05:32 controller-0 etcd[48861]: raft2020/12/02 03:05:32 INFO: f98dc20bce6225a0 switched to configuration voters=()
Dec 02 03:05:32 controller-0 etcd[48861]: raft2020/12/02 03:05:32 INFO: f98dc20bce6225a0 became follower at term 183
Dec 02 03:05:32 controller-0 etcd[48861]: raft2020/12/02 03:05:32 INFO: newRaft f98dc20bce6225a0 [peers: [], term: 183, commit: 3, applied: 0, lastindex: >
Dec 02 03:05:32 controller-0 etcd[48861]: simple token is not cryptographically signed
Dec 02 03:05:32 controller-0 etcd[48861]: starting server... [version: 3.4.10, cluster version: to_be_decided]
Dec 02 03:05:32 controller-0 etcd[48861]: raft2020/12/02 03:05:32 INFO: f98dc20bce6225a0 switched to configuration voters=(4203990652121993521)
Dec 02 03:05:32 controller-0 etcd[48861]: added member 3a57933972cb5131 [https://10.240.0.12:2380] to cluster 3e7cc799faffb625
Dec 02 03:05:32 controller-0 etcd[48861]: starting peer 3a57933972cb5131...
Dec 02 03:05:32 controller-0 etcd[48861]: started HTTP pipelining with peer 3a57933972cb5131
Dec 02 03:05:32 controller-0 etcd[48861]: started streaming with peer 3a57933972cb5131 (writer)
Dec 02 03:05:32 controller-0 etcd[48861]: started streaming with peer 3a57933972cb5131 (writer)
Dec 02 03:05:32 controller-0 etcd[48861]: started peer 3a57933972cb5131
somersbmatthews@controller-0:~$ journalctl -xe
Dec 02 03:06:32 controller-0 etcd[48861]: raft2020/12/02 03:06:32 INFO: f98dc20bce6225a0 is starting a new election at term 224
Dec 02 03:06:32 controller-0 etcd[48861]: raft2020/12/02 03:06:32 INFO: f98dc20bce6225a0 became candidate at term 225
Dec 02 03:06:32 controller-0 etcd[48861]: raft2020/12/02 03:06:32 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 225
Dec 02 03:06:32 controller-0 etcd[48861]: raft2020/12/02 03:06:32 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:06:32 controller-0 etcd[48861]: raft2020/12/02 03:06:32 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:06:32 controller-0 etcd[48861]: health check for peer 3a57933972cb5131 could not connect: dial tcp 10.240.0.12:2380: connect: connection refused
Dec 02 03:06:32 controller-0 etcd[48861]: health check for peer ffed16798470cab5 could not connect: dial tcp 10.240.0.11:2380: connect: connection refused
Dec 02 03:06:32 controller-0 etcd[48861]: health check for peer ffed16798470cab5 could not connect: dial tcp 10.240.0.11:2380: connect: connection refused
Dec 02 03:06:32 controller-0 etcd[48861]: health check for peer 3a57933972cb5131 could not connect: dial tcp 10.240.0.12:2380: connect: connection refused
Dec 02 03:06:34 controller-0 etcd[48861]: raft2020/12/02 03:06:34 INFO: f98dc20bce6225a0 is starting a new election at term 225
Dec 02 03:06:34 controller-0 etcd[48861]: raft2020/12/02 03:06:34 INFO: f98dc20bce6225a0 became candidate at term 226
Dec 02 03:06:34 controller-0 etcd[48861]: raft2020/12/02 03:06:34 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 226
Dec 02 03:06:34 controller-0 etcd[48861]: raft2020/12/02 03:06:34 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:06:34 controller-0 etcd[48861]: raft2020/12/02 03:06:34 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:06:35 controller-0 etcd[48861]: raft2020/12/02 03:06:35 INFO: f98dc20bce6225a0 is starting a new election at term 226
Dec 02 03:06:35 controller-0 etcd[48861]: raft2020/12/02 03:06:35 INFO: f98dc20bce6225a0 became candidate at term 227
Dec 02 03:06:35 controller-0 etcd[48861]: raft2020/12/02 03:06:35 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 227
Dec 02 03:06:35 controller-0 etcd[48861]: raft2020/12/02 03:06:35 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:06:35 controller-0 etcd[48861]: raft2020/12/02 03:06:35 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:06:35 controller-0 etcd[48861]: publish error: etcdserver: request timed out
Dec 02 03:06:37 controller-0 etcd[48861]: raft2020/12/02 03:06:37 INFO: f98dc20bce6225a0 is starting a new election at term 227
Dec 02 03:06:37 controller-0 etcd[48861]: raft2020/12/02 03:06:37 INFO: f98dc20bce6225a0 became candidate at term 228
Dec 02 03:06:37 controller-0 etcd[48861]: raft2020/12/02 03:06:37 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 228
Dec 02 03:06:37 controller-0 etcd[48861]: raft2020/12/02 03:06:37 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:06:37 controller-0 etcd[48861]: raft2020/12/02 03:06:37 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:06:37 controller-0 etcd[48861]: health check for peer 3a57933972cb5131 could not connect: dial tcp 10.240.0.12:2380: connect: connection refused
Dec 02 03:06:37 controller-0 etcd[48861]: health check for peer ffed16798470cab5 could not connect: dial tcp 10.240.0.11:2380: connect: connection refused
Dec 02 03:06:37 controller-0 etcd[48861]: health check for peer ffed16798470cab5 could not connect: dial tcp 10.240.0.11:2380: connect: connection refused
Dec 02 03:06:37 controller-0 etcd[48861]: health check for peer 3a57933972cb5131 could not connect: dial tcp 10.240.0.12:2380: connect: connection refused
Dec 02 03:06:38 controller-0 etcd[48861]: raft2020/12/02 03:06:38 INFO: f98dc20bce6225a0 is starting a new election at term 228
Dec 02 03:06:38 controller-0 etcd[48861]: raft2020/12/02 03:06:38 INFO: f98dc20bce6225a0 became candidate at term 229
Dec 02 03:06:38 controller-0 etcd[48861]: raft2020/12/02 03:06:38 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 229
Dec 02 03:06:38 controller-0 etcd[48861]: raft2020/12/02 03:06:38 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:06:38 controller-0 etcd[48861]: raft2020/12/02 03:06:38 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:06:39 controller-0 etcd[48861]: raft2020/12/02 03:06:39 INFO: f98dc20bce6225a0 is starting a new election at term 229
Dec 02 03:06:39 controller-0 etcd[48861]: raft2020/12/02 03:06:39 INFO: f98dc20bce6225a0 became candidate at term 230
Dec 02 03:06:39 controller-0 etcd[48861]: raft2020/12/02 03:06:39 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 230
Dec 02 03:06:39 controller-0 etcd[48861]: raft2020/12/02 03:06:39 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to 3a57933972cb5131 a>
Dec 02 03:06:39 controller-0 etcd[48861]: raft2020/12/02 03:06:39 INFO: f98dc20bce6225a0 [logterm: 1, index: 3] sent MsgVote request to ffed16798470cab5 a>
Dec 02 03:06:41 controller-0 etcd[48861]: raft2020/12/02 03:06:41 INFO: f98dc20bce6225a0 is starting a new election at term 230
Dec 02 03:06:41 controller-0 etcd[48861]: raft2020/12/02 03:06:41 INFO: f98dc20bce6225a0 became candidate at term 231
Dec 02 03:06:41 controller-0 etcd[48861]: raft2020/12/02 03:06:41 INFO: f98dc20bce6225a0 received MsgVoteResp from f98dc20bce6225a0 at term 231
So I redo the step that I think allows what is not being allowed above https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/03-compute-resources.md#firewall-rules:
somersbmatthews@controller-0:~$ gcloud compute firewall-rules create kubernetes-the-hard-way-allow-internal \
> --allow tcp,udp,icmp \
> --network kubernetes-the-hard-way \
> --source-ranges 10.240.0.0/24,10.200.0.0/16
Creating firewall...failed.
ERROR: (gcloud.compute.firewall-rules.create) Could not fetch resource:
- The resource 'projects/k8s-hard-way-2571/global/firewalls/kubernetes-the-hard-way-allow-internal' already exists
but I'm still getting the timeout errors above.
Any help is appreciated, thanks :)
Solution 1:[1]
It looks like you would need to change the source ranges of the VPC network with the update subcommand instead; try:
$ gcloud compute firewall-rules update kubernetes-the-hard-way-allow-internal --source-ranges 10.240.0.0/24,10.200.0.0/16
Solution 2:[2]
I encountered similar error . First , I did a mistake in using master-1 ip address in listen-peer-urls,listen-client-urls ,advertise-client-urls & listen-client-urls
Second , try to test if telnet works on telnet 2380. And if doesnt work , then open firewall on both masters
sudo firewall-cmd --add-port=2380/tcp --permanent
sudo systemctl restart firewalld
Also , it is necessary that both masters should not have considerable time difference.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Arnau C. |
| Solution 2 |