'Kubernetes does not start after restart system (Ubuntu)
I installed K8s on my two Ubuntus in VirtualBox (Master and Node01). After installation (I proceeded according K8s doc site) I typed kubectl get nodes and got bot servers in status Ready. But after restart systems I got this:
# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the
right host or port?
I checked kubelet service and it is running:
# systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2017-04-24 10:01:51 CEST; 15min ago
Docs: http://kubernetes.io/docs/
Main PID: 13128 (kubelet)
Tasks: 21
Memory: 48.2M
CPU: 58.014s
CGroup: /system.slice/kubelet.service
├─13128 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cluster-domain=cluster.local
└─13164 journalctl -k -f
Apr 24 10:16:40 master kubelet[13128]: I0424 10:16:40.204156 13128 kuberuntime_manager.go:752] Back-off 5m0s restarting failed container=weave pod=weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)
Apr 24 10:16:40 master kubelet[13128]: E0424 10:16:40.204694 13128 pod_workers.go:182] Error syncing pod 4b7bb2f0-2691-11e7-bfb6-080027229776 ("weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)"), skipping: fail
Apr 24 10:16:42 master kubelet[13128]: I0424 10:16:42.972302 13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2b59d0d9-2692-11e7-bfb6-080027229776-default-token-h3v7c" (spec.Name: "
Apr 24 10:16:48 master kubelet[13128]: I0424 10:16:48.949731 13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2bb42bc1-2692-11e7-bfb6-080027229776-default-token-h3v7c" (spec.Name: "
Apr 24 10:16:51 master kubelet[13128]: I0424 10:16:51.978663 13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2b023c31-2692-11e7-bfb6-080027229776-default-token-h3v7c" (spec.Name: "
Apr 24 10:16:52 master kubelet[13128]: I0424 10:16:52.909589 13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/4b7bb2f0-2691-11e7-bfb6-080027229776-default-token-gslqd" (spec.Name: "
Apr 24 10:16:53 master kubelet[13128]: I0424 10:16:53.186057 13128 kuberuntime_manager.go:458] Container {Name:weave Image:weaveworks/weave-kube:1.9.4 Command:[/home/weave/launch.sh] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:
Apr 24 10:16:53 master kubelet[13128]: I0424 10:16:53.188091 13128 kuberuntime_manager.go:742] checking backoff for container "weave" in pod "weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)"
Apr 24 10:16:53 master kubelet[13128]: I0424 10:16:53.188717 13128 kuberuntime_manager.go:752] Back-off 5m0s restarting failed container=weave pod=weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)
Apr 24 10:16:53 master kubelet[13128]: E0424 10:16:53.189136 13128 pod_workers.go:182] Error syncing pod 4b7bb2f0-2691-11e7-bfb6-080027229776 ("weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)"), skipping: fail
Here is systemd log file with restarted kubelet: Google Drive.
... I'm not sure what I missed in doc or what happend with kubelet. Can I ask you for help? :]
• Ubuntu version
cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
• Kernel
# uname -a
Linux ubuntu 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Kubectl version
# kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
• Kubeadm version
# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
• Kubelet version
# kubelet --version
Kubernetes v1.6.1
• Docker version
# docker version
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Solution 1:[1]
I had a bad exported variable KUBECONFIG which is needed by kubelet (history details are in a comment under question).
To ~/.zprofile I saved KUBECONFIG=$HOME/admin.conf which solved my problem.
After reloading ENV variables is kubelet working:
# kubectl get nodes
NAME STATUS AGE VERSION
master Ready 5d v1.6.1
node01 NotReady 5d v1.6.1
Solution 2:[2]
I had the same problem with kubernetes 1.12.3 and ubuntu 16.04.05. I then looked at the kubernetes log by running the command
$ journalctl -u kubelet
and then in the log i saw that k8s was complaining (exiting with status 255) about swap being on.
So i then turned swap off by running
$ swapoff -a
Then i edited fstab and commented out the entry for swap
$ vi /etc/fstab
#comment out line with swap
and then rebooting the system. After the system came back up, i verified that swap was disabled by running
$ free -m
and checking whether the row for swap has 0. It did.
Then i verified that kubeapi service had successfully started by executing
$ systemctl status kubelet
It has successfully started. I verified by also re-checking journalctl logs. Did not see swap error this time.
I verified k8s node status by running
$ kubectl get nodes
which was now working and showing expected output.
NOTE: I had KUBECONFIG set in my .bash_profile file as well, previously.
root@k8s-master:~# cat .bash_profile
export KUBECONFIG="/etc/kubernetes/admin.conf"
Solution 3:[3]
As the comment there, you really need check whether apiserver is started, because kubectl will talk to apiserver. While from you description and version of kubeadm, I believe this is a duplicate question I just answered, so I just copy the answer to here.
In current version of kubeadm(v1.6.1), insecure port of ApiServer is abandoned by default, you can verify this by checking api-server yaml file in /etc/kubernetes/manifests/kube-apiserver.yaml, there is kube-apiserver parameter --insecure-port=0.
You can
Correct this in a running cluster:
$ mv kube-apiserver.yaml ../kube-apiserver.yaml // edit ../kube-apiserver.yaml to remove --insecure-port=0 // or change it to --insecure-port=<WHATERER_YOUR_LIKE> $ mv ../kube-apiserver.yaml kube-apiserver.yamlDo it right at startup. You need a kubeadm config file to do this. A simple one would like:
apiVersion: kubeadm.k8s.io/v1alpha1 kind: MasterConfiguration apiServerExtraArgs: insecure-port: 8080 //or whatever you like // Then you can start a master node use `kubeadm init --config=<this-configure-file-path>`
Solution 4:[4]
I'm no expert, but I've found the problem to be that the containers for API server etc, not starting.
I've found for me its a docker unix sock permission issue:
$ chmod 666 /var/run/docker.sock
$ sudo systemctl restart docker
this resolved the issue for me
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Arash Hatami |
| Solution 2 | |
| Solution 3 | Community |
| Solution 4 |