'Minikube got stuck when creating container

I recently got started to learn Kubernetes by using Minikube locally in my Mac. Previously, I was able to start a local Kubernetes cluster with Minikube 0.10.0, created a deployment and viewed Kubernetes dashboard.

Yesterday I tried to delete the cluster and re-did everything from scratch. However, I found I cannot get the assets deployed and cannot view the dashboard. From what I saw, everything seemed to get stuck during container creation.

After I ran minikube start, it reported 

Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.

When I ran kubectl get pods --all-namespaces, it reported (pay attention to the STATUS column):

kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY     STATUS              RESTARTS   AGE
kube-system   kube-addon-manager-minikube   0/1       ContainerCreating   0          51s

docker ps showed nothing:

docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

minikube status tells me the VM and cluster are running:

minikubeVM: Running
localkube: Running

If I tried to create a deployment and an autoscaler, I was told they were created successfully:

kubectl create -f configs
deployment "hello-minikube" created
horizontalpodautoscaler "hello-minikube-autoscaler" created

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY     STATUS              RESTARTS   AGE
default       hello-minikube-661011369-1pgey   0/1       ContainerCreating   0          1m
default       hello-minikube-661011369-91iyw   0/1       ContainerCreating   0          1m
kube-system   kube-addon-manager-minikube      0/1       ContainerCreating   0          21m

When exposing the service, it said:

$ kubectl expose deployment hello-minikube --type=NodePort
service "hello-minikube" exposed

$ kubectl get service
NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
hello-minikube   10.0.0.32    <nodes>       8080/TCP   6s
kubernetes       10.0.0.1     <none>        443/TCP    22m

When I tried to access the service, I was told:

curl $(minikube service hello-minikube --url)
Waiting, endpoint for service is not ready yet...

docker ps still showed nothing. It looked to me everything got stuck when creating a container. I tried some other ways to work around this issue:

  1. Upgraded to minikube 0.11.0
  2. Use the xhyve driver instead of the Virtualbox driver
  3. Delete everything cached, like ~/.minikube, ~/.kube, and the cluster, and re-try

None of them worked for me.

Kubernetes is still new to me and I would like to know:

  1. How can I troubleshoot this kind of issue?
  2. What could be the cause of this issue?

Any help is appreciated. Thanks.

kubernetesminikube


Solution 1:[1]

It turned out to be a network problem in my case.

The pod status is "ContainerCreating", and I found during container creation, docker image will be pulled from gcr.io, which is inaccessible in China (blocked by GFW). Previous time it worked for me because I happened to connect to it via a VPN.

Solution 2:[2]

I didn't try minikube but I use kubernetes. With the information provided it is difficult to say the cause of the issue. Your minikube has no problem in creating resources but ContainerCreating is a problem related to docker daemon or improper communication between kube-api and docker daemon or some problem with kubelet.

You can try the following command:

kubectl describe po POD_NAME

This will give you the POD's events. Maybe this will provide a path to the root cause of issue.

You may also check the logs of kubelet to get the events.

Solution 3:[3]

I had this problem on Windows, but it was related to an NTLM proxy. I deleted the minikube VM then recreated it with the correct proxy settings for my CNTLM installation:

minikube start \
--docker-env http_proxy=http://10.0.2.2:3128 \
--docker-env https_proxy=http://10.0.2.2:3128 \
--docker-env no_proxy=localhost,127.0.0.1,::1,192.168.99.100

See https://blog.alexellis.io/minikube-behind-proxy/

Solution 4:[4]

The horizontalpodautoscaler (hpa) requires heapster to use. You'll need to run heapster in minikube for that to work. You can always debug these kinds of issues with minikube logs or interactively through the dashboard found at minikube dashboard.

You can find the steps to run heapster and grafana at https://github.com/kubernetes/heapster

Solution 5:[5]

For me, it takes several minutes before I see the ContainerCreating problem. After executing the following command:

systemctl status kube-controller-manager.service

I get this error:

Sync "default/redis-master-2229813293" failed with unable to create pods: No API token found for service account "default", retry after the token is automatically created and added to the service account.

There are two ways to solve this:

  1. Set the service account with token
  2. Remove the ServiceAccount setting of KUBE_ADMISSION_CONTROL in api-server

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 GabLeRoux
Solution 3 GabLeRoux
Solution 4 Matt Rickard
Solution 5 GabLeRoux

Related Questions