glusterfs, heketi and kubernetes auto provisioning problem

Question

I have a gluster node and i did test heketi and it is creating volumes using it's cli.

This is my storage class:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: myglusterfs
  annotations:  
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
  resturl: "http://x.x.x:8080"
  restuser: "admin"
  secretName: "heketi-secret"
  secretNamespace: "default"
  volumetype: "replicate:0"
  volumenameprefix: "k8s-dev"
  clusterid: "4d9a77f712zb12x57dd42477b993e9af"

When i create a sample PVC it will stuck on pending state:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: mypvc
spec:
 accessModes:
   - ReadWriteOnce
 resources:
  requests:
   storage: 1Gi

# kubectl get pvc
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc   Pending                                      myglusterfs    5m11s

# kubectl describe pvc mypvc 
Name:          mypvc
Namespace:     default
StorageClass:  myglusterfs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  ProvisioningFailed  14s (x10 over 6m9s)  persistentvolume-controller  Failed to provision volume with StorageClass "myglusterfs": failed to create volume: failed to create volume: see kube-controller-manager.log for details

When i see kube-controller-manager pod logs it's looks like this:

1 event.go:291] "Event occurred" object="default/mypvc" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="Failed to provision volume with StorageClass \"myglusterfs\": failed to create volume: failed to create volume: see kube-controller-manager.log for details"

The question is: How i can find out why the pvc is in pending mode for ever? where is detailed logs?

Answer 1

When there is only one node on gluster cluster the volume type must be none.

volumetype: "none"

Answer 2

Check the resturl IP address Call the command

curl http://x.x.x.x:8080/hello

The output of this command should be as follows

Hello from Heketiroot

Check the IP address heketi Pod

kubectl get pods -o wide | grep "heketi-"

IP-address x.x.x.x (resturl) and ip-address from output must be equal

Check the output of the command

kubectl get secrets | grep "heketi-secret"

The output should not be empty

Authorization must be enabled with restauthenabled: "true"

As a result, I got 3 yml configs

heketi-secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: cGFzc3dvcmQ=
type: kubernetes.io/glusterfs

data.key - echo -n "password" | base64

storage-class.yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gluster-heketi
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
  resturl: "http://x.x.x.x:8080"
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"
  volumetype: "replicate:3"
  volumenameprefix: "k8s-dev"
  gidMin: "40000"
  gidMax: "50000"
  clusterid: "2309963f1aee540437c2aabaeb7a6253"

test-pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gluster-pvc
  annotations:
   volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
  storageClassName: gluster-heketi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Apply these configs

kubectl create -f heketi-secret.yml
kubectl create -f storage-class.yml
kubectl create -f test-pvc.yml

Output of the kubectl get pvc command kubectl get pvc PersistentVolumeClaim Pending