'Is it possible to run gcsfuse without privileged mode inside GCP kubernetes?

Following this guide,I am trying to run gcsfuse inside a pod in GKE. Below is the deployment manifest that I am using:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gcsfuse-test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: gcsfuse-test
    spec:
      containers:
        - name: gcsfuse-test
          image: gcr.io/project123/gcs-test-fuse:latest
          securityContext:
            privileged: true
            capabilities:
              add:
                - SYS_ADMIN
          lifecycle:
            postStart:
              exec:
                command: ["mkdir", "/mnt"]
                command: ["gcsfuse", "-o", "nonempty", "cloudsql-p8p", "/mnt"]
            preStop:
              exec:
                command: ["fusermount", "-u", "/mnt"]

However, I would like to run gcsfuse without the privileged mode inside my GKE Cluster. I think (because of questions like these on SO) it is possible to run the docker image with certain flags and there will be no need to run it in privileged mode.

Is there any way in GKE to run gcsfuse without running the container in privileged mode?

kubernetesfusekubernetes-podgcsfuse


Solution 1:[1]

Privileged mode means you have all the capabilities enabled, see https://stackoverflow.com/a/36441605. So adding CAP_SYS_ADMIN looks redundant here in your example.

You can either give all the privileges or do something more fine-grained by mounting /dev/fuse and giving only SYS_ADMIN capability (which remains an important permission).

I think we can rephrase the question as : Can we run GCSFuse without the capability SYS_ADMIN ?

Actually it does not look feasible, you can find the related docker issue here : https://github.com/docker/for-linux/issues/321.

For most of projects it won't be a hard no-go. You may want to act in regard of your threat model and decide whether or not it may be a security risk for your production.

Solution 2:[2]

Edit Apr 26, 2022: for a further developed repo derived from this answer, see https://github.com/samos123/gke-gcs-fuse-unprivileged

Now it finally is possible to mount devices without privileged: true or CAP_SYS_ADMIN!

What you need is

I spent quite some time figuring this out so I hope sharing the information here will help someone else from the exploration.

I wrote my details findings in the Kubernetes Github issue about /dev/fuse. See an example setup in this comment and more technical details above that one.

Examples from the comment linked above:

Allow FUSE devices via Device Manager:

apiVersion: v1
kind: ConfigMap
metadata:
  name: smarter-device-manager
  namespace: device-manager
data:
  conf.yaml: |
    - devicematch: ^fuse$
      nummaxdevices: 20

Request /dev/fuse via Device Manager:

# Pod spec: 
          resources:
            limits:
              smarter-devices/fuse: 1
              memory: 512Mi
            requests:
              smarter-devices/fuse: 1
              cpu: 10m
              memory: 50Mi

Device Manager as a DaemonSet:

# https://gitlab.com/arm-research/smarter/smarter-device-manager/-/blob/master/smarter-device-manager-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: smarter-device-manager
  namespace: device-manager
  labels:
    name: smarter-device-manager
    role: agent
spec:
  selector:
    matchLabels:
      name: smarter-device-manager
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: smarter-device-manager
      annotations:
        node.kubernetes.io/bootstrap-checkpoint: "true"
    spec:
      ## kubectl label node pike5 smarter-device-manager=enabled
      # nodeSelector:
      #   smarter-device-manager : enabled
      priorityClassName: "system-node-critical"
      hostname: smarter-device-management
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: smarter-device-manager
        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager:v1.1.2
        imagePullPolicy: IfNotPresent
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        resources:
          limits:
            cpu: 100m
            memory: 15Mi
          requests:
            cpu: 10m
            memory: 15Mi
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
          - name: dev-dir
            mountPath: /dev
          - name: sys-dir
            mountPath: /sys
          - name: config
            mountPath: /root/config
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: dev-dir
          hostPath:
            path: /dev
        - name: sys-dir
          hostPath:
            path: /sys
        - name: config
          configMap:
            name: smarter-device-manager

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 hugoShaka
Solution 2 Seth

Related Questions