NFS volume mount results in exit code 32 on in Kubernetes?

ghz 1years ago ⋅ 2092 views

Question

I'm trying to mount an external nfs share in a Replication Controller. When I create the replication controller, the pod is pending. Getting the details on the pod, I get these events:

Events:
  FirstSeen             LastSeen            Count   From            SubobjectPath   Reason      Message
  Thu, 05 Nov 2015 11:28:33 -0700   Thu, 05 Nov 2015 11:28:33 -0700 1   {scheduler }                scheduled   Successfully assigned web-worker-hthjq to jolt-server-5
  Thu, 05 Nov 2015 11:28:43 -0700   Thu, 05 Nov 2015 11:28:43 -0700 1   {kubelet jolt-server-5}         failedMount Unable to mount volumes for pod "web-worker-hthjq_default": exit status 32
  Thu, 05 Nov 2015 11:28:43 -0700   Thu, 05 Nov 2015 11:28:43 -0700 1   {kubelet jolt-server-5}         failedSync  Error syncing pod, skipping: exit status 32

My set up is one master and one node on local machines. These machines are running CoreOS. The nfs share exists on another machine on the network. If I shell into the host machine, I can successfully mount the nfs share, so I believe the export is configured correctly. Looking around online, it seems like the only examples of using nfs shares are those defined within Kubernetes (one pod sharing with another). Is there not a way to mount an external share directly from Kubernetes (I don't want to have to mount it to the host machine and then mount from the host machine to the container).

Here's my Replication Controller:

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: web-worker
  name: web-worker
spec:
  replicas: 1
  selector:
    component: web-worker
  template:
    metadata:
      labels:
        app: task-queue
        component: web-worker
    spec:
      containers:
        - command:
            - /sbin/my_init
          image: quincy/php-apache2:latest
          name: web-worker
          ports:
            - containerPort: 80
              protocol: TCP
          volumeMounts:
            - name: code-base
              mountPath: /var/www/html
            - name: local-secrets
              mountPath: /secrets
      volumes:
        - name: code-base
          nfs:
            server: thuey.jolt.local
            path: "/Users/thuey/Sites/jolt/jolt-web"
        - name: local-secrets
          secret:
            secretName: local-secret

UPDATE

After thinking about it more, I realized the problem is probably that it can't find the server (thuey.jolt.local). It's probably just looking at the internal DNS. Is that accurate? If so, is there any way around that?

UPDATE

After attempting this again, it is now working mounting directly from the pod to the networked nfs server.

Answer

With @rwehner's help, I was finally able to figure it out. Checking the kubelet log revealed:

Output: mount.nfs: rpc.statd is not running but is required for remote locking.

As soon as I got rpcbind running, the mount worked.

#kubernetes