How to share storage between Kubernetes pods?

ghz 1years ago ⋅ 8440 views

Question

I am evaluating Kubernetes as a platform for our new application. For now, it looks all very exciting! However, I’m running into a problem: I’m hosting my cluster on GCE and I need some mechanism to share storage between two pods - the continous integration server and my application server. What’s the best way for doing this with kubernetes? None of the volume types seems to fit my needs, since GCE disks can’t be shared if one pod needs to write to the disk. NFS would be perfect, but seems to require special build options for the kubernetes cluster?

EDIT: Sharing storage seems to be a problem that I have encountered multiple times now using Kubernetes. There are multiple use cases where I'd just like to have one volume and hook it up to multiple pods (with write access). I can only assume that this would be a common use case, no?

EDIT2: For example, this page describes how to set up an Elasticsearch cluster, but wiring it up with persistent storage is impossible (as described here), which kind of renders it pointless :(

Answer

Firstly, do you really need multiple readers / writers?

From my experience of Kubernetes / micro-service architecture (MSA), the issue is often more related to your design pattern. One of the fundamental design patterns with MSA is the proper encapsulation of services, and this includes the data owned by each service.

In much the same way as OOP, your service should look after the data that is related to its area of concern and should allow access to this data to other services via an interface. This interface could be an API, messages handled directly or via a brokage service, or using protocol buffers and gRPC. Generally, multi-service access to data is an anti-pattern akin to global variables in OOP and most programming languages.

As an example, if you where looking to write logs, you should have a log service which each service can call with the relevant data it needs to log. Writing directly to a shared disk means that you'd need to update every container if you change your log directory structure, or decided to add extra functionality like sending emails on certain types of errors.

In the major percentage of cases, you should be using some form of minimal interface before resorting to using a file system, avoiding the unintended side-effects of Hyrum's law that you are exposed to when using a file system. Without proper interfaces / contracts between your services, you heavily reduce your ability to build maintainable and resilient services.

Ok, your situation is best solved using a file system. There are a number

of options...

There are obviously times when a file system that can handle multiple concurrent writers provides a superior solution over a more 'traditional' MSA forms of communication. Kubernetes supports a large number of volume types which can be found here. While this list is quite long, many of these volume types don't support multiple writers (also known as ReadWriteMany in Kubernetes).

Those volume types that do support ReadWriteMany can be found in [this table](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access- modes) and at the time of writing this is AzureFile, CephFS, Glusterfs, Quobyte, NFS and PortworxVolume.

There are also operators such as the popular rook.io which are powerful and provide some great features, but the learning curve for such systems can be a difficult climb when you just want a simple solution and keep moving forward.

The simplest approach.

In my experience, the best initial option is NFS. This is a great way to learn the basic ideas around ReadWriteMany Kubernetes storage, will serve most use cases and is the easiest to implement. After you've built a working knowledge of multi-service persistence, you can then make more informed decisions to use more feature rich offerings which will often require more work to implement.

The specifics for setting up NFS differ based on how and where your cluster is running and the specifics of your NFS service and I've previously written two articles on how to set up NFS for [on-prem clusters](https://ianbelcher.me/tech-blog/adding-persistance-to-on- premises-k8s-cluster) and using AWS NFS equivalent EFS on EKS clusters. These two articles give a good contrast for just how different implementations can be given your particular situation.

For a bare minimum example, you will firstly need an NFS service. If you're looking to do a quick test or you have low SLO requirements, following [this DO article](https://www.digitalocean.com/community/tutorials/how-to-set-up-an- nfs-mount-on-ubuntu-20-04) is a great quick primer for setting up NFS on Ubuntu. If you have an existing NAS which provides NFS and is accessible from your cluster, this will also work as well.

Once you have an NFS service, you can create a persistent volume similar to the following:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-name
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  nfs:
    server: 255.0.255.0 # IP address of your NFS service
    path: "/desired/path/in/nfs"

A caveat here is that your nodes will need binaries installed to use NFS, and I've discussed this more in my [on-prem cluster](https://ianbelcher.me/tech- blog/adding-persistance-to-on-premises-k8s-cluster) article. This is also the reason you need to use EFS when running on EKS as your nodes don't have the ability to connect to NFS.

Once you have the persistent volume set up, it is a simple case of using it like you would any other volume.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-name
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: p-name
          volumeMounts:
            - mountPath: /data
              name: v-name
      volumes:
        - name: v-name
          persistentVolumeClaim:
            claimName: pvc-name
#kubernetes