Creating a cluster from existing PVCs

[oberstet]

Hi all,

I've been trying to figure out how to create a simple RabbitMQ cluster with 3 nodes using the RabbitMQ Cluster Operator with pre-existing PVCs data-rabbitmq-0, data-rabbitmq-1 and data-rabbitmq-2.

Each RabbitMQ node 0, 1, 2 should only use the respective data-rabbitmq-0, data-rabbitmq-1 and data-rabbitmq-2 PVC.

No PVCs must be created dynamically.

How do I do that?

Is there an option to specify something like this (doesn't work, just to illustrate what I am looking for):

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
    name: hello-world
spec:
  replicas: 3
  pvcs:
    - data-rabbitmq-0
    - data-rabbitmq-1
    - data-rabbitmq-2

I've read the docs, examples, tried various things, no luck:( I'm not even sure the operator is able to or designed to do that.

How are people running multi node clusters on pre-provisioned storage, with volumes dedicated to RabbitMQ nodes (which seems crucial for high performance clustered storage on RabbitMQ)?

Any hints or help would be greatly appreciated!

Cheers,
/Tobias

[Zerpet]

This is tricky, although could be possible with some clever naming. It is complicated because the Operator uses VolumeClaimTemplate from the StatefulSet, see here:

# ...
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "my-storage-class"
      resources:
        requests:
          storage: 1Gi

The statefulset controller then creates the PVCs and mount thems to each Pod. It is challenging to define each PVC for each Pod "manually" because the statefulset uses a Pod template; this has the implications that all Pods will be "replicas" of each other, just with a different Pod name, volumes and volume mounts will be identical. The VolumeClaimTemplate solves this problem, as it attaches one PVC for each Pod.

How do you run on pre-provisioned storage? You can create your PVCs with the names that the StatefulSet controller will attempt to create. IIRC, it follows the pattern persistence-<rabbitmqcluster-name>-<index> where rabbitmqcluster-name would be hello-world in your example, and index is the index for each replica, starting at 0. For example, if you have 3 replicas, indexes would be 0, 1 and 2. By pre-creating the PVCs, the StatefulSet will just use them. Be aware tho, that if you delete the RabbitmqCluster, the PVCs will be deleted as well.

As an alternative to all this trickery, you could define a StorageClass (see docs) with a provisioner for your high performace storage, and set that name in the storage class field in RabbitmqCluste spec.

[oberstet]

@Zerpet thank you so much for your comments and help! and also for "confirming" that I did not miss some docs or simple things ... it's in my opinion really missing ... I've wasted many hours poking around for sth that should be a simple best practice recipe for production. anyways, thanks!

It is challenging to define each PVC for each Pod "manually" because the statefulset uses a Pod template;

yes, this is exactly my problem. in my desperation, I even dug through the code base, and I found the lines responsible for the PVC template

cluster-operator/config/crd/bases/rabbitmq.com_rabbitmqclusters.yaml

Line 3961 in bd1f682

volumeClaimTemplates:

and I found PersistentVolume.claimRef.

I also recognized that what I actually want to achieve is this:

• Multi-node RabbitMQ cluster with storage dedicated to and separate for each node
• Have that per-node storage coming from per-node PVs which are pre-created/existing

I actually do not absolutely need to precreate PVCs - my thinking was that pre-creating per-node PVCs 1:1 bound to per-node PVsm plus using those PVCs 1:1 for the Pods of the cluster would achieve above.

this is true, but could also be achieved if the stateful set created from the controller would automatically choose the right PV for every new node PVC created.

so I want to modify my PVs to have claimRef pointing to the (future) PVCs that will be created.

which means: I need to figure out the names of the PVCs that the controller will create (later).

as you mention, the naming pattern for PVCs dynamically created by a StatefulSet is:

persistence-<rabbitmqcluster-name>-<index>

is there a docs pages listing such name patterns? since there is this name based magic like claimRef where configuration isn't explicit, but implicit/magic. is there docs for such magic?

You can create your PVCs with the names that the StatefulSet controller will attempt to create.

this is even "more magic" ... maybe it chooses the right PV for a PVC .. or not. why can't I simply specify explicitly the binding of Cluster to Pods to PVCs to PVs? I don't get it.

hence my plan is to try setting:

PersistentVolume.claimRef = "persistence-<rabbitmqcluster-name>-<index>"

this way (I hope) a given PV can only ever be bound by the one right PVC. is that true? is it enough (for correct 1:1 between Pod, PVC and PV)?

As an alternative to all this trickery, you could define a StorageClass (see docs) with a provisioner for your high performace storage, and set that name in the storage class field in RabbitmqCluste spec.

using StorageClass - ah right. I will set that on my PVs as well!

[Zerpet]

as you mention, the naming pattern for PVCs dynamically created by a StatefulSet is:

persistence-<rabbitmqcluster-name>-<index>

is there a docs pages listing such name patterns? since there is this name based magic like claimRef where configuration isn't explicit, but implicit/magic. is there docs for such magic?

I don't recall such documentation. I know out of experience coding the operator. It's behaviour from the StatefulSet controller, and it's not something we could control, sadly.

You can create your PVCs with the names that the StatefulSet controller will attempt to create.

this is even "more magic" ... maybe it chooses the right PV for a PVC .. or not. why can't I simply specify explicitly the binding of Cluster to Pods to PVCs to PVs? I don't get it.

Design from upstream Kubernetes controllers, I guess 😕 I think you idea of "I want this exact PV, in this exact Node, to be bound to this exact PVC, and this exact Pod to use" is what everyone would expect from an imperative API. Kubernetes API is not imperative, but declarative. That may be a reason why this is so hard, we may be approaching the problem with the wrong mindset 🤷‍♂️

If I were to have a system with your requirements, I would try to put a layer on top of my storage (perhaps Rook? maybe just a storage class with a provisioner would do?), so that I could dynamically provision my PVs with "hints" (read, Kubernetes labels or annotations) as where I want them (read label selectors on K8s Node labels).
Once you have deployed successfully, subsequent Pod scheduling (e.g. you restart all your Pods), the kubernetes scheduler finds the PV restriction and schedules the Pod to run on the Node where the PV is. But in my experience, this causes more headaches than actually helps. You probably want your PVs for your data services replicated, to ensure availability. Rook project is a solution to systems that don't have replicated storage, or systems that are not "cloud native" 😉

[oberstet]

I don't recall such documentation.

thanks for confirming!

Design from upstream Kubernetes controllers .. imperative vs declarative ..

yes, agreed. it seems my "reflexes" might collide with the design, and fighting the design "makes no sense". so I will jump through whatever name or other magic is needed to "make it work";)

I want to identify and map pods in a stateful set to dedicated (RWO) persistent volumes (eg network block storage LUNs).

Using "storageClass" and "claimRef" to steer the mapping when auto-creating PVCs seems to be what I need to do.

I'll figure it out ..