hi, I figured it out. 1) the image created in ceph should only have the feature 'layering'. This can be created with the command: $ rbd create test-image --size=1024 --pool=kubernetes --image-feature layering 2) now the deployment should look like this:     ---     apiVersion: v1     kind: PersistentVolumeClaim     metadata:       name: rbd-static-pvc     spec:       accessModes:       - ReadWriteOnce       resources:         requests:           storage: 1Gi       volumeMode: Filesystem       volumeName: rbd-static-pv       storageClassName: ceph     ---     apiVersion: v1     kind: PersistentVolume     metadata:       name: rbd-static-pv     spec:       volumeMode: Filesystem       storageClassName: ceph       persistentVolumeReclaimPolicy: Retain       accessModes:       - ReadWriteOnce       capacity:         storage: 1Gi       csi:         driver: rbd.csi.ceph.com         fsType: ext4         nodeStageSecretRef:           name: csi-rbd-secret           namespace: ceph-system         volumeAttributes:           clusterID: "<clusterID>"           pool: "kubernetes"           staticVolume: "true"           # The imageFeatures must match the created ceph image exactly!           imageFeatures: "layering"         volumeHandle: test-image Where the <clusterID> must be replaced with the fsid. I wonder why this is not documented on the ceph homepage. I have written my own documentation here: https://github.com/imixs/imixs-cloud/tree/master/management/ceph === Ralph On 12.06.21 15:37, Ralph Soika wrote:

Hi,

I have setup a ceph cluster (octopus) and installed he rbd plugins/provisioner in my Kubernetes cluster. I can create dynamically FS and Block Volumes which is fine. For that I have created the following the following storageClass

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata:    name: ceph provisioner: rbd.csi.ceph.com parameters:    clusterID: <clusterID>    pool: kubernetes    imageFeatures: layering    csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret    csi.storage.k8s.io/provisioner-secret-namespace: ceph-system    csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret    csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system    csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret    csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Delete allowVolumeExpansion: true mountOptions:    - discard

This works fine for ephemeral dynamically crated volumes. But now I want to use durable volume with the reclaimPolicy:Retain.  I expect that I need to create the image in my kubernetes pool on the ceph cluster first - which I have done.

I defined the following new storage class with the reclaimPolicy 'Retain':

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata:    name: ceph-durable provisioner: rbd.csi.ceph.com parameters:    clusterID: <clusterID>    pool: kubernetes    imageFeatures: layering    csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret    csi.storage.k8s.io/provisioner-secret-namespace: ceph-system    csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret    csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system    csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret    csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Retain allowVolumeExpansion: true mountOptions:    - discard

And finally I created the following PersistentVolume and PersistentVolumeClaim:

--- kind: PersistentVolume apiVersion: v1 metadata:   name: demo-internal-index spec:   capacity:     storage: 1Gi   volumeMode: Filesystem   accessModes:     - ReadWriteOnce   claimRef:     namespace: office-demo-internal     name: index   csi:     driver: driver.ceph.io     fsType: ext4     volumeHandle: demo-internal-index   storageClassName: ceph-durable --- apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: index   namespace: office-demo-internal spec:   accessModes:     - ReadWriteOnce   storageClassName: ceph-durable   resources:     requests:       storage: 1Gi   volumeName: "demo-internal-index"

But this seems not to work and I can see the following deployment warning:

attachdetach-controller  AttachVolume.Attach failed for volume "demo-internal-index" : attachdetachment timeout for volume demo-internal-index

But the PV exists:

$ kubectl get pv NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS CLAIM                                  STORAGECLASS       REASON AGE demo-internal-index                        1Gi RWO Retain           Bound office-demo-internal/index ceph-durable 2m35s

and also the PVC exists:

$ kubectl get pvc -n office-demo-internal NAME    STATUS   VOLUME                CAPACITY   ACCESS MODES STORAGECLASS   AGE index   Bound    demo-internal-index   1Gi        RWO ceph-durable   53m

I guess my PV object is nonsense? Can someone provide me an example how to setup the PV object in Kubernetes. I only found examples where the Ceph Monitoring IPs and the user/password is configured within the PV object. But I would expect that this is covered by the storage class already?

Thanks for your help

=== Ralph

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io --

*Imixs Software Solutions GmbH* *Web:* www.imixs.com <http://www.imixs.com> *Phone:* +49 (0)89-452136 16 *Timezone:* Europe/Berlin - CET/CEST *Office:* Agnes-Pockels-Bogen 1, 80992 München Registergericht: Amtsgericht Muenchen, HRB 136045 Geschaeftsführer: Gaby Heinle u. Ralph Soika *Imixs* is an open source company, read more: www.imixs.org <http://www.imixs.org>