hi, I figured it out. 1) the image created in ceph should only have the feature 'layering'. This can be created with the command: $ rbd create test-image --size=1024 --pool=kubernetes --image-feature layering 2) now the deployment should look like this: --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rbd-static-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi volumeMode: Filesystem volumeName: rbd-static-pv storageClassName: ceph --- apiVersion: v1 kind: PersistentVolume metadata: name: rbd-static-pv spec: volumeMode: Filesystem storageClassName: ceph persistentVolumeReclaimPolicy: Retain accessModes: - ReadWriteOnce capacity: storage: 1Gi csi: driver: rbd.csi.ceph.com fsType: ext4 nodeStageSecretRef: name: csi-rbd-secret namespace: ceph-system volumeAttributes: clusterID: "<clusterID>" pool: "kubernetes" staticVolume: "true" # The imageFeatures must match the created ceph image exactly! imageFeatures: "layering" volumeHandle: test-image Where the <clusterID> must be replaced with the fsid. I wonder why this is not documented on the ceph homepage. I have written my own documentation here: https://github.com/imixs/imixs-cloud/tree/master/management/ceph === Ralph On 12.06.21 15:37, Ralph Soika wrote:
Hi,
I have setup a ceph cluster (octopus) and installed he rbd plugins/provisioner in my Kubernetes cluster. I can create dynamically FS and Block Volumes which is fine. For that I have created the following the following storageClass
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph provisioner: rbd.csi.ceph.com parameters: clusterID: <clusterID> pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-system csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Delete allowVolumeExpansion: true mountOptions: - discard
This works fine for ephemeral dynamically crated volumes. But now I want to use durable volume with the reclaimPolicy:Retain. I expect that I need to create the image in my kubernetes pool on the ceph cluster first - which I have done.
I defined the following new storage class with the reclaimPolicy 'Retain':
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-durable provisioner: rbd.csi.ceph.com parameters: clusterID: <clusterID> pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-system csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - discard
And finally I created the following PersistentVolume and PersistentVolumeClaim:
--- kind: PersistentVolume apiVersion: v1 metadata: name: demo-internal-index spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce claimRef: namespace: office-demo-internal name: index csi: driver: driver.ceph.io fsType: ext4 volumeHandle: demo-internal-index storageClassName: ceph-durable --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: index namespace: office-demo-internal spec: accessModes: - ReadWriteOnce storageClassName: ceph-durable resources: requests: storage: 1Gi volumeName: "demo-internal-index"
But this seems not to work and I can see the following deployment warning:
attachdetach-controller AttachVolume.Attach failed for volume "demo-internal-index" : attachdetachment timeout for volume demo-internal-index
But the PV exists:
$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE demo-internal-index 1Gi RWO Retain Bound office-demo-internal/index ceph-durable 2m35s
and also the PVC exists:
$ kubectl get pvc -n office-demo-internal NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE index Bound demo-internal-index 1Gi RWO ceph-durable 53m
I guess my PV object is nonsense? Can someone provide me an example how to setup the PV object in Kubernetes. I only found examples where the Ceph Monitoring IPs and the user/password is configured within the PV object. But I would expect that this is covered by the storage class already?
Thanks for your help
=== Ralph
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io --
*Imixs Software Solutions GmbH* *Web:* www.imixs.com <http://www.imixs.com> *Phone:* +49 (0)89-452136 16 *Timezone:* Europe/Berlin - CET/CEST *Office:* Agnes-Pockels-Bogen 1, 80992 München Registergericht: Amtsgericht Muenchen, HRB 136045 Geschaeftsführer: Gaby Heinle u. Ralph Soika *Imixs* is an open source company, read more: www.imixs.org <http://www.imixs.org>