Hi Edouard, For each subvolume listed by 'ceph fs subvolume ls cephfs csi', you could get its PV name with 'rados listomapvals' and then check if this PV still exists or not in K8s: $ ceph fs subvolume ls cephfs csi [ { "name": "csi-vol-fab753bf-c4c0-42d0-98d4-8dd1caf5055f" } ] $ rados listomapvals csi.volume.fab753bf-c4c0-42d0-98d4-8dd1caf5055f --pool=<cephfs-metadata-pool> --namespace=csi csi.imagename value (44 bytes) : 00000000 63 73 69 2d 76 6f 6c 2d 66 61 62 37 35 33 62 66 |csi-vol-fab753bf| 00000010 2d 63 34 63 30 2d 34 32 64 30 2d 39 38 64 34 2d |-c4c0-42d0-98d4-| 00000020 38 64 64 31 63 61 66 35 30 35 35 66 |8dd1caf5055f| 0000002c csi.volname value (40 bytes) : 00000000 70 76 63 2d 34 35 33 61 36 35 65 33 2d 61 31 33 |pvc-453a65e3-a13| 00000010 64 2d 34 64 34 39 2d 62 65 34 37 2d 33 32 62 37 |d-4d49-be47-32b7| 00000020 34 36 30 63 30 62 39 33 |460c0b93| 00000028 csi.volume.owner value (15 bytes) : 00000000 70 77 65 62 2d 6a 75 6c 69 65 69 6e 66 37 37 |pweb-julieinf77| 0000000f The csi.volname should be the name of the PV (here pvc-453a65e3-a13d-4d49-be47-32b7460c0b93) and the csi.volume.owner (if existent) should be the namespace where the PVC was created. You could also check "created_at", "mtime", and browse "path" information from the below command to evalute the value of this data: $ ceph fs subvolume info cephfs csi-vol-fab753bf-c4c0-42d0-98d4-8dd1caf5055f csi | grep -E 'created_at|mtime|path' "created_at": "2024-05-17 07:24:14", "mtime": "2024-10-23 02:00:07", "path": "/volumes/csi/csi-vol-fab753bf-c4c0-42d0-98d4-8dd1caf5055f/1bec4120-7c84-4268-b653-8e6421456df9", Regards, Frédéric. ----- Le 23 Oct 24, à 13:59, Edouard FAZENDA <e.fazenda@csti.ch> a écrit :

Dear Ceph Community,

Maybe you can help me with this

I have inconsistencies with the number of subvolumes in Cephfs in my Ceph Cluster and the number of Persistent Volumes of StorageClass Cephfs in my K8S cluster

I have K8S 1.25 version on my kubernetes cluster and use the Ceph CSI CephFs 3.8.1, my ceph cluster is 16.2.15 pacific release.

I have 35 Persistent Volumes of type CephFS

$ kubectl get pv -o custom-columns=name:.metadata.name,subvolume:.spec.csi.volumeAttributes.subvolumeName,storageclass:.spec.storageClassName | grep cephfs

pvc-0062e845-4a87-4a32-8f19-cbfff3b2789d csi-vol-919a2302-f964-11ed-b5e0-b615b8ae8847 csi-cephfs-sc

pvc-0298fc13-bc5f-489a-b88e-64d7cdc61f1e csi-vol-7ce2f152-d96c-4f54-b335-9ede672ce320 csi-cephfs-sc

pvc-09991b1e-b9f6-4a7c-b710-25131433c6e9 csi-vol-ef2e7be0-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc

pvc-19b8f683-b3e8-4476-abc3-9d7d7e4d8941 csi-vol-5a9ce4b8-1edb-11ed-8f9a-e6ec00c2f5e5 csi-cephfs-sc

pvc-2fc90e06-22b4-4e80-b33a-c4c5dd6a0baa csi-vol-760f34d1-4e26-11ed-af16-d67e0f01e63a csi-cephfs-sc

pvc-316aa495-3d52-45a5-aac3-4f64bf598aa3 csi-vol-9d135b18-8b43-402b-a606-a2c426527b20 csi-cephfs-sc

pvc-342b66f9-f6d5-413c-8634-e51a25d638a8 csi-vol-7e3aca09-8d76-11eb-95b9-ee71e85dde57 csi-cephfs-sc

pvc-48c664a6-073e-4498-9370-d4a718f11729 csi-vol-72d71655-9518-11ee-a81d-b2c17fca41c9 csi-cephfs-sc

pvc-5349d4ec-7e62-48b7-bd8c-8f718a13906b csi-vol-9748018f-4df9-11ed-af16-d67e0f01e63a csi-cephfs-sc

pvc-5db458a4-2dfb-46d7-8b23-2bde45a9bf46 csi-vol-bd87eade-e726-4a5d-993c-56dbece42420 csi-cephfs-sc

pvc-5fd1f3a5-d120-4283-8e2c-b6e1e3e4cbd1 csi-vol-e7ec3159-9d1c-11eb-82b9-aaa0cbaeebfa csi-cephfs-sc

pvc-669c35f9-9927-4e44-94d2-a9ddd0eda914 csi-vol-ef9539f6-9c5a-40d8-a89b-087ad7c6e9b3 csi-cephfs-sc

pvc-6909652a-2da6-4a13-a111-59a21ed685b2 csi-vol-a9a1fff7-550a-43a1-aa52-4f6710e1ca9d csi-cephfs-sc

pvc-72c2f70e-2976-4188-8d1a-f2928745097b csi-vol-51045501-f17b-4f4d-af08-100ea3db3db3 csi-cephfs-sc

pvc-73b1d3f3-f861-44d7-bbd0-43fc16f9e7a9 csi-vol-439769d7-9b94-4350-bfa2-fe5e9a182650 csi-cephfs-sc

pvc-76b04010-1e00-4fea-82b7-855ff83ab820 csi-vol-28852a9b-79c0-4eb7-afcf-b443ff9f867d csi-cephfs-sc

pvc-7e0b08ea-9d31-45a7-a098-588563c18f19 csi-vol-4a5086d9-62ad-11ee-80a7-323a25995550 csi-cephfs-sc

pvc-83853b0b-ab55-46be-ae3a-78f8d431e780 csi-vol-ef2e919c-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc

pvc-89c85899-c26a-4efd-91dd-cc891d303ad6 csi-vol-e8b89f50-fdf9-11ec-8f9a-e6ec00c2f5e5 csi-cephfs-sc

pvc-8a4c12b1-b7d8-4337-b218-7de2bd6eadde csi-vol-718d31a6-9b4a-11ee-a81d-b2c17fca41c9 csi-cephfs-sc

pvc-8b56ae01-1809-4552-9192-930bd1db95f8 csi-vol-a72f2d4c-1f8b-4a71-a846-2b7a6febbf82 csi-cephfs-sc

pvc-97b80840-fd60-436a-8223-5cf41bfe56ef csi-vol-0289ed46-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc

pvc-984eadce-c908-4e21-a558-70d12bcff3c2 csi-vol-ef2e8a6b-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc

pvc-9de21d9d-80ae-49f8-bea0-2a265945cfa6 csi-vol-c7b4f302-4d71-4777-b6c0-82720ebe8950 csi-cephfs-sc

pvc-9f5651b5-0cb2-47d2-889e-69b09d96fe36 csi-vol-9ba650df-3fa5-43f0-abea-5231643fb1bf csi-cephfs-sc

pvc-a2607a68-62a9-47a0-8f15-871359460556 csi-vol-028a40c4-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc

pvc-a84158f8-1d21-475a-9987-a0e8a3347c42 csi-vol-22835b9b-c31c-46c6-aec2-67b554c683d3 csi-cephfs-sc

pvc-b3d81d90-eaec-4d8a-8065-790d9c0e6ea7 csi-vol-c908a216-cf78-11eb-a202-52b7fddc54da csi-cephfs-sc

pvc-bcc5979b-f9cb-4c10-ab64-24424749856b csi-vol-e8dd361e-b6fb-4a19-94af-7cae07d4702c csi-cephfs-sc

pvc-bd09916e-a47d-429f-9371-ac779d3959b3 csi-vol-cc9573bc-6733-11ed-af16-d67e0f01e63a csi-cephfs-sc

pvc-dac9a949-cd28-4448-9c66-7bab365c3d25 csi-vol-21b0d822-6fc3-11ec-96cb-dec0a2aa44f8 csi-cephfs-sc

pvc-e0783e40-f49c-4d7c-a404-4554ce35341c csi-vol-0289e975-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc

pvc-f660eb28-2af4-45de-b00b-72f73e08745d csi-vol-4b6e4fd4-9e4e-4b0d-a022-305b8e1d8e84 csi-cephfs-sc

pvc-fd994df6-9ca5-47d5-af64-afec037fd5d1 csi-vol-ef2e42e3-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc

pvc-fe72f0a2-c260-4a40-acca-50f47d31c11a csi-vol-6c0efe87-d02c-11eb-a202-52b7fddc54da csi-cephfs-sc

I have 985 subvolumes in Cephfs csi volume , i have no idea on why i have this difference.

$ ceph fs subvolume ls cephfs csi | awk '{if ($2 != "") print $2}' | sed 's/"//g' | wc -l

1020

If i check the number of Volume Snapshots of StorageClass Cephfs via the command "kubectl get volumesnapshot -A | grep cephfs | wc -l" i have 177

If i check from the subvolumes of Cephfs that are mapped to the K8S PV i have 178 via the command "for subvolume in $(cat list.txt | xargs);do ceph fs subvolume snapshot ls cephfs $subvolume csi | awk '{if ($2 != "") print $2}' | sed 's/"//g' ;done | wc -l” I have 178

Which seems to be correct and consistent.

Are my checks valid to identifies subvolumes that are not anymore used by my K8S workload ?

Can i safely delete the subvolumes as they seems not associated with a persistent volume in Kubernetes ?

Thanks in advance for the help

Have a nice day

Best Regards, Edouard Fazenda.

Edouard FAZENDA

Technical Support

[ https://www.csti.ch/ | www.csti.ch ]

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io