Hello Seeking some help if I can clean the client mounting my volume? rbd status pool/image Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088 Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used It shows above client, but there is no such volume… Another similar issue… on dashboard… CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate. Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message.. CEPH version: 18.2.4 Regards Dev
Hi, have you checked if the image is in the trash? rbd -p {pool} trash ls You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again. I have to do that from time to time on a customer’s openstack cluster. Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen Thanks for your reply. I have the image available and it’s not under trash. When scaling a pod to different node using statefulset, pod gives mount issue. I was looking for a command if we can kill the client.id <http://client.id/> from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host. Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Similar issue https://github.com/ceph/ceph-csi/discussions/4410 Regards Dev On Tue, 21 Jan 2025 at 2:33 PM, Devender Singh <devender@netskrt.io> wrote:
Hello Eugen
Thanks for your reply. I have the image available and it’s not under trash.
When scaling a pod to different node using statefulset, pod gives mount issue.
I was looking for a command if we can kill the client.id from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host.
Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dev, Can you run the below command to check if this image is still considered as mapped by any ceph-csi nodeplugins? $ namespace=ceph-csi-rbd $ image=csi-vol-945c6a66-9129 $ for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done If it pops out in the output, get into the csi-rbdplugin container of the nodeplugin pod that listed the image and unmount/unmap it: $ kubectl -n $namespace exec -ti ceph-csi-rbd-nodeplugin-xxxxx -c csi-rbdplugin -- sh <---- please adjust nodepluding pod name here sh-4.4# sh-4.4# rbd device list id pool namespace image snap device 0 k8s-rgnl-disks csi-vol-945c6a66-9129 - /dev/rbd0 sh-4.4# umount /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd unmap /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd device list sh-4.4# Hope there's no typo. Regards, Frédéric. ----- Le 21 Jan 25, à 23:33, Devender Singh devender@netskrt.io a écrit :
Hello Eugen
Thanks for your reply. I have the image available and it’s not under trash.
When scaling a pod to different node using statefulset, pod gives mount issue.
I was looking for a command if we can kill the client.id <http://client.id/> from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host.
Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frederic Thanks for your email. We already verified those and tried killing them and upgrading the k8s and cis-plugin to but nothing helps. Below is the output.. did not report any volume.. # for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done ceph-csi-rbd-nodeplugin-48vs2 ceph-csi-rbd-nodeplugin-6zmjj ceph-csi-rbd-nodeplugin-7g6r5 ceph-csi-rbd-nodeplugin-bp84x ceph-csi-rbd-nodeplugin-bt6hh ceph-csi-rbd-nodeplugin-d4tww ceph-csi-rbd-nodeplugin-rtb68 ceph-csi-rbd-nodeplugin-t87db But still error ; # date;kubectl -n elastic describe pod/es-es-default-3 |grep -i warning Wed 22 Jan 2025 01:12:09 PM PST Warning FailedMount 2s (x13 over 21m) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66 is still being used Regards Dev
On Jan 21, 2025, at 11:50 PM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Dev,
Can you run the below command to check if this image is still considered as mapped by any ceph-csi nodeplugins?
$ namespace=ceph-csi-rbd $ image=csi-vol-945c6a66-9129 $ for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done
If it pops out in the output, get into the csi-rbdplugin container of the nodeplugin pod that listed the image and unmount/unmap it:
$ kubectl -n $namespace exec -ti ceph-csi-rbd-nodeplugin-xxxxx -c csi-rbdplugin -- sh <---- please adjust nodepluding pod name here sh-4.4# sh-4.4# rbd device list id pool namespace image snap device 0 k8s-rgnl-disks csi-vol-945c6a66-9129 - /dev/rbd0 sh-4.4# umount /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd unmap /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd device list sh-4.4#
Hope there's no typo.
Regards, Frédéric.
----- Le 21 Jan 25, à 23:33, Devender Singh devender@netskrt.io <mailto:devender@netskrt.io> a écrit :
Hello Eugen
Thanks for your reply. I have the image available and it’s not under trash.
When scaling a pod to different node using statefulset, pod gives mount issue.
I was looking for a command if we can kill the client.id <http://client.id/> <https://www.google.com/url?q=http://client.id/&source=gmail-imap&ust=1738137024000000&usg=AOvVaw10QRl9S7YS6pPaI6JKmdyy> from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host.
Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Dev, Is the image name 'csi-vol-945c6a66-9129' as in your first message or is it 'csi-vol-945c6a66' as in your last message? Because the command I sent to you grepped 'csi-vol-945c6a66-9129' and not 'csi-vol-945c6a66'... No clones or snapshots made by Velero on this image, right? If the image is still accessed by node 10.160.0.245, it should have a entry in path /sys/kernel/debug/ceph/$(ceph fsid).client12541259/... <----- client12541259 being the watchers session you get with 'rbd status' command without the '.' (dot) If so, then you could try to drain/cordon/reboot the K8s node and see if it clears up the watchers. Regards, Frédéric. ----- Le 22 Jan 25, à 22:13, Devender Singh <devender@netskrt.io> a écrit : Hello Frederic Thanks for your email. We already verified those and tried killing them and upgrading the k8s and cis-plugin to but nothing helps. Below is the output.. did not report any volume.. # for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done ceph-csi-rbd-nodeplugin-48vs2 ceph-csi-rbd-nodeplugin-6zmjj ceph-csi-rbd-nodeplugin-7g6r5 ceph-csi-rbd-nodeplugin-bp84x ceph-csi-rbd-nodeplugin-bt6hh ceph-csi-rbd-nodeplugin-d4tww ceph-csi-rbd-nodeplugin-rtb68 ceph-csi-rbd-nodeplugin-t87db But still error ; # date;kubectl -n elastic describe pod/es-es-default-3 |grep -i warning Wed 22 Jan 2025 01:12:09 PM PST Warning FailedMount 2s (x13 over 21m) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66 is still being used Regards Dev BQ_BEGIN On Jan 21, 2025, at 11:50 PM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote: Hi Dev, Can you run the below command to check if this image is still considered as mapped by any ceph-csi nodeplugins? $ namespace=ceph-csi-rbd $ image=csi-vol-945c6a66-9129 $ for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done If it pops out in the output, get into the csi-rbdplugin container of the nodeplugin pod that listed the image and unmount/unmap it: $ kubectl -n $namespace exec -ti ceph-csi-rbd-nodeplugin-xxxxx -c csi-rbdplugin -- sh <---- please adjust nodepluding pod name here sh-4.4# sh-4.4# rbd device list id pool namespace image snap device 0 k8s-rgnl-disks csi-vol-945c6a66-9129 - /dev/rbd0 sh-4.4# umount /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd unmap /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd device list sh-4.4# Hope there's no typo. Regards, Frédéric. ----- Le 21 Jan 25, à 23:33, Devender Singh [ mailto:devender@netskrt.io | devender@netskrt.io ] a écrit : BQ_BEGIN Hello Eugen Thanks for your reply. I have the image available and it’s not under trash. When scaling a pod to different node using statefulset, pod gives mount issue. I was looking for a command if we can kill the [ http://client.id/ | client.id ] < [ https://www.google.com/url?q=http://client.id/&source=gmail-imap&ust=1738137024000000&usg=AOvVaw10QRl9S7YS6pPaI6JKmdyy | https://www.google.com/url?q=http://client.id/&source=gmail-imap&ust=1738137024000000&usg=AOvVaw10QRl9S7YS6pPaI6JKmdyy ] > from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host. Regards Dev BQ_BEGIN On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote: Hi, have you checked if the image is in the trash? rbd -p {pool} trash ls You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again. I have to do that from time to time on a customer’s openstack cluster. Zitat von Devender Singh <devender@netskrt.io>: BQ_BEGIN Hello Seeking some help if I can clean the client mounting my volume? rbd status pool/image Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088 Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used It shows above client, but there is no such volume… Another similar issue… on dashboard… CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate. Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message.. CEPH version: 18.2.4 Regards Dev _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io BQ_END _______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ] BQ_END BQ_END BQ_END
Hello all Sorry for late reply.. I tried two things, 1. My cluster was using swap, I made swap off. 2. I started repair command on pool/image and seems it worked . But after that there was no command found to pause repair as it started deep scrubs too. How to unpause the repairs… Regards Dev
On Jan 23, 2025, at 5:56 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
ys/kernel/debug/ceph/$(ceph fsid).client12541259/...
Why would you want to unpause repairs? Do you suffer from performance issues during deep-scrubs? You could pause those with: ceph osd set nodeep-scrub But they would only reveal inconsistent PGs during deep-scrub. To enable automatic repairs during deep-scrub, you would need to first enable it. It's really unclear what your current cluster status is, maybe underlying Ceph issues are causing troubles, but you don't provide too many details. Zitat von Devender Singh <devender@netskrt.io>:
Hello all
Sorry for late reply..
I tried two things,
1. My cluster was using swap, I made swap off. 2. I started repair command on pool/image and seems it worked .
But after that there was no command found to pause repair as it started deep scrubs too. How to unpause the repairs…
Regards Dev
On Jan 23, 2025, at 5:56 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
ys/kernel/debug/ceph/$(ceph fsid).client12541259/...
Hello Eugen Thanks for your reply. ceph osd set nodeep-scrub is not stopping if repairs are running. Reapir started another set for deepscrub+repair which is not controlled using this command. When I started my cluster utilization as 74% and when it finished now my cluster is showing 43%(surprising figure, does a repair can shuffle that much? Even my osd’s where between 60-83%) utilized. Before starting repair I increased pool pg’s manually from 1024 to 2048 but my pool only came down to 84%, so I thought to run pool repair.. During the repair and high utilization my cluster was crying and giving SLOW OSD Communication from osd.x to osd.y and many number of times. Not sure what was it, how utilization came that much down, If repair can change thing this much then it should have command to pause/unpause too. Regards Dev
On Jan 25, 2025, at 1:15 AM, Eugen Block <eblock@nde.ag> wrote:
But they would only reveal inconsistent PGs during deep-scrub.
We really don't know what the cluster state was. I assume by removing the fifth host you removed the fullest OSD(s). But again, we only get a few bits of information, so it's still basically guessing. What exactly do you mean by "so I thought to run pool repair"? What exactly did you do? Zitat von Devender Singh <devender@netskrt.io>:
Hello Eugen
Thanks for your reply.
ceph osd set nodeep-scrub is not stopping if repairs are running. Reapir started another set for deepscrub+repair which is not controlled using this command.
When I started my cluster utilization as 74% and when it finished now my cluster is showing 43%(surprising figure, does a repair can shuffle that much? Even my osd’s where between 60-83%) utilized. Before starting repair I increased pool pg’s manually from 1024 to 2048 but my pool only came down to 84%, so I thought to run pool repair.. During the repair and high utilization my cluster was crying and giving SLOW OSD Communication from osd.x to osd.y and many number of times.
Not sure what was it, how utilization came that much down, If repair can change thing this much then it should have command to pause/unpause too.
Regards Dev
On Jan 25, 2025, at 1:15 AM, Eugen Block <eblock@nde.ag> wrote:
But they would only reveal inconsistent PGs during deep-scrub.
Hi, I've had a similar issue but outside of ceph-csi. Running a CRUD test to (create, map, write, read, unmap, and delete) an RBD in a short amount of time can result in it having a stuck watcher. I assume it is from mapping and unmapping very quickly (under 30 sec). What I have found is if you restart the primary osd for the header object, the watcher will go away assuming nothing is really watching it.
rbd info -p pool-name rbd-name # get the id from the output. ex: 1234 ceph osd map pool-name rbd_header.1234 # get the primary under acting pNNN ex 43 ceph osd down 43
This is the tracker <https://tracker.ceph.com/issues/58120> I'm watching and the backport says it should be fixed in 18.2.5 Hope this helps, Reid On Wed, Jan 22, 2025 at 4:14 PM Devender Singh <devender@netskrt.io> wrote:
Hello Frederic
Thanks for your email. We already verified those and tried killing them and upgrading the k8s and cis-plugin to but nothing helps. Below is the output.. did not report any volume..
# for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done ceph-csi-rbd-nodeplugin-48vs2 ceph-csi-rbd-nodeplugin-6zmjj ceph-csi-rbd-nodeplugin-7g6r5 ceph-csi-rbd-nodeplugin-bp84x ceph-csi-rbd-nodeplugin-bt6hh ceph-csi-rbd-nodeplugin-d4tww ceph-csi-rbd-nodeplugin-rtb68 ceph-csi-rbd-nodeplugin-t87db
But still error ; # date;kubectl -n elastic describe pod/es-es-default-3 |grep -i warning Wed 22 Jan 2025 01:12:09 PM PST Warning FailedMount 2s (x13 over 21m) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66 is still being used
Regards Dev
On Jan 21, 2025, at 11:50 PM, Frédéric Nass < frederic.nass@univ-lorraine.fr> wrote:
Hi Dev,
Can you run the below command to check if this image is still considered as mapped by any ceph-csi nodeplugins?
$ namespace=ceph-csi-rbd $ image=csi-vol-945c6a66-9129 $ for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done
If it pops out in the output, get into the csi-rbdplugin container of the nodeplugin pod that listed the image and unmount/unmap it:
$ kubectl -n $namespace exec -ti ceph-csi-rbd-nodeplugin-xxxxx -c csi-rbdplugin -- sh <---- please adjust nodepluding pod name here sh-4.4# sh-4.4# rbd device list id pool namespace image snap device 0 k8s-rgnl-disks csi-vol-945c6a66-9129 - /dev/rbd0 sh-4.4# umount /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd unmap /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd device list sh-4.4#
Hope there's no typo.
Regards, Frédéric.
----- Le 21 Jan 25, à 23:33, Devender Singh devender@netskrt.io <mailto: devender@netskrt.io> a écrit :
Hello Eugen
Thanks for your reply. I have the image available and it’s not under trash.
When scaling a pod to different node using statefulset, pod gives mount issue.
I was looking for a command if we can kill the client.id < http://client.id/> < https://www.google.com/url?q=http://client.id/&source=gmail-imap&ust=1738137024000000&usg=AOvVaw10QRl9S7YS6pPaI6JKmdyy
from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host.
Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto: ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Reid, Yep, it should definitely help if the client node (kernel) is not accessing the image anymore. Thanks for sharing the tracker. It's good to know that a fix is on the way. Cheers, Frédéric. ----- Le 23 Jan 25, à 15:02, Reid Guyett reid.guyett@gmail.com a écrit :
Hi,
I've had a similar issue but outside of ceph-csi. Running a CRUD test to (create, map, write, read, unmap, and delete) an RBD in a short amount of time can result in it having a stuck watcher. I assume it is from mapping and unmapping very quickly (under 30 sec). What I have found is if you restart the primary osd for the header object, the watcher will go away assuming nothing is really watching it.
rbd info -p pool-name rbd-name # get the id from the output. ex: 1234 ceph osd map pool-name rbd_header.1234 # get the primary under acting pNNN ex 43 ceph osd down 43
This is the tracker <https://tracker.ceph.com/issues/58120> I'm watching and the backport says it should be fixed in 18.2.5
Hope this helps, Reid
On Wed, Jan 22, 2025 at 4:14 PM Devender Singh <devender@netskrt.io> wrote:
Hello Frederic
Thanks for your email. We already verified those and tried killing them and upgrading the k8s and cis-plugin to but nothing helps. Below is the output.. did not report any volume..
# for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done ceph-csi-rbd-nodeplugin-48vs2 ceph-csi-rbd-nodeplugin-6zmjj ceph-csi-rbd-nodeplugin-7g6r5 ceph-csi-rbd-nodeplugin-bp84x ceph-csi-rbd-nodeplugin-bt6hh ceph-csi-rbd-nodeplugin-d4tww ceph-csi-rbd-nodeplugin-rtb68 ceph-csi-rbd-nodeplugin-t87db
But still error ; # date;kubectl -n elastic describe pod/es-es-default-3 |grep -i warning Wed 22 Jan 2025 01:12:09 PM PST Warning FailedMount 2s (x13 over 21m) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66 is still being used
Regards Dev
On Jan 21, 2025, at 11:50 PM, Frédéric Nass < frederic.nass@univ-lorraine.fr> wrote:
Hi Dev,
Can you run the below command to check if this image is still considered as mapped by any ceph-csi nodeplugins?
$ namespace=ceph-csi-rbd $ image=csi-vol-945c6a66-9129 $ for pod in $(kubectl -n $namespace get pods | grep -E 'rbdplugin|nodeplugin' | grep -v provisioner | awk '{print $1}'); do echo $pod; kubectl exec -it -n $namespace $pod -c csi-rbdplugin -- rbd device list | grep $image ; done
If it pops out in the output, get into the csi-rbdplugin container of the nodeplugin pod that listed the image and unmount/unmap it:
$ kubectl -n $namespace exec -ti ceph-csi-rbd-nodeplugin-xxxxx -c csi-rbdplugin -- sh <---- please adjust nodepluding pod name here sh-4.4# sh-4.4# rbd device list id pool namespace image snap device 0 k8s-rgnl-disks csi-vol-945c6a66-9129 - /dev/rbd0 sh-4.4# umount /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd unmap /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129 sh-4.4# rbd device list sh-4.4#
Hope there's no typo.
Regards, Frédéric.
----- Le 21 Jan 25, à 23:33, Devender Singh devender@netskrt.io <mailto: devender@netskrt.io> a écrit :
Hello Eugen
Thanks for your reply. I have the image available and it’s not under trash.
When scaling a pod to different node using statefulset, pod gives mount issue.
I was looking for a command if we can kill the client.id < http://client.id/> < https://www.google.com/url?q=http://client.id/&source=gmail-imap&ust=1738137024000000&usg=AOvVaw10QRl9S7YS6pPaI6JKmdyy
from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host.
Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto: ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear all Sending again. Our mail server was having some issue, not sure if you have received it or not. Exactly Similar issue, https://github.com/ceph/ceph-csi/discussions/4410 Regards Dev
On Jan 21, 2025, at 2:33 PM, Devender Singh <devender@netskrt.io> wrote:
Hello Eugen
Thanks for your reply. I have the image available and it’s not under trash.
When scaling a pod to different node using statefulset, pod gives mount issue.
I was looking for a command if we can kill the client.id <http://client.id/> from ceph. CEPH must have a command to kill its clients etc… Don’t understand why pod complaining about same volume name about a k8s host using it. Whereas its nowhere.. Not sure what to do in this situation.. We tried upgrading csi, k8s cluster. Renamed image and blocklisted the host. And renamed back image to its original image but still red status showing same client host.
Regards Dev
On Jan 21, 2025, at 12:16 PM, Eugen Block <eblock@nde.ag> wrote:
Hi,
have you checked if the image is in the trash?
rbd -p {pool} trash ls
You can try to restore the image if there is one, then blocklist the client to release the watcher, then delete the image again.
I have to do that from time to time on a customer’s openstack cluster.
Zitat von Devender Singh <devender@netskrt.io>:
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used
It shows above client, but there is no such volume…
Another similar issue… on dashboard…
CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate.
Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message..
CEPH version: 18.2.4
Regards Dev
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Devender Singh
-
Eugen Block
-
Frédéric Nass
-
Reid Guyett