Hello List, oversudden i can not mount a specific rbd device anymore: root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0 root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out) Any idea how to debug that mount? Tcpdump does show some active traffic. Cheers, Michael
On Tue, Jun 22, 2021 at 8:36 AM Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Any idea how to debug that mount? Tcpdump does show some active traffic.
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Have you checked the status of the cluster (ceph -s)? Are there any OSD issues, network problems (can you ping your MONs, OSD hosts)? Check your syslog on the client for any timeout entries. That should be a good start, and give you some diagnostic info. -- Alex Gorbachev ISS/Storcium
ceph -s is healthy. I started to do a xfs_repair on that block device now which seems to do something...: - agno = 1038 - agno = 1039 - agno = 1040 - agno = 1041 - agno = 1042 - agno = 1043 - agno = 1044 - agno = 1045 - agno = 1046 - agno = 1047 - agno = 1048 - agno = 1049 - agno = 1050 (i am new to xfs) but hat proofes that the block device is alive and accessible? Maybe I have a Filesystem problem? On Tue, Jun 22, 2021 at 3:33 PM Alex Gorbachev <ag@iss-integration.com> wrote:
On Tue, Jun 22, 2021 at 8:36 AM Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Any idea how to debug that mount? Tcpdump does show some active traffic.
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Have you checked the status of the cluster (ceph -s)? Are there any OSD issues, network problems (can you ping your MONs, OSD hosts)? Check your syslog on the client for any timeout entries. That should be a good start, and give you some diagnostic info.
-- Alex Gorbachev ISS/Storcium
On Tue, Jun 22, 2021 at 10:12 AM Ml Ml <mliebherr99@googlemail.com> wrote:
ceph -s is healthy. I started to do a xfs_repair on that block device now which seems to do something...:
- agno = 1038 - agno = 1039 - agno = 1040 - agno = 1041 - agno = 1042 - agno = 1043 - agno = 1044 - agno = 1045 - agno = 1046 - agno = 1047 - agno = 1048 - agno = 1049 - agno = 1050
(i am new to xfs) but hat proofes that the block device is alive and accessible? Maybe I have a Filesystem problem?
I would look in syslog, dmesg, and use a -v flag with mount to see if anything else comes up. Maybe parted /dev/rbd0 to see if you can access the partition table. xfs_repair -L can address any issues with XFS, but it should not break on its own
On Tue, Jun 22, 2021 at 3:33 PM Alex Gorbachev <ag@iss-integration.com> wrote:
On Tue, Jun 22, 2021 at 8:36 AM Ml Ml <mliebherr99@googlemail.com>
wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Any idea how to debug that mount? Tcpdump does show some active traffic.
Cheers, Michael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Have you checked the status of the cluster (ceph -s)? Are there any OSD issues, network problems (can you ping your MONs, OSD hosts)? Check your syslog on the client for any timeout entries. That should be a good start, and give you some diagnostic info.
-- Alex Gorbachev ISS/Storcium
On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Hi, there used to be some kernel lock issues when the kernel rbd client tried to access an OSD on the same machine. Not sure if these issues still exist (but I would guess so) and if you use your proxmox cluster in a hyperconverged manner (nodes providing VMs and storage service at the same time) you may just have been lucky that it had worked before. Instead of the kernel client mount you can try to export the volume as an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and mounting that. rbd-nbd runs in userspace and should not have that locking problem. Matthias
On Wed, Jun 23, 2021 at 9:59 AM Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Hi,
there used to be some kernel lock issues when the kernel rbd client tried to access an OSD on the same machine. Not sure if these issues still exist (but I would guess so) and if you use your proxmox cluster in a hyperconverged manner (nodes providing VMs and storage service at the same time) you may just have been lucky that it had worked before.
Instead of the kernel client mount you can try to export the volume as an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and mounting that. rbd-nbd runs in userspace and should not have that locking problem.
rbd-nbd is also susceptible to locking up in such setups, likely more so than krbd. Don't forget that it also has a kernel component and there are actually more opportunities for things to go sideways/lock up because there is an extra daemon involved allocating some additional memory for each I/O request. Thanks, Ilya
The rbd Client is not on one of the OSD Nodes. I now added a "backup-proxmox/cluster5a" to it and it works perfectly. Just that one rbd image sucks. The last thing i remember was to resize the Image from 6TB to 8TB and i then did a xfs_grow on it. Does that ring a bell? On Wed, Jun 23, 2021 at 11:25 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Wed, Jun 23, 2021 at 9:59 AM Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Hi,
there used to be some kernel lock issues when the kernel rbd client tried to access an OSD on the same machine. Not sure if these issues still exist (but I would guess so) and if you use your proxmox cluster in a hyperconverged manner (nodes providing VMs and storage service at the same time) you may just have been lucky that it had worked before.
Instead of the kernel client mount you can try to export the volume as an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and mounting that. rbd-nbd runs in userspace and should not have that locking problem.
rbd-nbd is also susceptible to locking up in such setups, likely more so than krbd. Don't forget that it also has a kernel component and there are actually more opportunities for things to go sideways/lock up because there is an extra daemon involved allocating some additional memory for each I/O request.
Thanks,
Ilya
On Fri, Jun 25, 2021 at 11:25 AM Ml Ml <mliebherr99@googlemail.com> wrote:
The rbd Client is not on one of the OSD Nodes.
I now added a "backup-proxmox/cluster5a" to it and it works perfectly. Just that one rbd image sucks. The last thing i remember was to resize the Image from 6TB to 8TB and i then did a xfs_grow on it.
Does that ring a bell?
It does seem like a filesystem problem so far but you haven't posted dmesg or other details. "mount" will not time out, if it's not returning due to hanging somewhere you would likely get "task ... blocked for ..." splats in dmesg. Thanks, Ilya
On Wed, Jun 23, 2021 at 11:25 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Wed, Jun 23, 2021 at 9:59 AM Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Hi,
there used to be some kernel lock issues when the kernel rbd client tried to access an OSD on the same machine. Not sure if these issues still exist (but I would guess so) and if you use your proxmox cluster in a hyperconverged manner (nodes providing VMs and storage service at the same time) you may just have been lucky that it had worked before.
Instead of the kernel client mount you can try to export the volume as an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and mounting that. rbd-nbd runs in userspace and should not have that locking problem.
rbd-nbd is also susceptible to locking up in such setups, likely more so than krbd. Don't forget that it also has a kernel component and there are actually more opportunities for things to go sideways/lock up because there is an extra daemon involved allocating some additional memory for each I/O request.
Thanks,
Ilya
I started the mount 15mins ago.: mount -nv /dev/rbd6 /mnt/backup-cluster5 ps: root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | \_ mount -nv /dev/rbd6 /mnt/backup-cluster5 There is no timout or ANY msg in dmesg until now. strace -p 1143 : seems to do nothing. iotop --pid=1143: uses about 50KB/sec it might mount after a few hours i gues... :-( On Fri, Jun 25, 2021 at 11:39 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 25, 2021 at 11:25 AM Ml Ml <mliebherr99@googlemail.com> wrote:
The rbd Client is not on one of the OSD Nodes.
I now added a "backup-proxmox/cluster5a" to it and it works perfectly. Just that one rbd image sucks. The last thing i remember was to resize the Image from 6TB to 8TB and i then did a xfs_grow on it.
Does that ring a bell?
It does seem like a filesystem problem so far but you haven't posted dmesg or other details. "mount" will not time out, if it's not returning due to hanging somewhere you would likely get "task ... blocked for ..." splats in dmesg.
Thanks,
Ilya
On Wed, Jun 23, 2021 at 11:25 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Wed, Jun 23, 2021 at 9:59 AM Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Hi,
there used to be some kernel lock issues when the kernel rbd client tried to access an OSD on the same machine. Not sure if these issues still exist (but I would guess so) and if you use your proxmox cluster in a hyperconverged manner (nodes providing VMs and storage service at the same time) you may just have been lucky that it had worked before.
Instead of the kernel client mount you can try to export the volume as an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and mounting that. rbd-nbd runs in userspace and should not have that locking problem.
rbd-nbd is also susceptible to locking up in such setups, likely more so than krbd. Don't forget that it also has a kernel component and there are actually more opportunities for things to go sideways/lock up because there is an extra daemon involved allocating some additional memory for each I/O request.
Thanks,
Ilya
Btw: dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec. So reading the block device seems to work?! On Fri, Jun 25, 2021 at 12:39 PM Ml Ml <mliebherr99@googlemail.com> wrote:
I started the mount 15mins ago.: mount -nv /dev/rbd6 /mnt/backup-cluster5
ps: root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
There is no timout or ANY msg in dmesg until now.
strace -p 1143 : seems to do nothing. iotop --pid=1143: uses about 50KB/sec
it might mount after a few hours i gues... :-(
On Fri, Jun 25, 2021 at 11:39 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Fri, Jun 25, 2021 at 11:25 AM Ml Ml <mliebherr99@googlemail.com> wrote:
The rbd Client is not on one of the OSD Nodes.
I now added a "backup-proxmox/cluster5a" to it and it works perfectly. Just that one rbd image sucks. The last thing i remember was to resize the Image from 6TB to 8TB and i then did a xfs_grow on it.
Does that ring a bell?
It does seem like a filesystem problem so far but you haven't posted dmesg or other details. "mount" will not time out, if it's not returning due to hanging somewhere you would likely get "task ... blocked for ..." splats in dmesg.
Thanks,
Ilya
On Wed, Jun 23, 2021 at 11:25 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Wed, Jun 23, 2021 at 9:59 AM Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote:
Hello List,
oversudden i can not mount a specific rbd device anymore:
root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0
root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out)
Hi,
there used to be some kernel lock issues when the kernel rbd client tried to access an OSD on the same machine. Not sure if these issues still exist (but I would guess so) and if you use your proxmox cluster in a hyperconverged manner (nodes providing VMs and storage service at the same time) you may just have been lucky that it had worked before.
Instead of the kernel client mount you can try to export the volume as an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and mounting that. rbd-nbd runs in userspace and should not have that locking problem.
rbd-nbd is also susceptible to locking up in such setups, likely more so than krbd. Don't forget that it also has a kernel component and there are actually more opportunities for things to go sideways/lock up because there is an extra daemon involved allocating some additional memory for each I/O request.
Thanks,
Ilya
participants (4)
-
Alex Gorbachev
-
Ilya Dryomov
-
Matthias Ferdinand
-
Ml Ml