rbd-nbd crashes Error: failed to read nbd request header: (33) Numerical argument out of domain
18 May
2021
18 May
'21
11:34 a.m.
Hi guys,
We are recently testing rbd-nbd using ceph N version. After map rbd
image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show
following errors when doing some read/write testing.
rbd-nbd log:
2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry:
waiting for nbd request
...
2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd
request header: (33) Numerical argument out of domain
2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io
requests, terminating
2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
0x564a2be2b3c0 unregister_update_watcher: handle=0
2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0
2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing
unregister
2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer: terminating
2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
0x564a2be2b3c0 close
dmesg:
[Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with
ordered data mode. Opts: discard
[Tue May 18 11:35:07 2021] block nbd0: shutting down sockets
[Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0,
sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
client host info:
centos7.x
kernel 5.4.109
It looks like the kernel nbd device shutdown its socket for some
reason, but we haven't figured it out. BTW, we have tried to turn
on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated
pool, but the error remains. It is more frequent for us to reproduce
when batch map, mkfs and mount rbd-nbd on different hosts
simultaneously.
Thanks for any suggestions.
Regards,
Zhi Zhang (David)
Contact: zhang.david2011(a)gmail.com
zhangz.david(a)outlook.com
18 May
18 May
7:28 p.m.
Could you please provide the full rbd-nbd log? If it is too large for
the attachment then may be via some public url?
--
Mykola Golub
On Tue, May 18, 2021 at 03:04:51PM +0800, Zhi Zhang wrote:
> Hi guys,
>
> We are recently testing rbd-nbd using ceph N version. After map rbd
> image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show
> following errors when doing some read/write testing.
>
> rbd-nbd log:
>
> 2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry:
> waiting for nbd request
> ...
> 2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd
> request header: (33) Numerical argument out of domain
> 2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io
> requests, terminating
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b3c0 unregister_update_watcher: handle=0
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing
> unregister
> 2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer: terminating
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b3c0 close
>
> dmesg:
>
> [Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with
> ordered data mode. Opts: discard
> [Tue May 18 11:35:07 2021] block nbd0: shutting down sockets
> [Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0,
> sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
>
> client host info:
>
> centos7.x
> kernel 5.4.109
>
>
> It looks like the kernel nbd device shutdown its socket for some
> reason, but we haven't figured it out. BTW, we have tried to turn
> on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated
> pool, but the error remains. It is more frequent for us to reproduce
> when batch map, mkfs and mount rbd-nbd on different hosts
> simultaneously.
>
> Thanks for any suggestions.
>
> Regards,
> Zhi Zhang (David)
> Contact: zhang.david2011(a)gmail.com
> zhangz.david(a)outlook.com
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
19 May
19 May
7:49 a.m.
On Tue, May 18, 2021 at 10:58 PM Mykola Golub <to.my.trociny(a)gmail.com>
wrote:
Could you please provide the full rbd-nbd log? If it is too large for
the attachment then may be via some public url?
ceph.rbd-client.log.bz2
<https://drive.google.com/file/d/1TuiGOrVAgKIJ3BUmiokG0cU12fnlQ3GR/view?usp=drive_web>
I uploaded it to google driver. Pls check it out.
--
Mykola Golub
On Tue, May 18, 2021 at 03:04:51PM +0800, Zhi Zhang wrote:
> Hi guys,
>
> We are recently testing rbd-nbd using ceph N version. After map rbd
> image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show
> following errors when doing some read/write testing.
>
> rbd-nbd log:
>
> 2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry:
> waiting for nbd request
> ...
> 2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd
> request header: (33) Numerical argument out of domain
> 2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io
> requests, terminating
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b3c0 unregister_update_watcher: handle=0
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing
> unregister
> 2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer:
terminating
> 2021-05-18 11:35:08.066 7efdea8d1a00 20
[]librbd::ImageState:
> 0x564a2be2b3c0 close
>
> dmesg:
>
> [Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with
> ordered data mode. Opts: discard
> [Tue May 18 11:35:07 2021] block nbd0: shutting down sockets
> [Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0,
> sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
>
> client host info:
>
> centos7.x
> kernel 5.4.109
>
>
> It looks like the kernel nbd device shutdown its socket for some
> reason, but we haven't figured it out. BTW, we have tried to turn
> on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated
> pool, but the error remains. It is more frequent for us to reproduce
> when batch map, mkfs and mount rbd-nbd on different hosts
> simultaneously.
>
> Thanks for any suggestions.
>
> Regards,
> Zhi Zhang (David)
> Contact: zhang.david2011(a)gmail.com
> zhangz.david(a)outlook.com
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:02 a.m.
On Wed, May 19, 2021 at 11:19 AM Zhi Zhang <zhang.david2011(a)gmail.com>
wrote:
On Tue, May 18, 2021 at 10:58 PM Mykola Golub <to.my.trociny(a)gmail.com>
wrote:
We found the reader_entry thread got zero byte when trying to read the nbd
request header, then rbd-nbd exited and closed the socket. But we haven't
figured out why read zero byte?
Could you please provide the full rbd-nbd log? If it is too large for
the attachment then may be via some public url?
ceph.rbd-client.log.bz2
<https://drive.google.com/file/d/1TuiGOrVAgKIJ3BUmiokG0cU12fnlQ3GR/view?usp=drive_web>
I uploaded it to google driver. Pls check it out.
--
Mykola Golub
On Tue, May 18, 2021 at 03:04:51PM +0800, Zhi Zhang wrote:
> Hi guys,
>
> We are recently testing rbd-nbd using ceph N version. After map rbd
> image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show
> following errors when doing some read/write testing.
>
> rbd-nbd log:
>
> 2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry:
> waiting for nbd request
> ...
> 2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd
> request header: (33) Numerical argument out of domain
> 2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io
> requests, terminating
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b3c0 unregister_update_watcher: handle=0
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0
> 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState:
> 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing
> unregister
> 2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer:
terminating
> 2021-05-18 11:35:08.066 7efdea8d1a00 20
[]librbd::ImageState:
> 0x564a2be2b3c0 close
>
> dmesg:
>
> [Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with
> ordered data mode. Opts: discard
> [Tue May 18 11:35:07 2021] block nbd0: shutting down sockets
> [Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0,
> sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
>
> client host info:
>
> centos7.x
> kernel 5.4.109
>
>
> It looks like the kernel nbd device shutdown its socket for some
> reason, but we haven't figured it out. BTW, we have tried to turn
> on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated
> pool, but the error remains. It is more frequent for us to reproduce
> when batch map, mkfs and mount rbd-nbd on different hosts
> simultaneously.
>
> Thanks for any suggestions.
>
> Regards,
> Zhi Zhang (David)
> Contact: zhang.david2011(a)gmail.com
> zhangz.david(a)outlook.com
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:23 p.m.
On Wed, May 19, 2021 at 11:32:04AM +0800, Zhi Zhang wrote:
On Wed, May 19, 2021 at 11:19 AM Zhi Zhang
<zhang.david2011(a)gmail.com>
wrote:
Ok. I was hoping to find some hint in the log, why the read from the
kernel could return without data, but I don't see it.
From experience it could happen when the rbd-nbd got stack or was too
slow so the kernel failed after timeout, but it looked different in
the logs AFAIR. Anyway you can try increasing the timeout using
rbd-nbd --timeout (--io-timeout in newer versions) option. The default
is 30 sec.
If it does not help, probably you will find a clue increasing the
kernel debug level for nbd (it seems it is possible to do).
--
Mykola Golub
On Tue, May 18, 2021 at 10:58 PM Mykola Golub <to.my.trociny(a)gmail.com>
wrote:
We found the reader_entry thread got zero byte when trying to read the nbd
request header, then rbd-nbd exited and closed the socket. But we haven't
figured out why read zero byte?
Could you please provide the full rbd-nbd log? If it is too large for
the attachment then may be via some public url?
ceph.rbd-client.log.bz2
<https://drive.google.com/file/d/1TuiGOrVAgKIJ3BUmiokG0cU12fnlQ3GR/view?usp=drive_web>
I uploaded it to google driver. Pls check it out.
1070
days inactive
1071
days old
4 comments
2 participants
participants (2)
-
Mykola Golub -
Zhi Zhang