On Wed, May 19, 2021 at 11:32:04AM +0800, Zhi Zhang wrote:

On Wed, May 19, 2021 at 11:19 AM Zhi Zhang <zhang.david2011@gmail.com> wrote:

...

On Tue, May 18, 2021 at 10:58 PM Mykola Golub <to.my.trociny@gmail.com> wrote:

...

Could you please provide the full rbd-nbd log? If it is too large for the attachment then may be via some public url?

ceph.rbd-client.log.bz2 <https://drive.google.com/file/d/1TuiGOrVAgKIJ3BUmiokG0cU12fnlQ3GR/view?usp=drive_web>

I uploaded it to google driver. Pls check it out.

We found the reader_entry thread got zero byte when trying to read the nbd request header, then rbd-nbd exited and closed the socket. But we haven't figured out why read zero byte?

Ok. I was hoping to find some hint in the log, why the read from the kernel could return without data, but I don't see it. From experience it could happen when the rbd-nbd got stack or was too slow so the kernel failed after timeout, but it looked different in the logs AFAIR. Anyway you can try increasing the timeout using rbd-nbd --timeout (--io-timeout in newer versions) option. The default is 30 sec. If it does not help, probably you will find a clue increasing the kernel debug level for nbd (it seems it is possible to do). -- Mykola Golub