Hi, Yeah, I've been testing various configurations since I sent my last email - all to no avail. So I'm back to the start with a brand new 4T image which is rbdmapped to /dev/rbd0. Its not formatted (yet) and so not mounted. Every time I attempt a mkfs.xfs /dev/rbd0 (or mkfs.xfs /dev/rbd/my_pool/my_image) I get the errors I previous mentioned and the resulting image then becomes unusable (in ever sense of the word). If I run a fdisk -l (before trying the mkfs.xfs) the rbd image shows up in the list - no, I don't actually do a full fdisk on the image. An rbd info my_pool:my_image shows the same expected values on both the host and ceph cluster. I've tried this with a whole bunch of different sized images from 100G to 4T and all fail in exactly the same way. (My previous successful 100G test I haven't been able to reproduce). I've also tried all of the above using an "admin" CephX(sp?) account - I always can connect via rbdmap, but as soon as I try an mkfs.xfs it fails. This failure also occurs with a mkfs.ext4 as well (all size drives). The Ceph Cluster is good (self reported and there are other hosts happily connected via CephFS) and this host also has a CephFS mapping which is working. Between running experiments I've gone over the Ceph Doco (again) and I can't work out what's going wrong. There's also nothing obvious/helpful jumping out at me from the logs/journal (sample below): ~~~ Mar 24 17:38:29 my_host.my_net.local kernel: rbd: rbd0: write at objno 524773 0~65536 result -1 Mar 24 17:38:29 my_host.my_net.local kernel: rbd: rbd0: write at objno 524772 65536~4128768 result -1 Mar 24 17:38:29 my_host.my_net.local kernel: rbd: rbd0: write result -1 Mar 24 17:38:29 my_host.my_net.local kernel: blk_print_req_error: 119 callbacks suppressed Mar 24 17:38:29 my_host.my_net.local kernel: I/O error, dev rbd0, sector 4298932352 op 0x1:(WRITE) flags 0x4000 phys_seg 1024 prio class 2 Mar 24 17:38:29 my_host.my_net.local kernel: rbd: rbd0: write at objno 524774 0~65536 result -1 Mar 24 17:38:29 my_host.my_net.local kernel: rbd: rbd0: write at objno 524773 65536~4128768 result -1 Mar 24 17:38:29 my_host.my_net.local kernel: rbd: rbd0: write result -1 Mar 24 17:38:29 my_host.my_net.local kernel: I/O error, dev rbd0, sector 4298940544 op 0x1:(WRITE) flags 0x4000 phys_seg 1024 prio class 2 ~~~ Any ideas what I should be looking at? And thank you for the help :-) On 24/03/2024 17:50, Alexander E. Patrakov wrote:
Hi,
Please test again, it must have been some network issue. A 10 TB RBD image is used here without any problems.