You ran a write fio job on the underlying device. That would make any software unhappy. Did you mean to run a read test? Or to test on an RBD volume or a filesystem built within an RBD volume?
Thanks for your answer !
Do you know by any chance why fio does this ? Are fio and ceph incompatible ?
Let's destroy this OSD then
Vivien
________________________________ De : Sinan Polat <sinan86polat@gmail.com> Envoyé : mercredi 23 juillet 2025 13:57:07 À : GLE, Vivien Cc : ceph-users@ceph.io Objet : Re: [ceph-users] Ceph OSD down (unable to mount object store)
Hi Vivien,
Your fio test has very likely destroyed the Ceph OSD block device and the problem is not just the symlink, it's data corruption on the underlying device.
Zap the drive, recreate the OSD and let your cluster rebalance.
Sinan
Op wo 23 jul 2025 om 14:10 schreef GLE, Vivien <Vivien.GLE@inist.fr<mailto:Vivien.GLE@inist.fr>>: Hi,
I did a fio bench and believe that it destroy one of my OSD, this is the command used :
fio --ioengine=libaio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio --ioengine=libaio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio
Log file after the command :
2025-07-23T08:41:44.448+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) close 2025-07-23T08:41:44.719+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block 2025-07-23T08:41:44.719+0000 734bf7f6e680 0 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-2/block failed: (22) Invalid argument 2025-07-23T08:41:44.720+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) open size 1000203091968 (0xe8e0c00000, 932 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2025-07-23T08:41:44.723+0000 734bf7f6e680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-2/block at offset 66: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] 2025-07-23T08:41:44.724+0000 734bf7f6e680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-2/block at offset 4096: End of buffer [buffer:2] 2025-07-23T08:41:44.724+0000 734bf7f6e680 -1 bluestore(/var/lib/ceph/osd/ceph-2) _check_main_bdev_label not all labels read properly 2025-07-23T08:41:44.724+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) close 2025-07-23T08:41:44.983+0000 734bf7f6e680 -1 osd.2 0 OSD:init: unable to mount object store 2025-07-23T08:41:44.983+0000 734bf7f6e680 -1 ** ERROR: osd init failed: (5) Input/output error
After checking I saw that the block soft link might be wrong
On a healthy osd :
# ll /var/lib/ceph/cluster-id/osd.5/ total 72 drwx------ 2 167 167 4096 Jul 22 10:36 ./ drwx------ 12 167 167 4096 Jul 22 10:30 ../ lrwxrwxrwx 1 167 167 93 Jul 15 14:39 block -> /dev/ceph-c31f0e16-0460-4bc5-9470-468270b4c68a/osd-block-72aa3074-e2f9-45f8-a468-03c02d36f1de
On my breaking osd :
/var/lib/ceph/cluster-id/osd.2# ll total 72 drwx------ 2 167 167 4096 Jul 23 10:41 ./ drwx------ 11 167 167 4096 Jul 22 10:35 ../ lrwxrwxrwx 1 167 167 111 Jul 23 10:41 block -> /dev/mapper/ceph--bd0ca671--2b89--4530--be2b--f41443822a91-osd--block--342578c3--4603--4023--a564--8fca6dcc1040
Is there a correct way of changing it or I'm going in the wrong direction ?
Thanks
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io