Hi Alison, I have observed exactly that with OSDs "converted" from ceph-disk to ceph-volume. Someone thought it would be a great idea to store the /dev-device name in the config instead of the uuid or any other stable device path: # cat /etc/ceph/osd/287-2eaf591b-bced-4097-9499-5fda071c6161.json { ... "block": { "path": "/dev/disk/by-partuuid/0c8a9f89-efa7-4c75-87ad-2f0d5aa2d649", "uuid": "0c8a9f89-efa7-4c75-87ad-2f0d5aa2d649" }, ... "data": { "path": "/dev/sdm1", "uuid": "2eaf591b-bced-4097-9499-5fda071c6161" }, ... } Funnily enough, it has the by-uuid path stored as well, but the /dev path is actually used during activation. My "fix" is to re-generate the OSD-json just before every ceph-disk OSD start. You seem to be using LVM OSDs already, so this is a bit weird (can't be the exact same issue). Still, I would not be surprised if you are bitten by something similar, some stored config (cache) overrides the actual drive location. It is really a bliss that the developers implemented a check that a partition actually points to the data with the correct OSD ID, otherwise our cluster would be rigged by now. I would start by using low-level commands (ceph-volume) directly to see if the issue is low-level or sits in some higher-level interface. Log-in to the OSD node and check what "ceph-volume inventory" says and if you can manually activate/deactivate the OSD on disk (be careful to include the --no-systemd option everywhere to avoid unintended change of persistent configurations). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: apeisker(a)fnal.gov &lt;apeisker(a)fnal.gov&gt; Sent: Friday, August 25, 2023 10:29 PM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: A couple OSDs not starting after host reboot Hi, Thank you for your reply. I don’t think the device names changed, but ceph seems to be confused about which device the OSD is on. It’s reporting that there are 2 OSDs on the same device although this is not true. ceph device ls-by-host <osd-node> | grep sdu ATA_HGST_HUH728080ALN600_VJH4GLUX sdu osd.665 ATA_HGST_HUH728080ALN600_VJH60MAX sdu osd.657 The osd.665 is actually on device sdm. Could this be the cause of the issue? Is there a way to correct it? Thanks, Alison _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, It looks like Igor is right, it does appear to be a corruption. ls /var/lib/ceph/252fcf9a-b169-11ed-87be-3cecef623f33/osd.665/ ceph_fsid config fsid keyring ready require_osd_release type unit.configured unit.created unit.image unit.meta unit.poststop unit.run unit.stop whoami head -c 4096 /dev/ceph-febad5a5-ba44-41aa-a39e-b9897f757752/osd-block-87e548f4-b9b5-4ed8-aca8-de703a341a50 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 What could have caused this to happen? The ceph-bluestore-tool isn’t able to repair it. Do I need to remove the OSD and create a new one? Thanks, Alison

Hi chenhui, there is still a work in progress to support multiple labels to avoid the issue (https://github.com/ceph/ceph/pull/55374). But this is of little help for your current case. If your disk is fine (meaning it's able to read/write block at offset 0) you might want to try to recover the label using label from a different OSD sitting on a similar(!!that's important!!!) main device. One needs to update osd uuid, whoami and osd_key fields after copying though. Here is the step-by-step procedure: 1. Copy OSD label (4K data block at offset 0) from source OSD's main device to the same location on the broken one: bs=4096 2. Learn broken OSD uuid, N denotes broken OSD id:                 "osd": N,                 "uuid": "6a4ca4ab-6a43-473c-b09c-b13bdd9def5c", 3. Set obtained uuid to copied OSD osd label set-label-key -k osd_uuid -v 6a4ca4ab-6a43-473c-b09c-b13bdd9def5c 4. Update whoami field in the copied label set-label-key -k whoami -v N 5. learn osd's key osd.1         key: AQDrvg9maKxvKxAAqAzqCeR6y0UqBSVIyDhppg== 6. Update osd_key field in the copied label set-label-key -k osd_key -v AQDrvg9maKxvKxAAqAzqCeR6y0UqBSVIyDhppg== 7. Prime OSD dir if it's been lost: <path-to-target-osd-folder> --command prime-osd-dir At this point OSD should be able to start if corrupted label was the only problem. Hope this helps, Igor. On 05/04/2024 05:50, xu chenhui wrote: > Hi， > Has there been any progress on this issue ? is there quick recover method? I have same problem with you that first 4k block of osd metadata is invalid. It will pay a heavy price to recreate osd. > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

On 05/04/2024 17:28, xu chenhui wrote: Unfortunately no. I recall a few cases like that but I doubt any one knows the root cause. You better upgrade to the latest pacific release. > > error info: > systemd[1]: Started Ceph osd.307 for 02eac9e0-d147-11ee-95de-f0b2b90ee048. > bash[39068]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-307 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79 --path /var/lib/ceph/osd/ceph-307 --no-mon-config > bash[39068]: stderr: failed to read label for /dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79: (2) No such file or directory > bash[39068]: --> RuntimeError: command returned non-zero exit status: 1 > > 2024-04-03T14:25:24.349+0000 7f90206fb3c0 10 bluestore(/dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79) _read_bdev_label > 2024-04-03T14:25:24.349+0000 7f90206fb3c0 2 bluestore(/dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input > > thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi, Do slow ops impact data integrity => No Can I generally ignore it => No :) This means that some client transactions are blocked for 120 sec (that's a lot). This could be a lock on the client side (CephFS, essentially), an incident on the infrastructure side (a disk about to fall, network instability, etc.), ... When this happens, you need to look at the blocked requests. If you systematically see an osd ID, then look at dmesg and the SMART of the disk. This can also be an architectural problem (for example, high IOPS load with osdmap on HDD, all multiplied by the erasure code) *David* Le ven. 5 avr. 2024 à 19:42, adam.ther &lt;adam.ther(a)aol.com&gt; a écrit :

Igor Fedotov wrote: Hi, Igor I am repairing the lost label problem. But I have doubt about what's mean "similar main device"? Our OSD and disk have a one-to-one relationship, without disk partitions. How can I find source_osd_block_device ? thanks