Hi Alison, I have observed exactly that with OSDs "converted" from ceph-disk to ceph-volume. Someone thought it would be a great idea to store the /dev-device name in the config instead of the uuid or any other stable device path: # cat /etc/ceph/osd/287-2eaf591b-bced-4097-9499-5fda071c6161.json { ... "block": { "path": "/dev/disk/by-partuuid/0c8a9f89-efa7-4c75-87ad-2f0d5aa2d649", "uuid": "0c8a9f89-efa7-4c75-87ad-2f0d5aa2d649" }, ... "data": { "path": "/dev/sdm1", "uuid": "2eaf591b-bced-4097-9499-5fda071c6161" }, ... } Funnily enough, it has the by-uuid path stored as well, but the /dev path is actually used during activation. My "fix" is to re-generate the OSD-json just before every ceph-disk OSD start. You seem to be using LVM OSDs already, so this is a bit weird (can't be the exact same issue). Still, I would not be surprised if you are bitten by something similar, some stored config (cache) overrides the actual drive location. It is really a bliss that the developers implemented a check that a partition actually points to the data with the correct OSD ID, otherwise our cluster would be rigged by now. I would start by using low-level commands (ceph-volume) directly to see if the issue is low-level or sits in some higher-level interface. Log-in to the OSD node and check what "ceph-volume inventory" says and if you can manually activate/deactivate the OSD on disk (be careful to include the --no-systemd option everywhere to avoid unintended change of persistent configurations). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: apeisker@fnal.gov <apeisker@fnal.gov> Sent: Friday, August 25, 2023 10:29 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: A couple OSDs not starting after host reboot Hi, Thank you for your reply. I don’t think the device names changed, but ceph seems to be confused about which device the OSD is on. It’s reporting that there are 2 OSDs on the same device although this is not true. ceph device ls-by-host <osd-node> | grep sdu ATA_HGST_HUH728080ALN600_VJH4GLUX sdu osd.665 ATA_HGST_HUH728080ALN600_VJH60MAX sdu osd.657 The osd.665 is actually on device sdm. Could this be the cause of the issue? Is there a way to correct it? Thanks, Alison _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, It looks like Igor is right, it does appear to be a corruption. ls /var/lib/ceph/252fcf9a-b169-11ed-87be-3cecef623f33/osd.665/ ceph_fsid config fsid keyring ready require_osd_release type unit.configured unit.created unit.image unit.meta unit.poststop unit.run unit.stop whoami head -c 4096 /dev/ceph-febad5a5-ba44-41aa-a39e-b9897f757752/osd-block-87e548f4-b9b5-4ed8-aca8-de703a341a50 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 What could have caused this to happen? The ceph-bluestore-tool isn’t able to repair it. Do I need to remove the OSD and create a new one? Thanks, Alison

Hi chenhui, there is still a work in progress to support multiple labels to avoid the issue (https://github.com/ceph/ceph/pull/55374). But this is of little help for your current case. If your disk is fine (meaning it's able to read/write block at offset 0) you might want to try to recover the label using label from a different OSD sitting on a similar(!!that's important!!!) main device. One needs to update osd uuid, whoami and osd_key fields after copying though. Here is the step-by-step procedure: 1. Copy OSD label (4K data block at offset 0) from source OSD's main device to the same location on the broken one:

dd if=<source_osd_block_device> of=<target_osd_block_device> count=1 bs=4096

2. Learn broken OSD uuid, N denotes broken OSD id:

ceph report | grep '"osd": N' -A 1                 "osd": N,                 "uuid": "6a4ca4ab-6a43-473c-b09c-b13bdd9def5c",

3. Set obtained uuid to copied OSD osd label

ceph-bluestore-tool --dev <target_osd_block_device> --command set-label-key -k osd_uuid -v 6a4ca4ab-6a43-473c-b09c-b13bdd9def5c

4. Update whoami field in the copied label

ceph-bluestore-tool --dev <target_osd_block_device> --command set-label-key -k whoami -v N

5. learn osd's key

ceph auth ls | grep osd.1 -A 2 osd.1         key: AQDrvg9maKxvKxAAqAzqCeR6y0UqBSVIyDhppg==

6. Update osd_key field in the copied label

ceph-bluestore-tool --dev <target_osd_block_device> --command set-label-key -k osd_key -v AQDrvg9maKxvKxAAqAzqCeR6y0UqBSVIyDhppg==

7. Prime OSD dir if it's been lost:

ceph-bluestore-tool --dev <target_osd_block_device> --path <path-to-target-osd-folder> --command prime-osd-dir

At this point OSD should be able to start if corrupted label was the only problem. Hope this helps, Igor. On 05/04/2024 05:50, xu chenhui wrote:

Hi， Has there been any progress on this issue ? is there quick recover method? I have same problem with you that first 4k block of osd metadata is invalid. It will pay a heavy price to recreate osd.

Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 05/04/2024 17:28, xu chenhui wrote:

Hi, Igor

Thank you for providing the repair procedure. I will try it when I am back to my workstation. Can you provide any possible reasons for this problem? Unfortunately no. I recall a few cases like that but I doubt any one knows the root cause. ceph version: v16.2.5

You better upgrade to the latest pacific release.

error info: systemd[1]: Started Ceph osd.307 for 02eac9e0-d147-11ee-95de-f0b2b90ee048. bash[39068]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-307 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79 --path /var/lib/ceph/osd/ceph-307 --no-mon-config bash[39068]: stderr: failed to read label for /dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79: (2) No such file or directory bash[39068]: --> RuntimeError: command returned non-zero exit status: 1

2024-04-03T14:25:24.349+0000 7f90206fb3c0 10 bluestore(/dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79) _read_bdev_label 2024-04-03T14:25:24.349+0000 7f90206fb3c0 2 bluestore(/dev/ceph-6b69b64c-7293-4530-9e25-28279308198e/osd-block-81fbaf55-7de2-4f21-97bf-7d79f045ee79) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input

thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, Do slow ops impact data integrity => No Can I generally ignore it => No :) This means that some client transactions are blocked for 120 sec (that's a lot). This could be a lock on the client side (CephFS, essentially), an incident on the infrastructure side (a disk about to fall, network instability, etc.), ... When this happens, you need to look at the blocked requests. If you systematically see an osd ID, then look at dmesg and the SMART of the disk. This can also be an architectural problem (for example, high IOPS load with osdmap on HDD, all multiplied by the erasure code) *David* Le ven. 5 avr. 2024 à 19:42, adam.ther <adam.ther@aol.com> a écrit :

Hello,

Do slow ops impact data integrity or can I generally ignore it? I'm loading 3 hosts with a 10GB link and it saturating the disks or the OSDs.

2024-04-05T15:33:10.625922+0000 mon.CEPHADM-1 [WRN] Health check update: 3 slow ops, oldest one blocked for 117 sec, daemons [osd.0,osd.13,osd.14,osd.17,osd.3,osd.4,osd.9] have slow ops. (SLOW_OPS)

2024-04-05T15:33:15.628271+0000 mon.CEPHADM-1 [WRN] Health check update: 2 slow ops, oldest one blocked for 123 sec, daemons [osd.0,osd.1,osd.14,osd.17,osd.3,osd.4,osd.9] have slow ops. (SLOW_OPS)

I guess more to the point, what the impact here?

Thanks,

Adam

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Igor Fedotov wrote:

Hi chenhui,

there is still a work in progress to support multiple labels to avoid the issue (https://github.com/ceph/ceph/pull/55374). But this is of little help for your current case.

If your disk is fine (meaning it's able to read/write block at offset 0) you might want to try to recover the label using label from a different OSD sitting on a similar(!!that's important!!!) main device. One needs to update osd uuid, whoami and osd_key fields after copying though. Here is the step-by-step procedure:

1. Copy OSD label (4K data block at offset 0) from source OSD's main device to the same location on the broken one:

...

dd if=<source_osd_block_device> of=<target_osd_block_device> count=1 bs=4096

Hi, Igor I am repairing the lost label problem. But I have doubt about what's mean "similar main device"? Our OSD and disk have a one-to-one relationship, without disk partitions. How can I find source_osd_block_device ? thanks