Correct way to replace working OSD disk keeping the same OSD ID
Dear Ceph users, I'm struggling to unbderstand what is the correct way to remove a working disk and replace it (e.g. for a disk upgrade) while keeping the same OSD ID. I did this several times following this procedure: https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd every time facing a different issue. This time it happened that after removal the OSD was marked as destroyed, so I went on with disk replacement and then with: ceph orch apply osd --all-available-devices but this created a new OSD with a new ID instead of reusing the old one, which remained in destroyed state. I must be missing something but I cannot figure out what. Thanks, Nicola
Dear Ceph users, I'm struggling to unbderstand what is the correct way to remove a working disk and replace it (e.g. for a disk upgrade) while keeping the same OSD ID.
There may or may not be good guides for reaching this goal, but as a long time ceph user I can only say that you should not care for the OSD id. If you have a process that works well but gives you a new ID, just go with that. There are so many other things that are important when running a ceph cluster that you don't need to add extra work on yourself by inventing a "rule" that replacement drives must have the same ID. -- May the most significant bit of your life be positive.
Thanks for your insight. So if I remove an OSD without --replace its ID won't be reused when I e.g. add a new host with new disks? Even if I completely remove it from the cluster? I'm asking because I maintain a failure log per OSD and I'd like to avoid that an OSD previously in host A migrates to host B at a certain point. thanks again, Nicola
hi, in case i care to keep an osd id (up to now) i do ... ceph osd destroy <osd id> --yes-i-really-mean-it ... replace disk ... [ ceph-volume lvm zap --destroy /dev/<new disk> ] ceph-volume lvm prepare --bluestore --osd-id <osd id> --data /dev/<new disk> [ --block.db /dev/<some volume group/<some logical volume> ] [ --block.wal /dev/<some other volume group/<some other logical volume> ] ... retrieve <osd fsid> i.e. with ceph-volume lvm list ... ceph-volume lvm activate --bluestore <osd id> <osd fsid> ... up to now that seems to work fine cheers, toBias From: "Nicola Mori" <mori@fi.infn.it> To: "Janne Johansson" <icepic.dz@gmail.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Wednesday, 11 December, 2024 11:44:57 Subject: [ceph-users] Re: Correct way to replace working OSD disk keeping the same OSD ID Thanks for your insight. So if I remove an OSD without --replace its ID won't be reused when I e.g. add a new host with new disks? Even if I completely remove it from the cluster? I'm asking because I maintain a failure log per OSD and I'd like to avoid that an OSD previously in host A migrates to host B at a certain point. thanks again, Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I replaced another disk, this time everything worked as expected following this procedure: 1) Drain and destroy the OSD: ceph orch osd rm <ID> --replace 2) Replace the disk. 3) Zap the new disk: ceph orch device zap <host> /dev/sd<X> --force 4) Manually create the new OSD: ceph orch daemon add osd <host>:/dev/sd<X> 5) Adjust the CRUSH weight for the new disk size: ceph osd crush reweight osd.<ID> <weight> At point 1 an OSD in destroyed state is left in the cluster; it is automatically replaced by another OSD with the same ID and associated to the new disk at point 4. Thanks to everybody for the help and suggestions. Nicola
Ceph uses the next free ID available, so IDs definitely will be reused if you free them up at some point. I'm not sure why '--all-available-devices' would suddenly choose a different ID than the OSD had when you marked it as "destroyed". But I also don't use that 'all-available-devices' flag in production clusters, only for testing. I like to have control over OSD specs, and if I replace a HDD with shared rocksDB, I don't want Ceph to deploy a standalone HDD OSD. BTW, if you already applied 'ceph orch apply osd --all-available-devices' once, you don't need to apply it again, that spec is stored. Do you have other OSD specs in place? What does 'ceph orch ls osd --export' show? Zitat von Nicola Mori <mori@fi.infn.it>:
Thanks for your insight. So if I remove an OSD without --replace its ID won't be reused when I e.g. add a new host with new disks? Even if I completely remove it from the cluster? I'm asking because I maintain a failure log per OSD and I'd like to avoid that an OSD previously in host A migrates to host B at a certain point. thanks again,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm struggling to unbderstand what is the correct way to remove a working disk and replace it (e.g. for a disk upgrade) while keeping the same OSD ID. I did this several times following this procedure:
https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd
every time facing a different issue. This time it happened that after removal the OSD was marked as destroyed, so I went on with disk replacement and then with:
ceph orch apply osd --all-available-devices
but this created a new OSD with a new ID instead of reusing the old one, which remained in destroyed state.
I used to ask this question also. But there is no need to have it the same number. You should not care about this. Having said that, last time I checked it is just using correct number. This is from old manual I have, where you can set the number sudo -u ceph ceph-osd -i 14 --mkfs --osd-uuid
participants (5)
-
Eugen Block
-
Janne Johansson
-
Marc
-
Nicola Mori
-
tobias tempel