ceph orch osd rm --zap --replace leaves cluster in odd state
Hi, I want to prepare a failed disk for replacement. I did: ceph orch osd rm 35 --zap --replace and it's now in the state "Done, waiting for purge", with 0 pgs, and REPLACE and ZAP set to true. It's been like this for some hours, and now my cluster is unhappy: [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.35 on host moss-be1002 not managed by cephadm (the OSD is down & out) ...and also neither the disk nor the relevant NVME LV has been zapped. I have my OSDs deployed via a spec: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" And before issuing the ceph orch osd rm I set that to be unmanaged (ceph orch set-unmanaged osd.rrd_single_NVMe), as obviously I don't want ceph to just try and re-make a new OSD on the sad disk. I'd expected from the docs[0] that what I did would leave me with a system ready for the failed disk to be swapped (and that I could then mark osd.rrd_single_NVMe as managed again, and a new OSD built), including removing/wiping the NVME lv so it can be removed. What did I do wrong? I don't much care about the OSD id (but obviously it's neater to not just incrementally increase OSD numbers every time a disk died), but I thought that telling ceph orch not to make new OSDs then using ceph orch osd rm to zap the disk and NVME lv would have been the way to go... Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd
What is the state of your PGs? could you post "ceph -s" I believe (but a bit of an assumption after encountering something similar myself) that under the hood cephadm is using the "ceph osd safe-to-destroy osd.X" command and when OSD.X is no longer running and not all PGs are active+clean (for instance in a active+remapped state) the safe-to-destroy command will return in the negative with the response "OSD.X not reporting stats, not all PGs are active+clean, cannot draw any conclusions" or some such msg. The cephadm osd removal would stall in that state until all PGs reach active+clean. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Tue, May 28, 2024 at 11:43 AM Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi,
I want to prepare a failed disk for replacement. I did: ceph orch osd rm 35 --zap --replace
and it's now in the state "Done, waiting for purge", with 0 pgs, and REPLACE and ZAP set to true. It's been like this for some hours, and now my cluster is unhappy:
[WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.35 on host moss-be1002 not managed by cephadm
(the OSD is down & out)
...and also neither the disk nor the relevant NVME LV has been zapped.
I have my OSDs deployed via a spec: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe"
And before issuing the ceph orch osd rm I set that to be unmanaged (ceph orch set-unmanaged osd.rrd_single_NVMe), as obviously I don't want ceph to just try and re-make a new OSD on the sad disk.
I'd expected from the docs[0] that what I did would leave me with a system ready for the failed disk to be swapped (and that I could then mark osd.rrd_single_NVMe as managed again, and a new OSD built), including removing/wiping the NVME lv so it can be removed.
What did I do wrong? I don't much care about the OSD id (but obviously it's neater to not just incrementally increase OSD numbers every time a disk died), but I thought that telling ceph orch not to make new OSDs then using ceph orch osd rm to zap the disk and NVME lv would have been the way to go...
Thanks,
Matthew
[0] https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 28/05/2024 17:07, Wesley Dillingham wrote:
What is the state of your PGs? could you post "ceph -s"
PGs all good: root@moss-be1001:/# ceph -s cluster: id: d7849d66-183c-11ef-b973-bc97e1bb7c18 health: HEALTH_WARN 1 stray daemon(s) not managed by cephadm services: mon: 3 daemons, quorum moss-be1001,moss-be1003,moss-be1002 (age 6d) mgr: moss-be1001.yibskr(active, since 6d), standbys: moss-be1003.rwdjgw osd: 48 osds: 47 up (since 2d), 47 in (since 2d) data: pools: 1 pools, 1 pgs objects: 6 objects, 19 MiB usage: 4.2 TiB used, 258 TiB / 263 TiB avail pgs: 1 active+clean The OSD is marked as "destroyed" in the osd tree: root@moss-be1001:/# ceph osd tree | grep -E '^35' 35 hdd 3.75999 osd.35 destroyed 0 1.00000 root@moss-be1001:/# ceph osd safe-to-destroy osd.35 ; echo $? OSD(s) 35 are safe to destroy without reducing data durability. 0 I should have said - this is a reef 18.2.2 cluster, cephadm deployed. Regards, Matthew
I guess it has something to do with your just one pg Oo But not sure how to move forward. Did you Revolver your issue? Greetings Mehmet Am 28. Mai 2024 18:12:35 MESZ schrieb Matthew Vernon <mvernon@wikimedia.org>:
On 28/05/2024 17:07, Wesley Dillingham wrote:
What is the state of your PGs? could you post "ceph -s"
PGs all good:
root@moss-be1001:/# ceph -s cluster: id: d7849d66-183c-11ef-b973-bc97e1bb7c18 health: HEALTH_WARN 1 stray daemon(s) not managed by cephadm
services: mon: 3 daemons, quorum moss-be1001,moss-be1003,moss-be1002 (age 6d) mgr: moss-be1001.yibskr(active, since 6d), standbys: moss-be1003.rwdjgw osd: 48 osds: 47 up (since 2d), 47 in (since 2d)
data: pools: 1 pools, 1 pgs objects: 6 objects, 19 MiB usage: 4.2 TiB used, 258 TiB / 263 TiB avail pgs: 1 active+clean
The OSD is marked as "destroyed" in the osd tree:
root@moss-be1001:/# ceph osd tree | grep -E '^35' 35 hdd 3.75999 osd.35 destroyed 0 1.00000
root@moss-be1001:/# ceph osd safe-to-destroy osd.35 ; echo $? OSD(s) 35 are safe to destroy without reducing data durability. 0
I should have said - this is a reef 18.2.2 cluster, cephadm deployed.
Regards,
Matthew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
ceph＠elchaka.de
-
Matthew Vernon
-
Wesley Dillingham