Hi! An OSD failed in our 16.2.15 cluster. I prepared it for removal and ran `ceph orch daemon rm osd.19 --force`. Somehow that didn't work as expected, so now we still have osd.19 in the crush map: -10 122.66965 host ceph02 19 1.00000 osd.19 down 0 1.00000 But OSD has been cleaned up on the host, although incompletely, as both block and block.db LVs still exist. If I try to remove the OSD again, I get an error: # ceph orch daemon rm osd.19 --force Error EINVAL: Unable to find daemon(s) ['osd.19'] How can I clean up this OSD and get rid of it completely, including the crush map? I would appreciate any suggestions or pointers. Best regards, Zakhar
I ended up manually cleaning up the OSD host, removing stale LVs and DM entries, and then purging the OSD with `ceph osd purge osd.19`. Looks like it's gone for good. /Z On Sat, 4 May 2024 at 08:29, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi!
An OSD failed in our 16.2.15 cluster. I prepared it for removal and ran `ceph orch daemon rm osd.19 --force`. Somehow that didn't work as expected, so now we still have osd.19 in the crush map:
-10 122.66965 host ceph02 19 1.00000 osd.19 down 0 1.00000
But OSD has been cleaned up on the host, although incompletely, as both block and block.db LVs still exist.
If I try to remove the OSD again, I get an error:
# ceph orch daemon rm osd.19 --force Error EINVAL: Unable to find daemon(s) ['osd.19']
How can I clean up this OSD and get rid of it completely, including the crush map? I would appreciate any suggestions or pointers.
Best regards, Zakhar
Hi, On 04.05.24 07:29, Zakhar Kirpichenko wrote:
If I try to remove the OSD again, I get an error:
# ceph orch daemon rm osd.19 --force Error EINVAL: Unable to find daemon(s) ['osd.19']
The command to remove and OSD in an cephadm orchestrated cluster is ceph orch osd rm 19 as per https://docs.ceph.com/en/reef/cephadm/services/osd/#remove-an-osd This will make sure that the OSD is not needed any more (data is drained etc). Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
participants (2)
-
Robert Sander
-
Zakhar Kirpichenko