OSD delete vs destroy vs purge
Dear Ceph users, I see that the OSD page of the Ceph dashboard offers three possibilities for "removing" an OSD: delete, destroy and purge. The delete operation has the possibility to flag the "Preserve OSD ID(s) for replacement." option. I searched for explanations of the differences between the three commands but I didn't find anything definitive, so I'd need some help with this. Thanks in advance, Nicola
Hi, I'll try to summarize as far as I understand the process, please correct me if I'm wrong. - delete: drain and then delete (optionally keep OSD ID) - destroy: mark as destroyed (to re-use OSD ID) - purge: remove everything I would call the "delete" option in the dashboard as a "safe delete", the OSD is only removed if it's "safe-to-destroy" which you can also check via CLI: quincy-1:~ # ceph osd safe-to-destroy 0 Error EBUSY: OSD(s) 0 have 20 pgs currently mapped to them. Only if the OSD has been drained successfully it will be eventually "purged" or marked as "destroyed" so you can re-use the ID (depends on the checkbox to preserve the ID). Here's some example from a Reef test cluster (don't mind the quincy hostnames), I deleted the OSD without wanting to replace it (note the "replace: False" output): ---snip--- [dashboard INFO controllers.osd] Start removing osd.0 (replace: False)... [dashboard INFO controllers.osd] Current removing OSDs [osd.0] [dashboard INFO controllers.osd] Wait until osd.0 is removed... [dashboard INFO request] [::ffff:<IP>:51896] [DELETE] [202] [2.011s] [admin] [68.0B] /api/osd/0 [cephadm INFO root] osd.0 crush weight is 0.009796142578125 [cephadm INFO root] osd.0 weight is now 0.0 [cephadm INFO root] osd.0 now down [cephadm INFO cephadm.serve] Removing daemon osd.0 from quincy-2 -- ports [] [cephadm INFO cephadm.services.cephadmservice] Removing key for osd.0 [cephadm INFO cephadm.services.osd] Successfully removed osd.0 on quincy-2 [cephadm INFO cephadm.services.osd] Successfully purged osd.0 on quincy-2 ---snip--- So it reweights the OSD to 0 (drained) and then purges it, so in the end "delete" does the same as purge (delete key and remove it from the crush tree), just safely with draining. If an OSD is already down and out and has no PGs anymore you also have the dashboard options to either "destroy" it (marked "destroyed" in the crush tree so you can re-use the ID) or "purge" it (everything is removed). In the CLI you can directly choose "destroy" or "purge" if you don't want to wait for the draining (be careful!). I hope that clears it up a bit. Regards, Eugen Zitat von Nicola Mori <mori@fi.infn.it>:
Dear Ceph users,
I see that the OSD page of the Ceph dashboard offers three possibilities for "removing" an OSD: delete, destroy and purge. The delete operation has the possibility to flag the "Preserve OSD ID(s) for replacement." option. I searched for explanations of the differences between the three commands but I didn't find anything definitive, so I'd need some help with this. Thanks in advance,
Nicola
Thanks Eugen for the explanation. To summarize what I understood: - delete from GUI simply does a drain+destroy; - destroy will preserve the OSD id so that it will be used by the next OSD that will be created on that host; - purge will remove everything, and the next OSD that will be crated will use a brand new id. It all seems clear now, but please correct me if I'm wrong, and thanks again.
Thanks Eugen for the explanation. To summarize what I understood: - delete from GUI simply does a drain+destroy; - destroy will preserve the OSD id so that it will be used by the next OSD that will be created on that host; - purge will remove everything, and the next OSD that will be created will use a brand new id.
Unless something has changed in recent releases, there are some nuances: Destroying an OSD leaves the OSD ID in the CRUSH map; it may be redeployed with the same ID when done so explicitly (ideally on the same device/slot). That ID will not unless I’m mistaken be used by any new OSD deployment unless specified explicitly. In the general case, OSD creation picks the lowest-numbered unused ID.
Yeah, that's basically it, also taking into account Anthony's response, of course. Zitat von Nicola Mori <mori@fi.infn.it>:
Thanks Eugen for the explanation. To summarize what I understood: - delete from GUI simply does a drain+destroy; - destroy will preserve the OSD id so that it will be used by the next OSD that will be created on that host; - purge will remove everything, and the next OSD that will be crated will use a brand new id.
It all seems clear now, but please correct me if I'm wrong, and thanks again.
participants (3)
-
Anthony D'Atri
-
Eugen Block
-
Nicola Mori