How to remove failed OSD & reuse it?
Hi guys. I've browsing through the net in a search of a relatively clear "howto" but I failed to find one. It's rather many, sometimes different notes/thoughts on how to deal with such/similar situation. Having a 3-node containerized cluster which lost osd - it crushed, there is nothing wrong with the node, nothing wrong with the disk, but never mind that. Is there a howto which covers containerized environment? One example I followed is: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/1.2.3/html/red... but it is - to me - clear, what to do with "broken" containers. I'm got to: -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.68359 root default -3 0 host podster1 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000 yet: -> $ ceph orch ps --daemon-type=osd NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.0 podster1.mine.priv error 7m ago 3w - 4096M <unknown> <unknown> <unknown> osd.1 podster3.mine.priv running (25h) 7m ago 3w 942M 4096M 19.2.3 aade1b12b8e6 d71051ea79dc osd.2 podster2.mine.priv running (6d) 7m ago 3w 1192M 4096M 19.2.3 aade1b12b8e6 e8d05142a73a osd.3 podster1.mine.priv error 7m ago 2w - 4096M <unknown> <unknown> <unknown> osd.4 podster2.mine.priv running (6d) 7m ago 2w 3293M 4096M 19.2.3 aade1b12b8e6 6116277f69d1 osd.5 podster3.mine.priv running (25h) 7m ago 2w 2963M 4096M 19.2.3 aade1b12b8e6 d671bf73cc01 what would be next bits needed to complete such removal&reuse/re-create of osd(s)? p.s. This a 'lab' setup so I'm not worried, but it'd be great to complete this process in a healthy manner. many thanks, L.
Hi, the docs [0] contain the OSD removal process: ceph orch osd rm <osd_id(s)> [--replace] [--force] [--zap] So in your case I'd just remove and zap the faulty OSDs: ceph orch osd rm 0 3 --force --zap If you have a managed OSD spec matching your setup, the orchestrator will simply redeploy OSDs on the wiped disks. Of course, in a production environment you'd need to be sure if it's safe to wipe an OSD. So maybe try without --force first to see if it will result in inactive PGs. Now that those two OSDs are already dead, there's no real danger here, but I just wanted to mention it. You can also start with one OSD and see if the process works for you. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#remove-an-osd Zitat von lejeczek <peljasz@yahoo.co.uk>:
Hi guys.
I've browsing through the net in a search of a relatively clear "howto" but I failed to find one. It's rather many, sometimes different notes/thoughts on how to deal with such/similar situation. Having a 3-node containerized cluster which lost osd - it crushed, there is nothing wrong with the node, nothing wrong with the disk, but never mind that. Is there a howto which covers containerized environment? One example I followed is: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/1.2.3/html/red... but it is - to me - clear, what to do with "broken" containers. I'm got to: -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.68359 root default -3 0 host podster1 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000
yet: -> $ ceph orch ps --daemon-type=osd NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.0 podster1.mine.priv error 7m ago 3w - 4096M <unknown> <unknown> <unknown> osd.1 podster3.mine.priv running (25h) 7m ago 3w 942M 4096M 19.2.3 aade1b12b8e6 d71051ea79dc osd.2 podster2.mine.priv running (6d) 7m ago 3w 1192M 4096M 19.2.3 aade1b12b8e6 e8d05142a73a osd.3 podster1.mine.priv error 7m ago 2w - 4096M <unknown> <unknown> <unknown> osd.4 podster2.mine.priv running (6d) 7m ago 2w 3293M 4096M 19.2.3 aade1b12b8e6 6116277f69d1 osd.5 podster3.mine.priv running (25h) 7m ago 2w 2963M 4096M 19.2.3 aade1b12b8e6 d671bf73cc01
what would be next bits needed to complete such removal&reuse/re-create of osd(s)? p.s. This a 'lab' setup so I'm not worried, but it'd be great to complete this process in a healthy manner. many thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
at this point - after having followed that Redhat's doc - as I showed with 'tree' results, with the 'orch' bits I get: -> $ ceph orch osd rm 0 3 --force --zap Unable to find OSDs: ['0', '3']
Oh right, in that case you'll have to clean up manually. On the affected host run cephadm ls --no-detail | grep osd This should show you the remainders of the faulty OSDs. Then run cephadm rm-daemon --name osd.0 --force --fsid {YOUR_CLUSTER_FSID} cephadm rm-daemon --name osd.3 --force --fsid {YOUR_CLUSTER_FSID} Check with 'podman|docker ps' if there are dead pods running and kill them, just to be sure. Wait a bit, the orchestrator should clean up 'ceph orch ps' output. Then you can zap the drives either with: ceph orch device zap <host> /dev/sdX --force or locally on the host with: cephadm ceph-volume lvm zap --destroy /dev/sdX Then set your OSD spec to managed and the orchestrator will redeploy OSDs. Zitat von lejeczek <peljasz@yahoo.co.uk>:
at this point - after having followed that Redhat's doc - as I showed with 'tree' results, with the 'orch' bits I get: -> $ ceph orch osd rm 0 3 --force --zap Unable to find OSDs: ['0', '3'] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm bit confused with 'un/manged' I had earlier - although I exec it after 'zapping' disk -> $ ceph orch apply osd --all-available-devices so ceph automatically re-deployed two "new" osd on that "failed" host. Now everything is 'HEALTH_OK' yet still: -> $ ceph orch ls | egrep osd osd 4 6m ago - <unmanaged> osd.all-available-devices 2 3m ago 28m * How does one square these things together? What is happening here? Where/how actually 'managed' osd happens (also with config setting/showing) takes place? many thanks, L.
If you look at ceph orch ls osd --export you'll see the details of both OSD specs you have in place, incl. an "unmanaged: true" statement for "osd". The existing OSDs were apparently deployed by that spec, but the newly built OSDs were deployed by your "osd.all-available-devices" spec. I'd recommend to clean that up (and only use all-available-devices if you're aware of the impact in prod). You could set the "osd" spec to managed again: ceph orch set-managed osd Then disable the all-available-devices: ceph orch set-unmanaged osd.all-available-devices Then remove the new OSDs again (and zap them) and let the remaining managed spec recreate them. There's an easier way by editing the unit.meta file, but this might be a good opportunity to learn how to deal with the spec files. Zitat von lejeczek <peljasz@yahoo.co.uk>:
I'm bit confused with 'un/manged' I had earlier - although I exec it after 'zapping' disk -> $ ceph orch apply osd --all-available-devices so ceph automatically re-deployed two "new" osd on that "failed" host. Now everything is 'HEALTH_OK' yet still: -> $ ceph orch ls | egrep osd osd 4 6m ago - <unmanaged> osd.all-available-devices 2 3m ago 28m *
How does one square these things together? What is happening here? Where/how actually 'managed' osd happens (also with config setting/showing) takes place?
many thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did not look at it earlier, but now: -> $ ceph orch ls ... osd 4 3m ago - <unmanaged> ...
but, now checking cmd history, I had tried that: -> $ ceph orch osd rm 3 --zap Scheduled OSD(s) for removal and ceph waited, I thought was stuck with: -> $ ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 3 podster1.mine.priv done, waiting for purge 0 False False True
participants (2)
-
Eugen Block
-
lejeczek