Is it possible (or meaningful) to revive old OSDs?
Hello, I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since. I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start. Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes". I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster? My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard? [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-os... [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-a... Many thanks!
Hi ceph-mail@rikdvk.mailer.me, you could squeeze the OSDs back in but it does not make sense. Just clean the disks with dd for example and add them as new disks to your cluster. Best, Malte Am 04.09.23 um 09:39 schrieb ceph-mail@rikdvk.mailer.me:
Hello,
I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since.
I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start.
Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes".
I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster?
My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard?
[1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-os... [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-a...
Many thanks!
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Resurrection usually only makes sense if fate or a certain someone resulted in enough overlapping removed OSDs that you can't meet min_size. I've had to a couple of times :-/ If an OSD is down for more than a short while, backfilling a redeployed OSD will likely be faster than waiting for it to peer and do deltas -- if it can at all.
On Sep 6, 2023, at 10:16, Malte Stroem <malte.stroem@gmail.com> wrote:
Hi ceph-mail@rikdvk.mailer.me,
you could squeeze the OSDs back in but it does not make sense.
Just clean the disks with dd for example and add them as new disks to your cluster.
Best, Malte
Am 04.09.23 um 09:39 schrieb ceph-mail@rikdvk.mailer.me:
Hello, I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since. I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start. Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes". I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster? My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard? [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-os... [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-a... Many thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, I agree with Anthony. If your cluster is healthy and you don't *need* to bring them back in it's going to be less work and time to just deploy them as new. I usually set norebalance, purge the osds in ceph, remove the vg from the disks and re-deploy. Then unset norebalance at the end once everything is peered and happy. This is so that it doesn't start moving stuff around when you purge. Rich On Thu, 7 Sept 2023 at 02:21, Anthony D'Atri <anthony.datri@gmail.com> wrote:
Resurrection usually only makes sense if fate or a certain someone resulted in enough overlapping removed OSDs that you can't meet min_size. I've had to a couple of times :-/
If an OSD is down for more than a short while, backfilling a redeployed OSD will likely be faster than waiting for it to peer and do deltas -- if it can at all.
On Sep 6, 2023, at 10:16, Malte Stroem <malte.stroem@gmail.com> wrote:
Hi ceph-mail@rikdvk.mailer.me,
you could squeeze the OSDs back in but it does not make sense.
Just clean the disks with dd for example and add them as new disks to your cluster.
Best, Malte
Am 04.09.23 um 09:39 schrieb ceph-mail@rikdvk.mailer.me:
Hello, I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since. I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start. Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes". I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster? My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard? [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-os... [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-a... Many thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks all for the advice, very helpful! The node also had a mon, which happily slotted right back into the cluster. The node's been up and running for a number of days now, but the systemd OSD processes don't seem to be trying continously, they're never progressing or getting a newer map. As mentioned, the cluster is otherwise healthy (only these OSDs, which are down and out), and I have spare capacity and no issue with min_size. And they've been out for a long time (months) so it's reasonable to guess that most PGs may have been touched. So, based on the advice, my plan is the following: 1. Set norebalance 2. One by one, do this for each OSD * Purge the OSD from the dashboard * cephadm ceph-volume lvm zap * cephadm may automatically find and add the OSD, otherwise I'll add it manually 3. use pgremapper<https://github.com/digitalocean/pgremapper> to prevent the OSDs to be filled 4. unset norebalance 5. Let the balancer gently flow data back into the OSDs over the next hours, days, weeks. Thanks all! ________________________________ From: Richard Bade 'hitrich at gmail.com' <ceph-mail@rikdvk.mailer.me> Sent: Thursday, September 7, 2023 01:25 To: ceph-mail@rikdvk.mailer.me <ceph-mail@rikdvk.mailer.me> Subject: Re: [ceph-users] Re: Is it possible (or meaningful) to revive old OSDs? Yes, I agree with Anthony. If your cluster is healthy and you don't *need* to bring them back in it's going to be less work and time to just deploy them as new. I usually set norebalance, purge the osds in ceph, remove the vg from the disks and re-deploy. Then unset norebalance at the end once everything is peered and happy. This is so that it doesn't start moving stuff around when you purge. Rich On Thu, 7 Sept 2023 at 02:21, Anthony D'Atri <anthony.datri@gmail.com> wrote:
Resurrection usually only makes sense if fate or a certain someone resulted in enough overlapping removed OSDs that you can't meet min_size. I've had to a couple of times :-/
If an OSD is down for more than a short while, backfilling a redeployed OSD will likely be faster than waiting for it to peer and do deltas -- if it can at all.
On Sep 6, 2023, at 10:16, Malte Stroem <malte.stroem@gmail.com> wrote:
Hi ceph-mail@rikdvk.mailer.me,
you could squeeze the OSDs back in but it does not make sense.
Just clean the disks with dd for example and add them as new disks to your cluster.
Best, Malte
Am 04.09.23 um 09:39 schrieb ceph-mail@rikdvk.mailer.me:
Hello, I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since. I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start. Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes". I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster? My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard? [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-os... [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-a... Many thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I did something like that in the past. If you have a sufficient amount of cold data in general and you can bring the OSDs back with their original IDs, recovery was significantly faster than rebalancing. It really depends how trivial the version update per object is. In my case it could re-use thousands of clean objects per dirty object. If you are unsure its probably best to do a wipe + rebalance. What can take quite a while at the beginning is the osdmap update if they were down for such a long time. The first boot until they show up as "in" will take a while. Set norecover and norebalance until you see in the OSD log that they have the latest OSD map version. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Malte Stroem <malte.stroem@gmail.com> Sent: Wednesday, September 6, 2023 4:16 PM To: ceph-mail@rikdvk.mailer.me; ceph-users@ceph.io Subject: [ceph-users] Re: Is it possible (or meaningful) to revive old OSDs? Hi ceph-mail@rikdvk.mailer.me, you could squeeze the OSDs back in but it does not make sense. Just clean the disks with dd for example and add them as new disks to your cluster. Best, Malte Am 04.09.23 um 09:39 schrieb ceph-mail@rikdvk.mailer.me:
Hello,
I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since.
I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start.
Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes".
I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster?
My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard?
[1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-os... [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-a...
Many thanks!
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
ceph-mail＠rikdvk.mailer.me
-
Frank Schilder
-
Malte Stroem
-
Richard Bade