Is it safe to set multiple OSD out across multiple failure domain?
Say we have 10 host with 10 OSDs and the failure domain is host. host0 osd 0 to 9 host1 osd 10 to 19 host2 osd 20 to 29 host3 osd 30 to 39 host4 osd 40 to 49 host5 osd 50 to 59 host6 osd 60 to 69 host7 osd 70 to 79 host8 osd 80 to 89 host9 osd 90 to 99 A pool have EC 4+2 and PG 2.1 has UP and ACTING the following OSDs 0,10,20,30,40 and 50. I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out? I would think and hope that Ceph do the right thing and the PG remain accessible since all OSD is available and data is being copied to other OSDs. -- Kai Stian Olstad
Hello Kai, "ceph osd out" is always safe, as the outed OSD continues to serve PGs that have not yet been migrated away from it. Before stopping osd.123, check that it indeed holds zero PGs: ceph osd safe-to-destroy 123 On Wed, Mar 12, 2025 at 6:41 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
Say we have 10 host with 10 OSDs and the failure domain is host.
host0 osd 0 to 9 host1 osd 10 to 19 host2 osd 20 to 29 host3 osd 30 to 39 host4 osd 40 to 49 host5 osd 50 to 59 host6 osd 60 to 69 host7 osd 70 to 79 host8 osd 80 to 89 host9 osd 90 to 99
A pool have EC 4+2 and PG 2.1 has UP and ACTING the following OSDs 0,10,20,30,40 and 50.
I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out?
I would think and hope that Ceph do the right thing and the PG remain accessible since all OSD is available and data is being copied to other OSDs.
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Den ons 12 mars 2025 kl 11:41 skrev Kai Stian Olstad <ceph+list@olstad.com>:
Say we have 10 host with 10 OSDs and the failure domain is host.
host0 osd 0 to 9 host1 osd 10 to 19 host2 osd 20 to 29 host3 osd 30 to 39 host4 osd 40 to 49 host5 osd 50 to 59 host6 osd 60 to 69 host7 osd 70 to 79 host8 osd 80 to 89 host9 osd 90 to 99
A pool have EC 4+2 and PG 2.1 has UP and ACTING the following OSDs 0,10,20,30,40 and 50.
I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out?
It is not safe. You have no control of where ALL the PGs place their pieces, so if you take out 3 OSDs immediately, by random chance, there will be one PG for which osd 0,10 and 30 holds a piece for it, and now EC can't piece together a 4+2 PG, since it only has 3 pieces, where it needs 4 in order to rebuild the data. Or counted inversely, EC X+2 means at the very most 2 drives can be lost before you lose data. It will also turn READONLY while rebuilding the two missing PGs, and turn RW again when at least one piece is recreated, but in terms of "can I remove 3 drives on EC 4+2" the answer is no. What you can do is lower the weight of them to 0.0 and ceph will start move PGs off them into the others, and when they have 0 PGs, you can purge them and stop the OSD process and remove the disk. But you must let ceph move the data off the OSDs first. -- May the most significant bit of your life be positive.
On Wed, Mar 12, 2025 at 9:18 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den ons 12 mars 2025 kl 11:41 skrev Kai Stian Olstad <ceph+list@olstad.com>:
Say we have 10 host with 10 OSDs and the failure domain is host.
host0 osd 0 to 9 host1 osd 10 to 19 host2 osd 20 to 29 host3 osd 30 to 39 host4 osd 40 to 49 host5 osd 50 to 59 host6 osd 60 to 69 host7 osd 70 to 79 host8 osd 80 to 89 host9 osd 90 to 99
A pool have EC 4+2 and PG 2.1 has UP and ACTING the following OSDs 0,10,20,30,40 and 50.
I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out?
It is not safe. You have no control of where ALL the PGs place their pieces, so if you take out 3 OSDs immediately, by random chance, there will be one PG for which osd 0,10 and 30 holds a piece for it, and now EC can't piece together a 4+2 PG, since it only has 3 pieces, where it needs 4 in order to rebuild the data. Or counted inversely, EC X+2 means at the very most 2 drives can be lost before you lose data. It will also turn READONLY while rebuilding the two missing PGs, and turn RW again when at least one piece is recreated, but in terms of "can I remove 3 drives on EC 4+2" the answer is no.
What you can do is lower the weight of them to 0.0 and ceph will start move PGs off them into the others, and when they have 0 PGs, you can purge them and stop the OSD process and remove the disk.
But you must let ceph move the data off the OSDs first.
Please note that the original post talks about not yanking OSD drives physically, but about running the "ceph osd out" command. This command is exactly equivalent to reweighing the OSD to zero. -- Alexander Patrakov
Den ons 12 mars 2025 kl 17:12 skrev Alexander Patrakov <patrakov@gmail.com>:
I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out?
It is not safe. [...] What you can do is lower the weight of them to 0.0 and ceph will start move PGs off them into the others, and when they have 0 PGs, you can purge them and stop the OSD process and remove the disk.
Please note that the original post talks about not yanking OSD drives physically, but about running the "ceph osd out" command. This command is exactly equivalent to reweighing the OSD to zero.
Yes, my bad. I misread the "ceph osd out" as "taking out". You are right, telling ceph via "ceph osd out" or reweighting it to zero is the same, and both works to make the cluster start emptying those OSDs. -- May the most significant bit of your life be positive.
On Wed, Mar 12, 2025 at 06:32:09PM +0100, Janne Johansson wrote:
Den ons 12 mars 2025 kl 17:12 skrev Alexander Patrakov <patrakov@gmail.com>:
I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out?
It is not safe. [...] What you can do is lower the weight of them to 0.0 and ceph will start move PGs off them into the others, and when they have 0 PGs, you can purge them and stop the OSD process and remove the disk.
Please note that the original post talks about not yanking OSD drives physically, but about running the "ceph osd out" command. This command is exactly equivalent to reweighing the OSD to zero.
Yes, my bad. I misread the "ceph osd out" as "taking out". You are right, telling ceph via "ceph osd out" or reweighting it to zero is the same, and both works to make the cluster start emptying those OSDs.
The reason I asked is that several months back I got a off list reply from a frequent poster on this list, that setting 3 OSDs out at the same time could give me incomplete PGs as a result. But a least now I have 2 saying is OK and 1 saying it's not so thank you Alexander and Janne. -- Kai Stian Olstad
I need to take 3 of them, 0, 10 and 30, out, is it safe to run out on all 3 OSDs at the same time with "ceph osd out 0 10 20" or do I need to take one after the other out?
It is not safe. [...] What you can do is lower the weight of them to 0.0 and ceph will start move PGs off them into the others, and when they have 0 PGs, you can purge them and stop the OSD process and remove the disk.
Please note that the original post talks about not yanking OSD drives physically, but about running the "ceph osd out" command. This command is exactly equivalent to reweighing the OSD to zero.
Yes, my bad. I misread the "ceph osd out" as "taking out". You are right, telling ceph via "ceph osd out" or reweighting it to zero is the same, and both works to make the cluster start emptying those OSDs.
The reason I asked is that several months back I got a off list reply from a frequent poster on this list, that setting 3 OSDs out at the same time could give me incomplete PGs as a result.
But a least now I have 2 saying is OK and 1 saying it's not so thank you Alexander and Janne.
Perhaps this person mistook your question same as I did first, since "removing" or "destroying" or "physically taking out" would all be bad, whereas telling the cluster to stop using them at its earliest convenience and waiting a bit (sometimes days) for it to finish is totally fine. It does depend on how you formulate the "action" verbs I guess. I wanted to make sure it was understood that given a large set of PGs, one or more of them will be exactly on those 3 OSDs, so this PG needs to move away in an orderly fashion, otherwise you will get a broken pool and clients that sooner or later will have stuck IO. -- May the most significant bit of your life be positive.
It’s context-dependent, where the OSDs are. If they’re all in the same failure domain it’s safe, if you have capacity to recover into. Across failure domains, usually not.
The reason I asked is that several months back I got a off list reply from a frequent poster on this list, that setting 3 OSDs out at the same time could give me incomplete PGs as a result.
But a least now I have 2 saying is OK and 1 saying it's not so thank you Alexander and Janne.
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Before I replied, I wanted to renew my confidence and do a small test in a lab environment. I also created a k4m2 pool with host as failure-domain, started to write data chunks into it in a while loop and then marked three of the OSDs "out" simultaneously. After a few seconds of repeering backfill kicks in, I/O to the pool continues without interruption. So yeah, I also think it's safe to mark them out at the same time. Zitat von Anthony D'Atri <aad@dreamsnake.net>:
It’s context-dependent, where the OSDs are. If they’re all in the same failure domain it’s safe, if you have capacity to recover into. Across failure domains, usually not.
The reason I asked is that several months back I got a off list reply from a frequent poster on this list, that setting 3 OSDs out at the same time could give me incomplete PGs as a result.
But a least now I have 2 saying is OK and 1 saying it's not so thank you Alexander and Janne.
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Mar 17, 2025 at 03:08:54PM +0000, Eugen Block wrote:
Before I replied, I wanted to renew my confidence and do a small test in a lab environment. I also created a k4m2 pool with host as failure-domain, started to write data chunks into it in a while loop and then marked three of the OSDs "out" simultaneously. After a few seconds of repeering backfill kicks in, I/O to the pool continues without interruption. So yeah, I also think it's safe to mark them out at the same time.
That is just awesome that you tested it Eugen, thank you very much. -- Kai Stian Olstad
On Tue, Mar 18, 2025, 5:02 PM Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Mon, Mar 17, 2025 at 03:08:54PM +0000, Eugen Block wrote:
Before I replied, I wanted to renew my confidence and do a small test in a lab environment. I also created a k4m2 pool with host as failure-domain, started to write data chunks into it in a while loop and then marked three of the OSDs "out" simultaneously. After a few seconds of repeering backfill kicks in, I/O to the pool continues without interruption. So yeah, I also think it's safe to mark them out at the same time.
That is just awesome that you tested it Eugen, thank you very much.
-- Kai Stian Olstad _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can do it as others have said. The risks are: - If OSDs fail while being outed before they have a chance to fully offload their PGs, it's still the same as stopping OSDs in different CRUSH domains. IOW, you are chancing fate a bit if you wait for a bunch of drives to pickup soft failures and then weight them out simultaneously. You are really only "safe" again and can stop OR let OSDs fail once PGs are all active+clean. - You must have enough free space and enough OSDs to inherit all the additional PGs. If you get stuck in a backfill_toofull situation, out so many OSDs that cause the PGs to exceed per OSD limits, etc... you can quickly find yourself in a pickle. Tyler
participants (6)
-
Alexander Patrakov
-
Anthony D'Atri
-
Eugen Block
-
Janne Johansson
-
Kai Stian Olstad
-
Tyler Stachecki