Theory about min_size and its implications
Hi! it is unclear for us what min_size means besides what it does. i hope someone can clear this up :) scenario: size is 3 and min_size is 2 2 rooms with 100 OSDs each and this crush rule "op": "take", "item": -10, "item_name": "default" "op": "choose_firstn", "num": 2, "type": "room" "op": "chooseleaf_firstn", "num": 2, "type": "host" "op": "emit" so if one room goes down/offline, around 50% of the PGs would be left with only 1 replica making them read-only. if we'd set min_size to 1 and one room goes down, user wouldn't still be able to access all PGs - but what is the problem with only one active PG? someone pointed out "split brain" but I am unsure about this. i think what happens in the worst case is this: only 1 PG is available, client writes changes to this PG, the disk of this 1 PG dies as well - so i guess we'd need to restore the data from the 2 offline PGs in the room that is down and we would have lots of trouble with restoring and also with data inconsistency, right? thank you!
but what is the problem with only one active PG? someone pointed out "split brain" but I am unsure about this.
I think Paxos will ensure that split-brain doesn’t happen by virtue of needing >50% of the mon quorum to be up.
i think what happens in the worst case is this: only 1 PG is available, client writes changes to this PG, the disk of this 1 PG dies as well
That’s part of it. Overlapping failures *do* happen.
- so i guess we'd need to restore the data from the 2 offline PGs in the room that is down and we would have lots of trouble with restoring and also with data inconsistency, right?
Yes.
great, thank you Anthony! :) so size 4 / min_size 2 would be a lot better (of course) we have to stay at 3/2 for now though, because our OSDs are filled 60% in sum maybe someone can answer additional questions: - what is the best practice to avoid a full OSD scenario, where ceph tries to recreate all 3 replicas in one of the two rooms when the other room is down? because 3 replicas don't fit in one room obviously. - when does it make sense to migrate a pool to Erasure Coding, when doesn't it make sense and does Erasure Coding NEED a caching tier as some article on the internet stated? I think using a caching tier helps with migrating but other then that I don't understand why we'd need a caching tier.
so size 4 / min_size 2 would be a lot better (of course)
More copies (or parity) are always more reliable, but one quickly gets into diminishing returns. In your scenario you might look into stretch mode, which currently would require 4 replicas. In the future maybe it could support EC with a carefully-chosen profile.
we have to stay at 3/2 for now though, because our OSDs are filled 60% in sum
maybe someone can answer additional questions:
- what is the best practice to avoid a full OSD scenario, where ceph tries to recreate all 3 replicas in one of the two rooms when the other room is down? because 3 replicas don't fit in one room obviously.
If your CRUSH rule limits replicas to 2 per room, then you’ll have roughly half of your PGs undersized and inactive, and the other half undersized and active. In such a situation you could temporarily set min_size=1 on a given pool, with associated risks. Do you have any ability to spread the nodes across *3* rooms?
- when does it make sense to migrate a pool to Erasure Coding, when doesn't it make sense and does Erasure Coding NEED a caching tier as some article on the internet stated? I think using a caching tier helps with migrating but other then that I don't understand why we'd need a caching tier.
Cache tiers are deprecated, AIUI RHCS even compiles them out of their packages. They can be tricky to get right. As far as when EC makes sense, there are lots of factors, including: * Your media type * How important write throughput/latency are * Your workload. If e.g. your workload is RGW or CephFS with predominately tiny objects, EC can result in increased space amplification on bucket pools.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 02.03.23 09:16, stefan.pinter@bearingpoint.com wrote:
so if one room goes down/offline, around 50% of the PGs would be left with only 1 replica making them read-only.
Most people forget the other half of the cluster in such a scenario. For us humans it is obvious that one room is down, because we can see it from the outside. The OSDs only see that they do not have connectivity to their peering partners. They do not know if this is because the other hosts are down or just the network in between. It could be the case that just the line between both rooms is dead and then you have 2 copies running in one room and only one in the other. If you now allow changes in the "smaller" room in addition to changes in the room with two copies you immediately get a conflict as soon as the network connection between both rooms is reestablished. This is why min_size=1 is a really bad idea outside of a desaster scenario where the other two copies are completely lost to a fire. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
thank you Robert! this sounds like split brain again... but we have a quorum system by using 3 monitor nodes. so only the room with the majority of the ceph-mons is available for I/O. if the room with the majority of the ceph-mons is the one that is cut-off, I suppose we'd need to do this: go into the room, shut off the two ceph-mons and then force the other room to start working, right? shuting down down the 2 ceph-mons (in the offline room) would avoid the split-brain scenario: 'If you now allow changes in the "smaller" room in addition to changes in the room with two copies you immediately get a conflict as soon as the network connection between both rooms is reestablished.' or am I wrong? but well, I get why keeping min_size at 2 makes sense but wasn't sure about setting it to 1 in certain scenarios
Den fre 3 mars 2023 kl 01:07 skrev <stefan.pinter@bearingpoint.com>:
it is unclear for us what min_size means besides what it does. i hope someone can clear this up :) someone pointed out "split brain" but I am unsure about this.
i think what happens in the worst case is this: only 1 PG is available, client writes changes to this PG, the disk of this 1 PG dies as well - so i guess we'd need to restore the data from the 2 offline PGs in the room that is down and we would have lots of trouble with restoring and also with data inconsistency, right?
Do not assume the last PG needs to die in a horrible fire, killing several DC operators with it, it only takes a REALLY small outage, a fluke restart, a short network glitch for it to be away, while still having the last acknowledged write to it, making any 1-2-3-4-5 other PG replicas of it useless, since the cluster will know that this one missing OSD has the latest writes, and no other PG can replace it, it can't be "repaired" from others. Perhaps this PG was in a bad place on the platter but no one noticed since it wasn't the primary replica of this PG, but when it comes back and has to replicate all of its (new or old) data to the other replicas, you find that faulty part of the magnetics. This is very much like we all did long ago, putting some 10 large drives in a big raid-5, you get a silent error somewhere on drive5, but the "patrol reads" don't end up there for a long while, because it takes time to read all sectors on all disks. Then drive 3 fails, and suddenly ALL 9 remaining drives need to read ALL their sectors in order to rebuild to the replacement (or hotspare) drive. Now you find the unfound problem on drive5 and hence can't rebuild it all reliably to the spare drive, and someone is going to lose their data, small or big. We hoped that "two drive failures at the same time, on the same day is very unlikely", which is true, except we did not get them on the same day, we just noticed them both that bad day. This causes rumours about "same brand drives fail in clusters" which in some sense is still true, but the above scenario gives it extra validation it doesn't deserve. So after repeating those kinds of mistakes for a bit too long, storage admins decided that losing client data is not worth the "savings" one thinks one does when you cheap out on SATA disks, as if this was the main cost driver in your organisation and not the data or the employees time and tears. This is why ceph defaults to 3 copies, preferably on 3+ different hosts and min_size 2. If we are down to one, we spend 100% of the capacity to make another copy fast, not keep taking on IO in a really really bad situation. There are settings for stretched clusters, that sort of have you use 2+2 copies on each site and so forth, but you really should consider what happens at each side when your DC-DC link falls over. Which side wins, and why. Perhaps it is better to have separate clusters with some sync mechanism, and let each side go as standalone islands, perhaps some other solution is better, but from the wisdom of many ceph admins we see that noone recommends min_size 1 as a permanent way of "solving" anything. Even replica=2 is frowned upon, because any surprise there puts you at size=1 immediately. -- May the most significant bit of your life be positive.
This is not speculation: I have personally experienced this with an inherited 2R cluster.
On Mar 3, 2023, at 04:07, Janne Johansson <icepic.dz@gmail.com> wrote:
Do not assume the last PG needs to die in a horrible fire, killing several DC operators with it, it only takes a REALLY small outage, a fluke restart, a short network glitch for it to be away, while still having the last acknowledged write to it, making any 1-2-3-4-5 other PG replicas of it useless, since the cluster will know that this one missing OSD has the latest writes, and no other PG can replace it, it can't be "repaired" from others.
hi! thanks to all of you, I appreciate this very much! I will have to go through all of your messages a few more times and do some research. so our rule from the intial post does make sure, that, when 1 room goes down it does NOT try to restore 3 replicas in the remaining room but it will only try to create a second replica for each PG that only has 1 at the moment of the outage? i mean: will the ceph cluster try to restore 2 replicas of each PG in the remaining room when every OSD from room 1 is marked as out?
participants (5)
-
Anthony D'Atri
-
Anthony D'Atri
-
Janne Johansson
-
Robert Sander
-
stefan.pinter＠bearingpoint.com