Luminous and mimic: adding OSD can crash mon(s) and lead to loss of quorum
Hi everyone, there are a couple of bug reports about this in Redmine but only one (unanswered) mailing list message[1] that I could find. So I figured I'd raise the issue here again and copy the original reporters of the bugs (they are BCC'd, because in case they are no longer subscribed it wouldn't be appropriate to share their email addresses with the list). This is about https://tracker.ceph.com/issues/40029, and https://tracker.ceph.com/issues/39978 (the latter of which was recently closed as a duplicate of the former). In short, it appears that at least in luminous and mimic (I haven't tried nautilus yet), it's possible to crash a mon when attempting to add a new OSD as it's trying to inject itself into the crush map under its host bucket, when that host bucket does not exist yet. What's worse is that when the OSD's "ceph osd new" process has thus crashed the leader mon, a new leader is elected and in case the "ceph osd new" process is still running on the OSD node, it will promptly connect to that mon, and kill it too. This then continues until sufficiently many mons have died for quorum to be lost. The recovery steps appear to involve - killing the "ceph osd new" process, - restarting mons until you regain quorum, - and then running "ceph osd purge" to drop the problematic OSD entry from the crushmap and osdmap. The issue can apparently be worked around by adding the host buckets to the crushmap manually before adding the new OSDs, but surely this isn't intended to be a prerequisite, at least not to the point of mons crashing otherwise? Also I am guessing that this is some weird corner case rooted in an unusual combination of contributing factors, because otherwise I am guessing more people would be bitten by this problem. Anyone able to share their thoughts on this one? Have more people run into this? Cheers, Florian [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034880.html — interestingly I could find this message in the pipermail archive but none in the one that my MUA keeps for me. So perhaps that message wasn't delivered to all subscribers, which might be why it has gone unanswered.
Is this reproducible with crushtool? ceph osd getcrushmap -o crushmap crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host hostname-that-doesnt-exist-yet -o crushmap.modified Does it still happen if the crushmap is decompiled and recompiled? (crushtool -d and crushtool -c) Replacing XX with the osd ID you tried to add. Posting your (binary) crushmap would be helpful to debug this. (see crushtool -d for what information this file contains) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Aug 23, 2019 at 1:10 PM Florian Haas <florian@citynetwork.eu> wrote:
Hi everyone,
there are a couple of bug reports about this in Redmine but only one (unanswered) mailing list message[1] that I could find. So I figured I'd raise the issue here again and copy the original reporters of the bugs (they are BCC'd, because in case they are no longer subscribed it wouldn't be appropriate to share their email addresses with the list).
This is about https://tracker.ceph.com/issues/40029, and https://tracker.ceph.com/issues/39978 (the latter of which was recently closed as a duplicate of the former).
In short, it appears that at least in luminous and mimic (I haven't tried nautilus yet), it's possible to crash a mon when attempting to add a new OSD as it's trying to inject itself into the crush map under its host bucket, when that host bucket does not exist yet.
What's worse is that when the OSD's "ceph osd new" process has thus crashed the leader mon, a new leader is elected and in case the "ceph osd new" process is still running on the OSD node, it will promptly connect to that mon, and kill it too. This then continues until sufficiently many mons have died for quorum to be lost.
The recovery steps appear to involve
- killing the "ceph osd new" process, - restarting mons until you regain quorum, - and then running "ceph osd purge" to drop the problematic OSD entry from the crushmap and osdmap.
The issue can apparently be worked around by adding the host buckets to the crushmap manually before adding the new OSDs, but surely this isn't intended to be a prerequisite, at least not to the point of mons crashing otherwise?
Also I am guessing that this is some weird corner case rooted in an unusual combination of contributing factors, because otherwise I am guessing more people would be bitten by this problem.
Anyone able to share their thoughts on this one? Have more people run into this?
Cheers, Florian
[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034880.html — interestingly I could find this message in the pipermail archive but none in the one that my MUA keeps for me. So perhaps that message wasn't delivered to all subscribers, which might be why it has gone unanswered. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 23/08/2019 13:34, Paul Emmerich wrote:
Is this reproducible with crushtool?
Not for me.
ceph osd getcrushmap -o crushmap crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host hostname-that-doesnt-exist-yet -o crushmap.modified Replacing XX with the osd ID you tried to add.
Just checking whether this was intentional. As the issue pops up when adding an new OSD *on* a new host, not moving an existing OSD *to* a new host, I would have used --add-item here. Is there a specific reason why you're suggesting to test with --update-item? At any rate, I tried with multiple different combinations (this is on a 12.2.12 test cluster; I can't test this in production): 0. Get the current reference crushmap: # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05846 root default -5 0.01949 host daisy 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd getcrushmap -o crushmap 11 1. "Update" a nonexistent OSD belonging to a nonexistent host (your suggestion): # crushtool -i crushmap --update-item 59 0.01949 osd.59 --loc host nonexistent -o crushmap-update-nonexistent-to-nonexistent # ceph osd setcrushmap -i crushmap-update-nonexistent-to-nonexistent 12 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01949 host nonexistent 59 0.01949 osd.59 DNE 0 -1 0.05846 root default -5 0.01949 host daisy 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd setcrushmap -i crushmap 13 2. Add a nonexistent OSD belonging to a nonexistent host (I think this is functionally identical): # crushtool -i crushmap --add-item 59 0.01949 osd.59 --loc host nonexistent -o crushmap-add-nonexistent-to-nonexistent # ceph osd setcrushmap -i crushmap-add-nonexistent-to-nonexistent 14 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01949 host nonexistent 59 0.01949 osd.59 DNE 0 -1 0.05846 root default -5 0.01949 host daisy 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd setcrushmap -i crushmap 15 3. Move an existing OSD to a nonexistent host: # crushtool -i crushmap --update-item 0 0.01949 osd.0 --loc host nonexistent -o crushmap-update-existing-to-nonexistent # ceph osd setcrushmap -i crushmap-update-existing-to-nonexistent 16 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01949 host nonexistent 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -1 0.03897 root default -5 0 host daisy -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd setcrushmap -i crushmap 17 None of these crashed any mon. However, there's this line in the bug report: -19> 2019-08-22 10:08:11.897364 7f93797ab700 0 mon.cc-ceph-osd11-fra1@0(leader).osd e302401 create-or-move crush item name 'osd.59' initial_weight 1.6374 at location {host=cc-ceph-osd26-fra1,root=default} So it's not trying to move the item to just a nonexistent host, but to a nonexistent host *in the default root*. So I retried the above commands with "--loc host nonexistent --loc root default". No change other than everything showing up under default; no mon crash. And then I tried one more which was to *first* add just a new OSD under the default root, and *then* moving that OSD to a new, nonexistent host, also under the default root. Again, no mon crash. So I'm afraid I am unable to reproduce this with crushtool and setcrushmap. And I can't get my mons to crash with "ceph osd crush move", either: ceph osd crush move osd.59 host=nonexistent root=default moved item id 59 name 'osd.59' to location {host=nonexistent,root=default} in crush map Cheers, Florian
On Fri, Aug 23, 2019 at 3:54 PM Florian Haas <florian@citynetwork.eu> wrote:
On 23/08/2019 13:34, Paul Emmerich wrote:
Is this reproducible with crushtool?
Not for me.
ceph osd getcrushmap -o crushmap crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host hostname-that-doesnt-exist-yet -o crushmap.modified Replacing XX with the osd ID you tried to add.
Just checking whether this was intentional. As the issue pops up when adding an new OSD *on* a new host, not moving an existing OSD *to* a new host, I would have used --add-item here. Is there a specific reason why you're suggesting to test with --update-item?
yes, update should map to create or move which it should use internally
At any rate, I tried with multiple different combinations (this is on a 12.2.12 test cluster; I can't test this in production):
which also ran into this bug? The idea of using crushtool is to not crash your production cluster but just the local tool. Paul
0. Get the current reference crushmap:
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05846 root default -5 0.01949 host daisy 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd getcrushmap -o crushmap 11
1. "Update" a nonexistent OSD belonging to a nonexistent host (your suggestion):
# crushtool -i crushmap --update-item 59 0.01949 osd.59 --loc host nonexistent -o crushmap-update-nonexistent-to-nonexistent # ceph osd setcrushmap -i crushmap-update-nonexistent-to-nonexistent 12 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01949 host nonexistent 59 0.01949 osd.59 DNE 0 -1 0.05846 root default -5 0.01949 host daisy 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd setcrushmap -i crushmap 13
2. Add a nonexistent OSD belonging to a nonexistent host (I think this is functionally identical):
# crushtool -i crushmap --add-item 59 0.01949 osd.59 --loc host nonexistent -o crushmap-add-nonexistent-to-nonexistent # ceph osd setcrushmap -i crushmap-add-nonexistent-to-nonexistent 14 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01949 host nonexistent 59 0.01949 osd.59 DNE 0 -1 0.05846 root default -5 0.01949 host daisy 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd setcrushmap -i crushmap 15
3. Move an existing OSD to a nonexistent host:
# crushtool -i crushmap --update-item 0 0.01949 osd.0 --loc host nonexistent -o crushmap-update-existing-to-nonexistent # ceph osd setcrushmap -i crushmap-update-existing-to-nonexistent 16 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.01949 host nonexistent 0 hdd 0.01949 osd.0 up 1.00000 1.00000 -1 0.03897 root default -5 0 host daisy -7 0.01949 host eric 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -3 0.01949 host frank 2 hdd 0.01949 osd.2 up 1.00000 1.00000 # ceph osd setcrushmap -i crushmap 17
None of these crashed any mon.
However, there's this line in the bug report:
-19> 2019-08-22 10:08:11.897364 7f93797ab700 0 mon.cc-ceph-osd11-fra1@0(leader).osd e302401 create-or-move crush item name 'osd.59' initial_weight 1.6374 at location {host=cc-ceph-osd26-fra1,root=default}
So it's not trying to move the item to just a nonexistent host, but to a nonexistent host *in the default root*.
So I retried the above commands with "--loc host nonexistent --loc root default". No change other than everything showing up under default; no mon crash.
And then I tried one more which was to *first* add just a new OSD under the default root, and *then* moving that OSD to a new, nonexistent host, also under the default root. Again, no mon crash.
So I'm afraid I am unable to reproduce this with crushtool and setcrushmap.
And I can't get my mons to crash with "ceph osd crush move", either:
ceph osd crush move osd.59 host=nonexistent root=default moved item id 59 name 'osd.59' to location {host=nonexistent,root=default} in crush map
Cheers, Florian
On 23/08/2019 22:14, Paul Emmerich wrote:
On Fri, Aug 23, 2019 at 3:54 PM Florian Haas <florian@citynetwork.eu> wrote:
On 23/08/2019 13:34, Paul Emmerich wrote:
Is this reproducible with crushtool?
Not for me.
ceph osd getcrushmap -o crushmap crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host hostname-that-doesnt-exist-yet -o crushmap.modified Replacing XX with the osd ID you tried to add.
Just checking whether this was intentional. As the issue pops up when adding an new OSD *on* a new host, not moving an existing OSD *to* a new host, I would have used --add-item here. Is there a specific reason why you're suggesting to test with --update-item?
yes, update should map to create or move which it should use internally
At any rate, I tried with multiple different combinations (this is on a 12.2.12 test cluster; I can't test this in production):
which also ran into this bug? The idea of using crushtool is to not crash your production cluster but just the local tool.
Ah, gotcha. I thought you wanted me to be able to at least do "ceph osd setcrushmap" with the resulting crushmap, which would require a running cluster. So yes, doing this completely offline shows that you're definitely on to something. I am able to crash crushtool with the original crushmap, and what it appears to be falling over on is a choose_args map in there. I've updated the bug report with this comment: https://tracker.ceph.com/issues/40029#note-11 It would seem that there are two workarounds at this stage for pre-Nautilus users with a choose_args map in their crushmap, and who for some reason are unable to upgrade to Nautilus yet: 1. Add host buckets manually before adding new OSDs. 2. Drop any choose_args map from their crushmap. As it happens I am not aware of any way to do #2 other than - using getcrushmap, - decompiling the crushmap, - dropping the choose_args map from the textual representation of the crushmap, - recompiling, and then - using setcrushmap. Are you, by any chance? Thanks again for your help! Cheers, Florian
participants (2)
-
Florian Haas
-
Paul Emmerich