23 Aug
2019
23 Aug
'19
4:09 p.m.
Hi everyone,
there are a couple of bug reports about this in Redmine but only one
(unanswered) mailing list message[1] that I could find. So I figured I'd
raise the issue here again and copy the original reporters of the bugs
(they are BCC'd, because in case they are no longer subscribed it
wouldn't be appropriate to share their email addresses with the list).
This is about https://tracker.ceph.com/issues/40029, and
https://tracker.ceph.com/issues/39978 (the latter of which was recently
closed as a duplicate of the former).
In short, it appears that at least in luminous and mimic (I haven't
tried nautilus yet), it's possible to crash a mon when attempting to add
a new OSD as it's trying to inject itself into the crush map under its
host bucket, when that host bucket does not exist yet.
What's worse is that when the OSD's "ceph osd new" process has thus
crashed the leader mon, a new leader is elected and in case the "ceph
osd new" process is still running on the OSD node, it will promptly
connect to that mon, and kill it too. This then continues until
sufficiently many mons have died for quorum to be lost.
The recovery steps appear to involve
- killing the "ceph osd new" process,
- restarting mons until you regain quorum,
- and then running "ceph osd purge" to drop the problematic OSD entry
from the crushmap and osdmap.
The issue can apparently be worked around by adding the host buckets to
the crushmap manually before adding the new OSDs, but surely this isn't
intended to be a prerequisite, at least not to the point of mons
crashing otherwise?
Also I am guessing that this is some weird corner case rooted in an
unusual combination of contributing factors, because otherwise I am
guessing more people would be bitten by this problem.
Anyone able to share their thoughts on this one? Have more people run
into this?
Cheers,
Florian
[1]
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034880.html
— interestingly I could find this message in the pipermail archive but
none in the one that my MUA keeps for me. So perhaps that message wasn't
delivered to all subscribers, which might be why it has gone unanswered.
23 Aug
23 Aug
4:34 p.m.
Is this reproducible with crushtool?
ceph osd getcrushmap -o crushmap
crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host
hostname-that-doesnt-exist-yet -o crushmap.modified
Does it still happen if the crushmap is decompiled and recompiled?
(crushtool -d and crushtool -c)
Replacing XX with the osd ID you tried to add. Posting your (binary)
crushmap would be helpful to debug this. (see crushtool -d for what
information this file contains)
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Aug 23, 2019 at 1:10 PM Florian Haas <florian(a)citynetwork.eu> wrote:
>
> Hi everyone,
>
> there are a couple of bug reports about this in Redmine but only one
> (unanswered) mailing list message[1] that I could find. So I figured I'd
> raise the issue here again and copy the original reporters of the bugs
> (they are BCC'd, because in case they are no longer subscribed it
> wouldn't be appropriate to share their email addresses with the list).
>
> This is about https://tracker.ceph.com/issues/40029, and
> https://tracker.ceph.com/issues/39978 (the latter of which was recently
> closed as a duplicate of the former).
>
> In short, it appears that at least in luminous and mimic (I haven't
> tried nautilus yet), it's possible to crash a mon when attempting to add
> a new OSD as it's trying to inject itself into the crush map under its
> host bucket, when that host bucket does not exist yet.
>
> What's worse is that when the OSD's "ceph osd new" process has
thus
> crashed the leader mon, a new leader is elected and in case the "ceph
> osd new" process is still running on the OSD node, it will promptly
> connect to that mon, and kill it too. This then continues until
> sufficiently many mons have died for quorum to be lost.
>
> The recovery steps appear to involve
>
> - killing the "ceph osd new" process,
> - restarting mons until you regain quorum,
> - and then running "ceph osd purge" to drop the problematic OSD entry
> from the crushmap and osdmap.
>
> The issue can apparently be worked around by adding the host buckets to
> the crushmap manually before adding the new OSDs, but surely this isn't
> intended to be a prerequisite, at least not to the point of mons
> crashing otherwise?
>
> Also I am guessing that this is some weird corner case rooted in an
> unusual combination of contributing factors, because otherwise I am
> guessing more people would be bitten by this problem.
>
> Anyone able to share their thoughts on this one? Have more people run
> into this?
>
> Cheers,
> Florian
>
>
>
> [1]
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034880.html
> — interestingly I could find this message in the pipermail archive but
> none in the one that my MUA keeps for me. So perhaps that message wasn't
> delivered to all subscribers, which might be why it has gone unanswered.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:54 p.m.
On 23/08/2019 13:34, Paul Emmerich wrote:
Is this reproducible with crushtool?
Not for me.
ceph osd getcrushmap -o crushmap
crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host
hostname-that-doesnt-exist-yet -o crushmap.modified
Replacing XX with the osd ID you tried to add.
Just checking whether this was intentional. As the issue pops up when
adding an new OSD *on* a new host, not moving an existing OSD *to* a new
host, I would have used --add-item here. Is there a specific reason why
you're suggesting to test with --update-item?
At any rate, I tried with multiple different combinations (this is on a
12.2.12 test cluster; I can't test this in production):
0. Get the current reference crushmap:
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.05846 root default
-5 0.01949 host daisy
0 hdd 0.01949 osd.0 up 1.00000 1.00000
-7 0.01949 host eric
1 hdd 0.01949 osd.1 up 1.00000 1.00000
-3 0.01949 host frank
2 hdd 0.01949 osd.2 up 1.00000 1.00000
# ceph osd getcrushmap -o crushmap
11
1. "Update" a nonexistent OSD belonging to a nonexistent host (your
suggestion):
# crushtool -i crushmap --update-item 59 0.01949 osd.59 --loc host
nonexistent -o crushmap-update-nonexistent-to-nonexistent
# ceph osd setcrushmap -i crushmap-update-nonexistent-to-nonexistent
12
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-9 0.01949 host nonexistent
59 0.01949 osd.59 DNE 0
-1 0.05846 root default
-5 0.01949 host daisy
0 hdd 0.01949 osd.0 up 1.00000 1.00000
-7 0.01949 host eric
1 hdd 0.01949 osd.1 up 1.00000 1.00000
-3 0.01949 host frank
2 hdd 0.01949 osd.2 up 1.00000 1.00000
# ceph osd setcrushmap -i crushmap
13
2. Add a nonexistent OSD belonging to a nonexistent host (I think this
is functionally identical):
# crushtool -i crushmap --add-item 59 0.01949 osd.59 --loc host
nonexistent -o crushmap-add-nonexistent-to-nonexistent
# ceph osd setcrushmap -i crushmap-add-nonexistent-to-nonexistent
14
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9
0.01949 host nonexistent
59 0.01949 osd.59 DNE 0
-1 0.05846 root default
-5 0.01949 host daisy
0 hdd 0.01949 osd.0 up 1.00000 1.00000
-7 0.01949 host eric
1 hdd 0.01949 osd.1 up 1.00000 1.00000
-3 0.01949 host frank
2 hdd 0.01949 osd.2 up 1.00000 1.00000
# ceph osd setcrushmap -i crushmap
15
3. Move an existing OSD to a nonexistent host:
# crushtool -i crushmap --update-item 0 0.01949 osd.0 --loc host
nonexistent -o crushmap-update-existing-to-nonexistent
# ceph osd setcrushmap -i crushmap-update-existing-to-nonexistent
16
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-9 0.01949 host nonexistent
0 hdd 0.01949 osd.0 up 1.00000 1.00000
-1 0.03897 root default
-5 0 host daisy
-7 0.01949 host eric
1 hdd 0.01949 osd.1 up 1.00000 1.00000
-3 0.01949 host frank
2 hdd 0.01949 osd.2 up 1.00000 1.00000
# ceph osd setcrushmap -i crushmap
17
None of these crashed any mon.
However, there's this line in the bug report:
-19> 2019-08-22 10:08:11.897364 7f93797ab700 0
mon.cc-ceph-osd11-fra1(a)0(leader).osd e302401 create-or-move crush item
name 'osd.59' initial_weight 1.6374 at location
{host=cc-ceph-osd26-fra1,root=default}
So it's not trying to move the item to just a nonexistent host, but to a
nonexistent host *in the default root*.
So I retried the above commands with "--loc host nonexistent --loc root
default". No change other than everything showing up under default; no
mon crash.
And then I tried one more which was to *first* add just a new OSD under
the default root, and *then* moving that OSD to a new, nonexistent host,
also under the default root. Again, no mon crash.
So I'm afraid I am unable to reproduce this with crushtool and setcrushmap.
And I can't get my mons to crash with "ceph osd crush move", either:
ceph osd crush move osd.59 host=nonexistent root=default
moved item id 59 name 'osd.59' to location
{host=nonexistent,root=default} in crush map
Cheers,
Florian
24 Aug
24 Aug
1:14 a.m.
On Fri, Aug 23, 2019 at 3:54 PM Florian Haas <florian(a)citynetwork.eu> wrote:
On 23/08/2019 13:34, Paul Emmerich wrote:
yes, update should map to create or move which it should use internally
Is this reproducible with crushtool?
Not for me.
ceph osd getcrushmap -o crushmap
crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host
hostname-that-doesnt-exist-yet -o crushmap.modified
Replacing XX with the osd ID you tried to add.
Just checking whether this was intentional. As the issue pops up when
adding an new OSD *on* a new host, not moving an existing OSD *to* a new
host, I would have used --add-item here. Is there a specific reason why
you're suggesting to test with --update-item?
At any rate, I tried with multiple different combinations (this is on a
12.2.12 test cluster; I can't test this in production):
which also ran into this bug? The idea of using crushtool is to not
crash your production cluster but just the local tool.
Paul
>
>
> 0. Get the current reference crushmap:
>
> # ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 0.05846 root default
> -5 0.01949 host daisy
> 0 hdd 0.01949 osd.0 up 1.00000 1.00000
> -7 0.01949 host eric
> 1 hdd 0.01949 osd.1 up 1.00000 1.00000
> -3 0.01949 host frank
> 2 hdd 0.01949 osd.2 up 1.00000 1.00000
> # ceph osd getcrushmap -o crushmap
> 11
>
>
> 1. "Update" a nonexistent OSD belonging to a nonexistent host (your
> suggestion):
>
> # crushtool -i crushmap --update-item 59 0.01949 osd.59 --loc host
> nonexistent -o crushmap-update-nonexistent-to-nonexistent
> # ceph osd setcrushmap -i crushmap-update-nonexistent-to-nonexistent
> 12
> # ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -9 0.01949 host nonexistent
> 59 0.01949 osd.59 DNE 0
> -1 0.05846 root default
> -5 0.01949 host daisy
> 0 hdd 0.01949 osd.0 up 1.00000 1.00000
> -7 0.01949 host eric
> 1 hdd 0.01949 osd.1 up 1.00000 1.00000
> -3 0.01949 host frank
> 2 hdd 0.01949 osd.2 up 1.00000 1.00000
> # ceph osd setcrushmap -i crushmap
> 13
>
>
> 2. Add a nonexistent OSD belonging to a nonexistent host (I think this
> is functionally identical):
>
> # crushtool -i crushmap --add-item 59 0.01949 osd.59 --loc host
> nonexistent -o crushmap-add-nonexistent-to-nonexistent
> # ceph osd setcrushmap -i crushmap-add-nonexistent-to-nonexistent
> 14
> # ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9
> 0.01949 host nonexistent
> 59 0.01949 osd.59 DNE 0
> -1 0.05846 root default
> -5 0.01949 host daisy
> 0 hdd 0.01949 osd.0 up 1.00000 1.00000
> -7 0.01949 host eric
> 1 hdd 0.01949 osd.1 up 1.00000 1.00000
> -3 0.01949 host frank
> 2 hdd 0.01949 osd.2 up 1.00000 1.00000
> # ceph osd setcrushmap -i crushmap
> 15
>
>
> 3. Move an existing OSD to a nonexistent host:
>
> # crushtool -i crushmap --update-item 0 0.01949 osd.0 --loc host
> nonexistent -o crushmap-update-existing-to-nonexistent
> # ceph osd setcrushmap -i crushmap-update-existing-to-nonexistent
> 16
> # ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -9 0.01949 host nonexistent
> 0 hdd 0.01949 osd.0 up 1.00000 1.00000
> -1 0.03897 root default
> -5 0 host daisy
> -7 0.01949 host eric
> 1 hdd 0.01949 osd.1 up 1.00000 1.00000
> -3 0.01949 host frank
> 2 hdd 0.01949 osd.2 up 1.00000 1.00000
> # ceph osd setcrushmap -i crushmap
> 17
>
>
> None of these crashed any mon.
>
> However, there's this line in the bug report:
>
> -19> 2019-08-22 10:08:11.897364 7f93797ab700 0
> mon.cc-ceph-osd11-fra1(a)0(leader).osd e302401 create-or-move crush item
> name 'osd.59' initial_weight 1.6374 at location
> {host=cc-ceph-osd26-fra1,root=default}
>
> So it's not trying to move the item to just a nonexistent host, but to a
> nonexistent host *in the default root*.
>
> So I retried the above commands with "--loc host nonexistent --loc root
> default". No change other than everything showing up under default; no
> mon crash.
>
> And then I tried one more which was to *first* add just a new OSD under
> the default root, and *then* moving that OSD to a new, nonexistent host,
> also under the default root. Again, no mon crash.
>
> So I'm afraid I am unable to reproduce this with crushtool and setcrushmap.
>
>
> And I can't get my mons to crash with "ceph osd crush move", either:
>
> ceph osd crush move osd.59 host=nonexistent root=default
> moved item id 59 name 'osd.59' to location
> {host=nonexistent,root=default} in crush map
>
>
> Cheers,
> Florian
26 Aug
26 Aug
4:34 p.m.
On 23/08/2019 22:14, Paul Emmerich wrote:
On Fri, Aug 23, 2019 at 3:54 PM Florian Haas
<florian(a)citynetwork.eu> wrote:
Ah, gotcha. I thought you wanted me to be able to at least do "ceph osd
setcrushmap" with the resulting crushmap, which would require a running
cluster.
So yes, doing this completely offline shows that you're definitely on to
something. I am able to crash crushtool with the original crushmap, and
what it appears to be falling over on is a choose_args map in there.
I've updated the bug report with this comment:
https://tracker.ceph.com/issues/40029#note-11
It would seem that there are two workarounds at this stage for
pre-Nautilus users with a choose_args map in their crushmap, and who for
some reason are unable to upgrade to Nautilus yet:
1. Add host buckets manually before adding new OSDs.
2. Drop any choose_args map from their crushmap.
As it happens I am not aware of any way to do #2 other than
- using getcrushmap,
- decompiling the crushmap,
- dropping the choose_args map from the textual representation of the
crushmap,
- recompiling, and then
- using setcrushmap.
Are you, by any chance?
Thanks again for your help!
Cheers,
Florian
On 23/08/2019 13:34, Paul Emmerich wrote:
yes, update should map to create or move which it should use internally
Is this reproducible with crushtool?
Not for me.
ceph osd getcrushmap -o crushmap
crushtool -i crushmap --update-item XX 1.0 osd.XX --loc host
hostname-that-doesnt-exist-yet -o crushmap.modified
Replacing XX with the osd ID you tried to add.
Just checking whether this was intentional. As the issue pops up when
adding an new OSD *on* a new host, not moving an existing OSD *to* a new
host, I would have used --add-item here. Is there a specific reason why
you're suggesting to test with --update-item?
At any rate, I tried with multiple different combinations (this is on a
12.2.12 test cluster; I can't test this in production):
which also ran into this bug? The idea of using crushtool is to not
crash your production cluster but just the local tool.
1703
days inactive
1706
days old
4 comments
2 participants
participants (2)
-
Florian Haas -
Paul Emmerich