EC PGs stuck activating, 2^31-1 as OSD ID, automatic recovery not kicking in
Hi, We have a cluster where we mix HDDs and NVMe drives using device classes with a specific crush role for each class. One of our NVMe drives physically died which caused some of our PGs to go into this state: pg 26.ac is stuck undersized for 60830.991784, current state activating+undersized+degraded+remapped, last acting [353,373,368,377,2147483647,350] pg 26.d1 is stuck undersized for 60830.587711, current state activating+undersized+degraded+remapped, last acting [343,2147483647,347,358,366,355] pg 26.e1 is stuck undersized for 60830.980585, current state activating+undersized+degraded+remapped, last acting [340,349,370,2147483647,360,376] ... and so on. Recovery never happened and we had to manually restart all affected OSDs for all PGs stuck in such a state. The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered. The issue seems to be going back to 2015: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-May/001661.html however no solution... I'm more concerned about the cluster not being able to recover (it's a 4+2 EC pool across 12 hosts - plenty of room to heal) than about the weird print-out. The VMs who wanted to access data in any of the affected PGs of course died. Are we missing some settings to let the cluster self-heal even for EC pools? First EC pool in production :) Cheers, Zoltan
On Fri, Nov 22, 2019 at 9:33 PM Zoltan Arnold Nagy <zoltan@linux.vnet.ibm.com> wrote:
The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered.
no, this value is intentional (and shows up as 'None' on higher level tools), it means no mapping could be found; check your crush map and crush rule Paul
The issue seems to be going back to 2015: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-May/001661.html however no solution...
I'm more concerned about the cluster not being able to recover (it's a 4+2 EC pool across 12 hosts - plenty of room to heal) than about the weird print-out.
The VMs who wanted to access data in any of the affected PGs of course died.
Are we missing some settings to let the cluster self-heal even for EC pools? First EC pool in production :)
Cheers, Zoltan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2019-11-22 21:45, Paul Emmerich wrote:
On Fri, Nov 22, 2019 at 9:33 PM Zoltan Arnold Nagy <zoltan@linux.vnet.ibm.com> wrote:
The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered.
no, this value is intentional (and shows up as 'None' on higher level tools), it means no mapping could be found
thanks, didn't know.
check your crush map and crush rule
if it were indeed a crush rule or map issue, it would not have been resolved by just restarting the primary OSD of the PG, would it? the crush rule was created by running ceph osd erasure-code-profile set ec42 k=4 m=2 crush-device-class=nvme where the default failure domain is host; as I said we have 12 hosts, so I don't see anything wrong here - it's all default... this is why I suspect a bug, just don't have any evidence other than that it happened to us :)
How many PGs in the pools? This maybe the CRUSH can not get the proper OSD. you can check param tunable choose_total_tries in crush tunables, try increase it like this: ceph osd getcrushmap -o crush crushtool -d crush -o crush.txt sed -i 's/tunable choose_total_tries 50/tunable choose_total_tries 150/g' crush.txt crushtool -c crush.txt -o crush.new ceph osd setcrushmap -i crush.new Zoltan Arnold Nagy <zoltan@linux.vnet.ibm.com> 于2019年11月23日周六 上午5:26写道：
On 2019-11-22 21:45, Paul Emmerich wrote:
On Fri, Nov 22, 2019 at 9:33 PM Zoltan Arnold Nagy <zoltan@linux.vnet.ibm.com> wrote:
The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered.
no, this value is intentional (and shows up as 'None' on higher level tools), it means no mapping could be found
thanks, didn't know.
check your crush map and crush rule
if it were indeed a crush rule or map issue, it would not have been resolved by just restarting the primary OSD of the PG, would it?
the crush rule was created by running
ceph osd erasure-code-profile set ec42 k=4 m=2 crush-device-class=nvme
where the default failure domain is host; as I said we have 12 hosts, so I don't see anything wrong here - it's all default...
this is why I suspect a bug, just don't have any evidence other than that it happened to us :) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 22.11.19 23:45, Paul Emmerich wrote:
tools), it means no mapping could be found; check your crush map and crush rule
Most simple way to get into this state is to change OSDs' reweight on small cluster where number of OSDs equal to EC n+k. I do not know exactly, but seems that straw2 crush algorithm has no default/fallback policy to pick osd for pg. For example if you have 8 osd, failure_domain=osd and ec profile with k=5,m=3 and then you eventually change reweight of some osd then crush algorithm fails to satisfy both requirements: - failure domain - number of pgs on osd be proportional to weight Btw I've newer saw any explanation of decision not to pick OSD in this case. I mean why not to pick OSD and leave PG in degraded state is better then violate weight requirements?
participants (4)
-
Aleksey Gutikov
-
huang jun
-
Paul Emmerich
-
Zoltan Arnold Nagy