7 Jun
2021
7 Jun
'21
7:16 p.m.
Hello,
I implemented a new cluster with 48 Nodes á 24 OSDs.
I have a replicated pool with 4 replica. The crushrule distributes the
replicas to different racks.
With this cluster I tested a upgrade from Nautilis (14.2.20) to Octopus
(15.2.13). The update itself worked well until I began the restarts of
the OSDs in the 4th rack. Since then I get slow ops while stopping
OSDs. I think something happend here, after all replica partners are
running on the new version. This issue remains after completing the
upgrade.
With Nautilus I had similar issues with slow ops when stopping OSDs. I
could resolve this with the option „osd_fast_shutdown → false“. I let
this option set to false while upgrading. For testing/debugging, I set
this to true (default value) and got better results when stopping OSDs,
but the problem is not completely vanished.
Had someone else this problem and could fix it? What can I do to get
rid of slow ops when resarting OSDs?
All Servers are connected with 2x10G network links
Manuel
10 Jun
10 Jun
2:38 p.m.
Hi,
has no one a idea what could cause this issue. Or how I could debug it?
In some days I have to go live with this cluster. If I don't have a
solution I have to go live with nautilus.
Manuel
On Mon, 7 Jun 2021 15:46:18 +0200
Manuel Lausch <manuel.lausch(a)1und1.de> wrote:
Hello,
I implemented a new cluster with 48 Nodes á 24 OSDs.
I have a replicated pool with 4 replica. The crushrule distributes the
replicas to different racks.
With this cluster I tested a upgrade from Nautilis (14.2.20) to
Octopus (15.2.13). The update itself worked well until I began the
restarts of the OSDs in the 4th rack. Since then I get slow ops while
stopping OSDs. I think something happend here, after all replica
partners are running on the new version. This issue remains after
completing the upgrade.
With Nautilus I had similar issues with slow ops when stopping OSDs. I
could resolve this with the option „osd_fast_shutdown → false“. I let
this option set to false while upgrading. For testing/debugging, I set
this to true (default value) and got better results when stopping
OSDs, but the problem is not completely vanished.
Had someone else this problem and could fix it? What can I do to get
rid of slow ops when resarting OSDs?
All Servers are connected with 2x10G network links
Manuel
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Manuel Lausch
Systemadministrator
Storage Services
1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 |
76135 Karlsruhe | Germany Phone: +49 721 91374-1847
E-Mail: manuel.lausch(a)1und1.de | Web: www.1und1.de
Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452
Geschäftsführer: Alexander Charles, Thomas Ludwig, Jan Oetjen, Sascha
Vollmer
Member of United Internet
Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte
Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat
sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie
bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem
bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern,
weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu
verwenden.
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient of this e-mail, you are hereby
notified that saving, distribution or use of the content of this e-mail
in any way is prohibited. If you have received this e-mail in error,
please notify the sender and delete the e-mail.
2:58 p.m.
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
Hi,
has no one a idea what could cause this issue. Or how I could debug it?
In some days I have to go live with this cluster. If I don't have a
solution I have to go live with nautilus.
Hi Manuel,
I had similar issues with Octopus and i am thus stuck with Nautilus.
Can you debug the slow ops and see if the slow ops are caused by the status "waiting
for readable".
I suspected that it has something to do with the new feature in Octopus to read from all
OSDs regardless if
they are master for a PG or not.
Can you also verify that osd_op_queue_cut_off is set to high and that icmp rate limiting
is disabled on your hosts?
Peter
9:15 p.m.
Hi Peter,
your suggestion pointed me to the right spot.
I didn't know about the feature, that ceph will read from replica
PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc:
"check_laggy" and "check_laggy_requeue". On both is first a check, if
the partners have the octopus features. if not, the function is
skipped. This explains the beginning of the problem after about the
half cluster was updated.
To verifiy this, I added "return true" in the first line of the
functions. The issue is gone with it. But
I don't know what problems this could trigger. I know, the root cause
is not fixed with it.
I think I will open a bug ticket with this knowlage.
osd_op_queue_cutoff is set to high
and a icmp rate limiting should not happen
Thanks
Manuel
On Thu, 10 Jun 2021 11:28:48 +0200
Peter Lieven <pl(a)kamp.de> wrote:
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
Hi,
has no one a idea what could cause this issue. Or how I could debug
it?
In some days I have to go live with this cluster. If I don't have a
solution I have to go live with nautilus.
Hi Manuel,
I had similar issues with Octopus and i am thus stuck with Nautilus.
Can you debug the slow ops and see if the slow ops are caused by the
status "waiting for readable".
I suspected that it has something to do with the new feature in
Octopus to read from all OSDs regardless if
they are master for a PG or not.
Can you also verify that osd_op_queue_cut_off is set to high and that
icmp rate limiting is disabled on your hosts?
Peter
9:24 p.m.
Hi,
At which point in the update procedure did you
ceph osd require-osd-release octopus
?
And are you sure it was set to nautilus before the update? (`ceph osd dump`
will show)
Cheers , Dan
On Thu, Jun 10, 2021, 5:45 PM Manuel Lausch <manuel.lausch(a)1und1.de> wrote:
Hi Peter,
your suggestion pointed me to the right spot.
I didn't know about the feature, that ceph will read from replica
PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc:
"check_laggy" and "check_laggy_requeue". On both is first a check,
if
the partners have the octopus features. if not, the function is
skipped. This explains the beginning of the problem after about the
half cluster was updated.
To verifiy this, I added "return true" in the first line of the
functions. The issue is gone with it. But
I don't know what problems this could trigger. I know, the root cause
is not fixed with it.
I think I will open a bug ticket with this knowlage.
osd_op_queue_cutoff is set to high
and a icmp rate limiting should not happen
Thanks
Manuel
On Thu, 10 Jun 2021 11:28:48 +0200
Peter Lieven <pl(a)kamp.de> wrote:
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi,
has no one a idea what could cause this issue. Or how I could debug
it?
In some days I have to go live with this cluster. If I don't have a
solution I have to go live with nautilus.
Hi Manuel,
I had similar issues with Octopus and i am thus stuck with Nautilus.
Can you debug the slow ops and see if the slow ops are caused by the
status "waiting for readable".
I suspected that it has something to do with the new feature in
Octopus to read from all OSDs regardless if
they are master for a PG or not.
Can you also verify that osd_op_queue_cut_off is set to high and that
icmp rate limiting is disabled on your hosts?
Peter
9:35 p.m.
Hi Dan,
The cluster was initialy deployed with nautilus (14.2.20). I am sure
require-osd-release was nautilus at this point.
I did set this to octopus, after all components was updatated.
Manuel
On Thu, 10 Jun 2021 17:54:49 +0200
Dan van der Ster <dan(a)vanderster.com> wrote:
> Hi,
>
> At which point in the update procedure did you
>
> ceph osd require-osd-release octopus
>
> ?
>
> And are you sure it was set to nautilus before the update? (`ceph osd
> dump` will show)
>
> Cheers , Dan
>
>
9:47 p.m.
Ok sounds correct. This was the only thing that came to mind which might
explain your problem.
Cheers, Dan
On Thu, Jun 10, 2021, 6:06 PM Manuel Lausch <manuel.lausch(a)1und1.de> wrote:
Hi Dan,
The cluster was initialy deployed with nautilus (14.2.20). I am sure
require-osd-release was nautilus at this point.
I did set this to octopus, after all components was updatated.
Manuel
On Thu, 10 Jun 2021 17:54:49 +0200
Dan van der Ster <dan(a)vanderster.com> wrote:
Hi,
At which point in the update procedure did you
ceph osd require-osd-release octopus
?
And are you sure it was set to nautilus before the update? (`ceph osd
dump` will show)
Cheers , Dan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11 Jun
11 Jun
2:37 p.m.
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
Hi Peter,
your suggestion pointed me to the right spot.
I didn't know about the feature, that ceph will read from replica
PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc:
"check_laggy" and "check_laggy_requeue". On both is first a check,
if
the partners have the octopus features. if not, the function is
skipped. This explains the beginning of the problem after about the
half cluster was updated.
To verifiy this, I added "return true" in the first line of the
functions. The issue is gone with it. But
I don't know what problems this could trigger. I know, the root cause
is not fixed with it.
I think I will open a bug ticket with this knowlage.
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and
peering started.
My cluster was a fresh octopus install so I think the min osd release was set to octopus.
Is it in general safe to stay with this switch at nautilus and run octopus to run a
maintained release?
osd_op_queue_cutoff is set to high
and a icmp rate limiting should not happen
It could if you choose fast shutdown and connections to the OSD daemon are refused with
icmp port unreachable?!
Peter
3:18 p.m.
On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven <pl(a)kamp.de> wrote:
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
I would guess not -- one should follow the upgrade instructions as
documented. (I was merely confirming that Manuel had indeed followed
that procedure).
IMHO this issue, as described by Manuel, is not understood.
Manuel could you create a tracker and upload any relevant logs from
when the slow requests began?
-- dan
>
>
> >
> >
> > osd_op_queue_cutoff is set to high
> > and a icmp rate limiting should not happen
>
>
> It could if you choose fast shutdown and connections to the OSD daemon are refused
with icmp port unreachable?!
>
>
> Peter
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi Peter,
your suggestion pointed me to the right spot.
I didn't know about the feature, that ceph will read from replica
PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc:
"check_laggy" and "check_laggy_requeue". On both is first a check,
if
the partners have the octopus features. if not, the function is
skipped. This explains the beginning of the problem after about the
half cluster was updated.
To verifiy this, I added "return true" in the first line of the
functions. The issue is gone with it. But
I don't know what problems this could trigger. I know, the root cause
is not fixed with it.
I think I will open a bug ticket with this knowlage.
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and
peering started.
My cluster was a fresh octopus install so I think the min osd release was set to
octopus.
Is it in general safe to stay with this switch at nautilus and run octopus to run a
maintained release?
3:41 p.m.
Am 11.06.21 um 11:48 schrieb Dan van der Ster:
On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven
<pl(a)kamp.de> wrote:
I would be interested if the time is also spent in "waiting for readable"
state.
To check this I reduced the slow ops timeout to 1 second and then fetched the info about
the
slow ops from the osd daemons that were affected.
Manuel, can you check this as well?
Thanks,
Peter
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
I would guess not -- one should follow the upgrade instructions
as
documented. (I was merely confirming that Manuel had indeed followed
that procedure).
IMHO this issue, as described by Manuel, is not understood.
Manuel could you create a tracker and upload any relevant logs from
when the slow requests began? Hi Peter,
your suggestion pointed me to the right spot.
I didn't know about the feature, that ceph will read from replica
PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc:
"check_laggy" and "check_laggy_requeue". On both is first a check,
if
the partners have the octopus features. if not, the function is
skipped. This explains the beginning of the problem after about the
half cluster was updated.
To verifiy this, I added "return true" in the first line of the
functions. The issue is gone with it. But
I don't know what problems this could trigger. I know, the root cause
is not fixed with it.
I think I will open a bug ticket with this knowlage.
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and
peering started.
My cluster was a fresh octopus install so I think the min osd release was set to
octopus.
Is it in general safe to stay with this switch at nautilus and run octopus to run a
maintained release?
15 Jun
15 Jun
7:58 p.m.
Hi,
yeah. I wasn't aware, that I have set "osd op complaint time" to 5
seconds. AFAIK the default is 32. So I get slow ops already after 5
instead of 32 seconds. Thats why I think no one before have noticed
this before.
My application which uses the cluster will throw timeouts after 6
seconds, thats why we set this down.
For clarification, with Nautilus, restarting OSDs worked just fine. The
problems started while upgradeing to Ocotopus.
Here is some OSD debug output from one OSD while stopping some other.
(debug_osd 10/10, grepped for laggy and waiting)
# cat /var/log/ceph/ceph-osd.313.log | egrep '(laggy|waiting)'
2021-06-15T16:13:51.605+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.47b6( v
43901'122473 (39559'119595,43901'122473] local-lis/les=43918/43919 n=15195
ec=34033/13925 lis/c=43335/43335 les/c/f=43336/43336/0 sis=43918 pruub=12.087973374s)
[313,761,693] r=0 lpr=43918 pi=[43335,43918)/1 crt=43901'122473 lcod 43894'122471
mlcod 0'0 active+undersized+degraded pruub 4840.521138536s@395 mbc={} trimq=[a4~1]
ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub
4840.521138536s > mnow 4828.443041141s
2021-06-15T16:13:51.620+0200 7fa824c33700 10 osd.313 pg_epoch: 43919 pg[2.1aee( v
43913'115951 (39517'112986,43913'115951] local-lis/les=43918/43919 n=15215
ec=13938/13925 lis/c=43243/43243 les/c/f=43244/43244/0 sis=43918 pruub=11.520263431s)
[313,671,763] r=0 lpr=43918 pi=[43243,43918)/1 crt=43913'115951 lcod 43911'115949
mlcod 0'0 active+undersized+degraded pruub 4839.953296047s@367 mbc={} trimq=[a4~1]
ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub
4839.953296047s > mnow 4828.458182468s
2021-06-15T16:13:51.621+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.5aee( v
43850'116365 (39537'113382,43850'116365] local-lis/les=43918/43919 n=14880
ec=35934/13925 lis/c=43243/43243 les/c/f=43244/43244/0 sis=43918 pruub=11.676727555s)
[313,671,763] r=0 lpr=43918 pi=[43243,43918)/1 crt=43850'116365 lcod 43850'116363
mlcod 0'0 active+undersized+degraded pruub 4840.106063793s@367 mbc={} ps=[a2~1,a4~1]]
state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.106063793s
> mnow 4828.459634014s
2021-06-15T16:13:51.626+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.39dd( v
43915'116280 (39489'113285,43915'116280] local-lis/les=43918/43919 n=14951
ec=13954/13925 lis/c=43331/43331 les/c/f=43332/43332/0 sis=43918 pruub=7.720345059s)
[313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43915'116280 lcod 43875'116278
mlcod 0'0 active+undersized+degraded pruub 4836.150186323s@419 mbc={} ps=[a2~1,a4~1]]
state<Started/Primary/Active>: waiting for prior_readable_until_ub 4836.150186323s
> mnow 4828.463817104s
2021-06-15T16:13:51.626+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.79dd( v
43911'116563 (39565'113592,43911'116563] local-lis/les=43918/43919 n=15008
ec=35977/13925 lis/c=43331/43331 les/c/f=43332/43332/0 sis=43918 pruub=7.874664992s)
[313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43911'116563 lcod 43901'116562
mlcod 0'0 active+undersized+degraded pruub 4836.305711908s@419 mbc={} ps=[a2~1,a4~1]]
state<Started/Primary/Active>: waiting for prior_readable_until_ub 4836.305711908s
> mnow 4828.464542182s
2021-06-15T16:13:51.627+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.79dd( v
43911'116563 (39565'113592,43911'116563] local-lis/les=43918/43919 n=15008
ec=35977/13925 lis/c=43918/43331 les/c/f=43919/43332/0 sis=43918 pruub=7.840968898s)
[313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43911'116563 lcod 43901'116562
mlcod 0'0 active+undersized+degraded+wait pruub 4836.305711908s@419 mbc={}
ps=[a2~1,a4~1]] check_laggy PG is WAIT state
2021-06-15T16:13:51.627+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.79dd( v
43911'116563 (39565'113592,43911'116563] local-lis/les=43918/43919 n=15008
ec=35977/13925 lis/c=43918/43331 les/c/f=43919/43332/0 sis=43918 pruub=7.840968898s)
[313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43911'116563 lcod 43901'116562
mlcod 0'0 active+undersized+degraded+wait pruub 4836.305711908s@419 mbc={}
ps=[a2~1,a4~1]] check_laggy not readable
2021-06-15T16:13:51.632+0200 7fa824c33700 10 osd.313 pg_epoch: 43919 pg[2.7b6( v
43915'120583 (39610'117714,43915'120583] local-lis/les=43918/43919 n=15292
ec=13928/13925 lis/c=43285/43285 les/c/f=43286/43286/0 sis=43918 pruub=11.740174435s)
[313,761,686] r=0 lpr=43918 pi=[43285,43918)/1 crt=43915'120583 lcod 43911'120581
mlcod 0'0 active+undersized+degraded pruub 4840.170557038s@395 mbc={} trimq=[a4~1]
ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub
4840.170557038s > mnow 4828.470484105s
2021-06-15T16:13:51.638+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.1a61( v
43901'117636 (39730'114897,43901'117636] local-lis/les=43918/43919 n=15068
ec=13938/13925 lis/c=43356/43356 les/c/f=43357/43357/0 sis=43918 pruub=12.126607104s)
[313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43901'117636 lcod 43881'117635
mlcod 0'0 active+undersized+degraded pruub 4840.556990188s@269 mbc={} trimq=[a4~1]
ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub
4840.556990188s > mnow 4828.476309716s
2021-06-15T16:13:51.638+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.5a61( v
43911'114881 (39797'112139,43911'114881] local-lis/les=43918/43919 n=15046
ec=35934/13925 lis/c=43356/43356 les/c/f=43357/43357/0 sis=43918 pruub=11.819966361s)
[313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43911'114881 lcod 43911'114879
mlcod 0'0 active+undersized+degraded pruub 4840.250617987s@269 mbc={} ps=[a2~1,a4~1]]
state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.250617987s
> mnow 4828.476475169s
2021-06-15T16:13:51.646+0200 7fa826c37700 10 osd.313 pg_epoch: 43919 pg[2.7dfa( v
43894'121976 (39366'119005,43894'121976] local-lis/les=43918/43919 n=15008
ec=35982/13925 lis/c=43227/43227 les/c/f=43228/43228/0 sis=43918 pruub=10.205329712s)
[313,868,699] r=0 lpr=43918 pi=[43227,43918)/1 crt=43894'121976 lcod 43894'121974
mlcod 0'0 active+undersized+degraded pruub 4838.635614602s@367 mbc={} ps=[a2~1,a4~1]]
state<Started/Primary/Active>: waiting for prior_readable_until_ub 4838.635614602s
> mnow 4828.484055710s
2021-06-15T16:13:51.648+0200 7fa826436700 10 osd.313 pg_epoch: 43919 pg[2.3dfa( v
43915'121818 (39519'118828,43915'121818] local-lis/les=43918/43919 n=15079
ec=13956/13925 lis/c=43227/43227 les/c/f=43228/43228/0 sis=43918 pruub=10.158707684s)
[313,868,699] r=0 lpr=43918 pi=[43227,43918)/1 crt=43915'121818 lcod 43894'121816
mlcod 0'0 active+undersized+degraded pruub 4838.588209355s@367 mbc={} trimq=[a4~1]
ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub
4838.588209355s > mnow 4828.486504603s
2021-06-15T16:14:02.617+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.1a61( v
43901'117636 (39730'114897,43901'117636] local-lis/les=43918/43919 n=15068
ec=13938/13925 lis/c=43918/43356 les/c/f=43919/43357/0 sis=43918 pruub=12.080355303s)
[313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43901'117636 lcod 43881'117635
mlcod 0'0 active+undersized+degraded+wait pruub 4840.556990188s@269 mbc={}
trimq=[a4~1] ps=[a2~1]] check_laggy PG is WAIT state
2021-06-15T16:14:02.618+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.1a61( v
43901'117636 (39730'114897,43901'117636] local-lis/les=43918/43919 n=15068
ec=13938/13925 lis/c=43918/43356 les/c/f=43919/43357/0 sis=43918 pruub=12.080355303s)
[313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43901'117636 lcod 43881'117635
mlcod 0'0 active+undersized+degraded+wait pruub 4840.556990188s@269 mbc={}
trimq=[a4~1] ps=[a2~1]] check_laggy not readable
On Fri, 11 Jun 2021 12:11:24 +0200
Peter Lieven <pl(a)kamp.de> wrote:
I would be interested if the time is also spent in "waiting for
readable" state.
To check this I reduced the slow ops timeout to 1 second and then
fetched the info about the
slow ops from the osd daemons that were affected.
Manuel, can you check this as well?
Thanks,
Peter
Am 11.06.21 um 11:48 schrieb Dan van der Ster:
> On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven <pl(a)kamp.de> wrote:
>> Am 10.06.21 um 17:45 schrieb Manuel Lausch:
I wonder if I faced the same issue. The issue I had
occured when
OSDs came back up and peering started.
My cluster was a fresh octopus install so I think the min osd
release was set to octopus.
Is it in general safe to stay with this switch at nautilus and run
octopus to run a maintained release?
I would guess not -- one should follow the
upgrade instructions as
documented. (I was merely confirming that Manuel had indeed followed
that procedure).
IMHO this issue, as described by Manuel, is not understood.
Manuel could you create a tracker and upload any relevant logs from
when the slow requests began?
11 Jun
11 Jun
5:04 p.m.
Okay, I poked around a bit more and found this document:
https://docs.ceph.com/en/latest/dev/osd_internals/stale_read/
I don't understand exactly what it is all about and how it works,
and what the intetion is behind it. But there is one config option
mentiond: "osd_pool_default_read_lease_ratio" This is defaulted to 0.8.
Multiplied with the osd_hearbeat_grace (which is default 20) it sets
that "read lease" to 16 seconds ?!
I set this ratio to 0.2 which leads to 4 seconds lease time. With that,
the problem is solved. No more slow ops.
Until now, I thought that this is a problem on huge clusters. But with
this setting I assumed that this should be a issue with quite small
cluster as well. So I tested it with a 3 Node 12 OSD SSD Cluster on
octopus with the same issues.
I can't believe I am the first one, which have this problem.
Manuel
On Thu, 10 Jun 2021 17:45:02 +0200
Manuel Lausch <manuel.lausch(a)1und1.de> wrote:
> Hi Peter,
>
> your suggestion pointed me to the right spot.
> I didn't know about the feature, that ceph will read from replica
> PGs.
>
> So on. I found two functions in the osd/PrimaryLogPG.cc:
> "check_laggy" and "check_laggy_requeue". On both is first a
check, if
> the partners have the octopus features. if not, the function is
> skipped. This explains the beginning of the problem after about the
> half cluster was updated.
>
> To verifiy this, I added "return true" in the first line of the
> functions. The issue is gone with it. But
> I don't know what problems this could trigger. I know, the root cause
> is not fixed with it.
> I think I will open a bug ticket with this knowlage.
>
>
> osd_op_queue_cutoff is set to high
> and a icmp rate limiting should not happen
>
>
> Thanks
> Manuel
1047
days inactive
1055
days old
11 comments
3 participants
participants (3)
-
Dan van der Ster -
Manuel Lausch -
Peter Lieven