slow ops at restarting OSDs (octopus)
Hello, I implemented a new cluster with 48 Nodes á 24 OSDs. I have a replicated pool with 4 replica. The crushrule distributes the replicas to different racks. With this cluster I tested a upgrade from Nautilis (14.2.20) to Octopus (15.2.13). The update itself worked well until I began the restarts of the OSDs in the 4th rack. Since then I get slow ops while stopping OSDs. I think something happend here, after all replica partners are running on the new version. This issue remains after completing the upgrade. With Nautilus I had similar issues with slow ops when stopping OSDs. I could resolve this with the option „osd_fast_shutdown → false“. I let this option set to false while upgrading. For testing/debugging, I set this to true (default value) and got better results when stopping OSDs, but the problem is not completely vanished. Had someone else this problem and could fix it? What can I do to get rid of slow ops when resarting OSDs? All Servers are connected with 2x10G network links Manuel
Hi, has no one a idea what could cause this issue. Or how I could debug it? In some days I have to go live with this cluster. If I don't have a solution I have to go live with nautilus. Manuel On Mon, 7 Jun 2021 15:46:18 +0200 Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hello,
I implemented a new cluster with 48 Nodes á 24 OSDs. I have a replicated pool with 4 replica. The crushrule distributes the replicas to different racks.
With this cluster I tested a upgrade from Nautilis (14.2.20) to Octopus (15.2.13). The update itself worked well until I began the restarts of the OSDs in the 4th rack. Since then I get slow ops while stopping OSDs. I think something happend here, after all replica partners are running on the new version. This issue remains after completing the upgrade.
With Nautilus I had similar issues with slow ops when stopping OSDs. I could resolve this with the option „osd_fast_shutdown → false“. I let this option set to false while upgrading. For testing/debugging, I set this to true (default value) and got better results when stopping OSDs, but the problem is not completely vanished.
Had someone else this problem and could fix it? What can I do to get rid of slow ops when resarting OSDs?
All Servers are connected with 2x10G network links
Manuel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Manuel Lausch Systemadministrator Storage Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lausch@1und1.de | Web: www.1und1.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Alexander Charles, Thomas Ludwig, Jan Oetjen, Sascha Vollmer Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail.
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
Hi,
has no one a idea what could cause this issue. Or how I could debug it?
In some days I have to go live with this cluster. If I don't have a solution I have to go live with nautilus.
Hi Manuel, I had similar issues with Octopus and i am thus stuck with Nautilus. Can you debug the slow ops and see if the slow ops are caused by the status "waiting for readable". I suspected that it has something to do with the new feature in Octopus to read from all OSDs regardless if they are master for a PG or not. Can you also verify that osd_op_queue_cut_off is set to high and that icmp rate limiting is disabled on your hosts? Peter
Hi Peter, your suggestion pointed me to the right spot. I didn't know about the feature, that ceph will read from replica PGs. So on. I found two functions in the osd/PrimaryLogPG.cc: "check_laggy" and "check_laggy_requeue". On both is first a check, if the partners have the octopus features. if not, the function is skipped. This explains the beginning of the problem after about the half cluster was updated. To verifiy this, I added "return true" in the first line of the functions. The issue is gone with it. But I don't know what problems this could trigger. I know, the root cause is not fixed with it. I think I will open a bug ticket with this knowlage. osd_op_queue_cutoff is set to high and a icmp rate limiting should not happen Thanks Manuel On Thu, 10 Jun 2021 11:28:48 +0200 Peter Lieven <pl@kamp.de> wrote:
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
Hi,
has no one a idea what could cause this issue. Or how I could debug it?
In some days I have to go live with this cluster. If I don't have a solution I have to go live with nautilus.
Hi Manuel,
I had similar issues with Octopus and i am thus stuck with Nautilus.
Can you debug the slow ops and see if the slow ops are caused by the status "waiting for readable".
I suspected that it has something to do with the new feature in Octopus to read from all OSDs regardless if
they are master for a PG or not.
Can you also verify that osd_op_queue_cut_off is set to high and that icmp rate limiting is disabled on your hosts?
Peter
Hi, At which point in the update procedure did you ceph osd require-osd-release octopus ? And are you sure it was set to nautilus before the update? (`ceph osd dump` will show) Cheers , Dan On Thu, Jun 10, 2021, 5:45 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi Peter,
your suggestion pointed me to the right spot. I didn't know about the feature, that ceph will read from replica PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc: "check_laggy" and "check_laggy_requeue". On both is first a check, if the partners have the octopus features. if not, the function is skipped. This explains the beginning of the problem after about the half cluster was updated.
To verifiy this, I added "return true" in the first line of the functions. The issue is gone with it. But I don't know what problems this could trigger. I know, the root cause is not fixed with it. I think I will open a bug ticket with this knowlage.
osd_op_queue_cutoff is set to high and a icmp rate limiting should not happen
Thanks Manuel
On Thu, 10 Jun 2021 11:28:48 +0200 Peter Lieven <pl@kamp.de> wrote:
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
Hi,
has no one a idea what could cause this issue. Or how I could debug it?
In some days I have to go live with this cluster. If I don't have a solution I have to go live with nautilus.
Hi Manuel,
I had similar issues with Octopus and i am thus stuck with Nautilus.
Can you debug the slow ops and see if the slow ops are caused by the status "waiting for readable".
I suspected that it has something to do with the new feature in Octopus to read from all OSDs regardless if
they are master for a PG or not.
Can you also verify that osd_op_queue_cut_off is set to high and that icmp rate limiting is disabled on your hosts?
Peter
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, The cluster was initialy deployed with nautilus (14.2.20). I am sure require-osd-release was nautilus at this point. I did set this to octopus, after all components was updatated. Manuel On Thu, 10 Jun 2021 17:54:49 +0200 Dan van der Ster <dan@vanderster.com> wrote:
Hi,
At which point in the update procedure did you
ceph osd require-osd-release octopus
?
And are you sure it was set to nautilus before the update? (`ceph osd dump` will show)
Cheers , Dan
Ok sounds correct. This was the only thing that came to mind which might explain your problem. Cheers, Dan On Thu, Jun 10, 2021, 6:06 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi Dan,
The cluster was initialy deployed with nautilus (14.2.20). I am sure require-osd-release was nautilus at this point. I did set this to octopus, after all components was updatated.
Manuel
On Thu, 10 Jun 2021 17:54:49 +0200 Dan van der Ster <dan@vanderster.com> wrote:
Hi,
At which point in the update procedure did you
ceph osd require-osd-release octopus
?
And are you sure it was set to nautilus before the update? (`ceph osd dump` will show)
Cheers , Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
Hi Peter,
your suggestion pointed me to the right spot. I didn't know about the feature, that ceph will read from replica PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc: "check_laggy" and "check_laggy_requeue". On both is first a check, if the partners have the octopus features. if not, the function is skipped. This explains the beginning of the problem after about the half cluster was updated.
To verifiy this, I added "return true" in the first line of the functions. The issue is gone with it. But I don't know what problems this could trigger. I know, the root cause is not fixed with it. I think I will open a bug ticket with this knowlage.
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and peering started. My cluster was a fresh octopus install so I think the min osd release was set to octopus. Is it in general safe to stay with this switch at nautilus and run octopus to run a maintained release?
osd_op_queue_cutoff is set to high and a icmp rate limiting should not happen
It could if you choose fast shutdown and connections to the OSD daemon are refused with icmp port unreachable?! Peter
On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven <pl@kamp.de> wrote:
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
Hi Peter,
your suggestion pointed me to the right spot. I didn't know about the feature, that ceph will read from replica PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc: "check_laggy" and "check_laggy_requeue". On both is first a check, if the partners have the octopus features. if not, the function is skipped. This explains the beginning of the problem after about the half cluster was updated.
To verifiy this, I added "return true" in the first line of the functions. The issue is gone with it. But I don't know what problems this could trigger. I know, the root cause is not fixed with it. I think I will open a bug ticket with this knowlage.
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and peering started.
My cluster was a fresh octopus install so I think the min osd release was set to octopus.
Is it in general safe to stay with this switch at nautilus and run octopus to run a maintained release?
I would guess not -- one should follow the upgrade instructions as documented. (I was merely confirming that Manuel had indeed followed that procedure). IMHO this issue, as described by Manuel, is not understood. Manuel could you create a tracker and upload any relevant logs from when the slow requests began? -- dan
osd_op_queue_cutoff is set to high and a icmp rate limiting should not happen
It could if you choose fast shutdown and connections to the OSD daemon are refused with icmp port unreachable?!
Peter
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 11.06.21 um 11:48 schrieb Dan van der Ster:
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
Hi Peter,
your suggestion pointed me to the right spot. I didn't know about the feature, that ceph will read from replica PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc: "check_laggy" and "check_laggy_requeue". On both is first a check, if the partners have the octopus features. if not, the function is skipped. This explains the beginning of the problem after about the half cluster was updated.
To verifiy this, I added "return true" in the first line of the functions. The issue is gone with it. But I don't know what problems this could trigger. I know, the root cause is not fixed with it. I think I will open a bug ticket with this knowlage.
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and peering started.
My cluster was a fresh octopus install so I think the min osd release was set to octopus.
Is it in general safe to stay with this switch at nautilus and run octopus to run a maintained release? I would guess not -- one should follow the upgrade instructions as documented. (I was merely confirming that Manuel had indeed followed
On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven <pl@kamp.de> wrote: that procedure).
IMHO this issue, as described by Manuel, is not understood.
Manuel could you create a tracker and upload any relevant logs from when the slow requests began?
I would be interested if the time is also spent in "waiting for readable" state. To check this I reduced the slow ops timeout to 1 second and then fetched the info about the slow ops from the osd daemons that were affected. Manuel, can you check this as well? Thanks, Peter
Hi, yeah. I wasn't aware, that I have set "osd op complaint time" to 5 seconds. AFAIK the default is 32. So I get slow ops already after 5 instead of 32 seconds. Thats why I think no one before have noticed this before. My application which uses the cluster will throw timeouts after 6 seconds, thats why we set this down. For clarification, with Nautilus, restarting OSDs worked just fine. The problems started while upgradeing to Ocotopus. Here is some OSD debug output from one OSD while stopping some other. (debug_osd 10/10, grepped for laggy and waiting) # cat /var/log/ceph/ceph-osd.313.log | egrep '(laggy|waiting)' 2021-06-15T16:13:51.605+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.47b6( v 43901'122473 (39559'119595,43901'122473] local-lis/les=43918/43919 n=15195 ec=34033/13925 lis/c=43335/43335 les/c/f=43336/43336/0 sis=43918 pruub=12.087973374s) [313,761,693] r=0 lpr=43918 pi=[43335,43918)/1 crt=43901'122473 lcod 43894'122471 mlcod 0'0 active+undersized+degraded pruub 4840.521138536s@395 mbc={} trimq=[a4~1] ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.521138536s > mnow 4828.443041141s 2021-06-15T16:13:51.620+0200 7fa824c33700 10 osd.313 pg_epoch: 43919 pg[2.1aee( v 43913'115951 (39517'112986,43913'115951] local-lis/les=43918/43919 n=15215 ec=13938/13925 lis/c=43243/43243 les/c/f=43244/43244/0 sis=43918 pruub=11.520263431s) [313,671,763] r=0 lpr=43918 pi=[43243,43918)/1 crt=43913'115951 lcod 43911'115949 mlcod 0'0 active+undersized+degraded pruub 4839.953296047s@367 mbc={} trimq=[a4~1] ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4839.953296047s > mnow 4828.458182468s 2021-06-15T16:13:51.621+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.5aee( v 43850'116365 (39537'113382,43850'116365] local-lis/les=43918/43919 n=14880 ec=35934/13925 lis/c=43243/43243 les/c/f=43244/43244/0 sis=43918 pruub=11.676727555s) [313,671,763] r=0 lpr=43918 pi=[43243,43918)/1 crt=43850'116365 lcod 43850'116363 mlcod 0'0 active+undersized+degraded pruub 4840.106063793s@367 mbc={} ps=[a2~1,a4~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.106063793s > mnow 4828.459634014s 2021-06-15T16:13:51.626+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.39dd( v 43915'116280 (39489'113285,43915'116280] local-lis/les=43918/43919 n=14951 ec=13954/13925 lis/c=43331/43331 les/c/f=43332/43332/0 sis=43918 pruub=7.720345059s) [313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43915'116280 lcod 43875'116278 mlcod 0'0 active+undersized+degraded pruub 4836.150186323s@419 mbc={} ps=[a2~1,a4~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4836.150186323s > mnow 4828.463817104s 2021-06-15T16:13:51.626+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.79dd( v 43911'116563 (39565'113592,43911'116563] local-lis/les=43918/43919 n=15008 ec=35977/13925 lis/c=43331/43331 les/c/f=43332/43332/0 sis=43918 pruub=7.874664992s) [313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43911'116563 lcod 43901'116562 mlcod 0'0 active+undersized+degraded pruub 4836.305711908s@419 mbc={} ps=[a2~1,a4~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4836.305711908s > mnow 4828.464542182s 2021-06-15T16:13:51.627+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.79dd( v 43911'116563 (39565'113592,43911'116563] local-lis/les=43918/43919 n=15008 ec=35977/13925 lis/c=43918/43331 les/c/f=43919/43332/0 sis=43918 pruub=7.840968898s) [313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43911'116563 lcod 43901'116562 mlcod 0'0 active+undersized+degraded+wait pruub 4836.305711908s@419 mbc={} ps=[a2~1,a4~1]] check_laggy PG is WAIT state 2021-06-15T16:13:51.627+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.79dd( v 43911'116563 (39565'113592,43911'116563] local-lis/les=43918/43919 n=15008 ec=35977/13925 lis/c=43918/43331 les/c/f=43919/43332/0 sis=43918 pruub=7.840968898s) [313,1107,657] r=0 lpr=43918 pi=[43331,43918)/1 crt=43911'116563 lcod 43901'116562 mlcod 0'0 active+undersized+degraded+wait pruub 4836.305711908s@419 mbc={} ps=[a2~1,a4~1]] check_laggy not readable 2021-06-15T16:13:51.632+0200 7fa824c33700 10 osd.313 pg_epoch: 43919 pg[2.7b6( v 43915'120583 (39610'117714,43915'120583] local-lis/les=43918/43919 n=15292 ec=13928/13925 lis/c=43285/43285 les/c/f=43286/43286/0 sis=43918 pruub=11.740174435s) [313,761,686] r=0 lpr=43918 pi=[43285,43918)/1 crt=43915'120583 lcod 43911'120581 mlcod 0'0 active+undersized+degraded pruub 4840.170557038s@395 mbc={} trimq=[a4~1] ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.170557038s > mnow 4828.470484105s 2021-06-15T16:13:51.638+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.1a61( v 43901'117636 (39730'114897,43901'117636] local-lis/les=43918/43919 n=15068 ec=13938/13925 lis/c=43356/43356 les/c/f=43357/43357/0 sis=43918 pruub=12.126607104s) [313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43901'117636 lcod 43881'117635 mlcod 0'0 active+undersized+degraded pruub 4840.556990188s@269 mbc={} trimq=[a4~1] ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.556990188s > mnow 4828.476309716s 2021-06-15T16:13:51.638+0200 7fa825c35700 10 osd.313 pg_epoch: 43919 pg[2.5a61( v 43911'114881 (39797'112139,43911'114881] local-lis/les=43918/43919 n=15046 ec=35934/13925 lis/c=43356/43356 les/c/f=43357/43357/0 sis=43918 pruub=11.819966361s) [313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43911'114881 lcod 43911'114879 mlcod 0'0 active+undersized+degraded pruub 4840.250617987s@269 mbc={} ps=[a2~1,a4~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4840.250617987s > mnow 4828.476475169s 2021-06-15T16:13:51.646+0200 7fa826c37700 10 osd.313 pg_epoch: 43919 pg[2.7dfa( v 43894'121976 (39366'119005,43894'121976] local-lis/les=43918/43919 n=15008 ec=35982/13925 lis/c=43227/43227 les/c/f=43228/43228/0 sis=43918 pruub=10.205329712s) [313,868,699] r=0 lpr=43918 pi=[43227,43918)/1 crt=43894'121976 lcod 43894'121974 mlcod 0'0 active+undersized+degraded pruub 4838.635614602s@367 mbc={} ps=[a2~1,a4~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4838.635614602s > mnow 4828.484055710s 2021-06-15T16:13:51.648+0200 7fa826436700 10 osd.313 pg_epoch: 43919 pg[2.3dfa( v 43915'121818 (39519'118828,43915'121818] local-lis/les=43918/43919 n=15079 ec=13956/13925 lis/c=43227/43227 les/c/f=43228/43228/0 sis=43918 pruub=10.158707684s) [313,868,699] r=0 lpr=43918 pi=[43227,43918)/1 crt=43915'121818 lcod 43894'121816 mlcod 0'0 active+undersized+degraded pruub 4838.588209355s@367 mbc={} trimq=[a4~1] ps=[a2~1]] state<Started/Primary/Active>: waiting for prior_readable_until_ub 4838.588209355s > mnow 4828.486504603s 2021-06-15T16:14:02.617+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.1a61( v 43901'117636 (39730'114897,43901'117636] local-lis/les=43918/43919 n=15068 ec=13938/13925 lis/c=43918/43356 les/c/f=43919/43357/0 sis=43918 pruub=12.080355303s) [313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43901'117636 lcod 43881'117635 mlcod 0'0 active+undersized+degraded+wait pruub 4840.556990188s@269 mbc={} trimq=[a4~1] ps=[a2~1]] check_laggy PG is WAIT state 2021-06-15T16:14:02.618+0200 7fa825434700 10 osd.313 pg_epoch: 43919 pg[2.1a61( v 43901'117636 (39730'114897,43901'117636] local-lis/les=43918/43919 n=15068 ec=13938/13925 lis/c=43918/43356 les/c/f=43919/43357/0 sis=43918 pruub=12.080355303s) [313,1116,951] r=0 lpr=43918 pi=[43356,43918)/1 crt=43901'117636 lcod 43881'117635 mlcod 0'0 active+undersized+degraded+wait pruub 4840.556990188s@269 mbc={} trimq=[a4~1] ps=[a2~1]] check_laggy not readable On Fri, 11 Jun 2021 12:11:24 +0200 Peter Lieven <pl@kamp.de> wrote:
Am 11.06.21 um 11:48 schrieb Dan van der Ster:
On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven <pl@kamp.de> wrote:
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
I wonder if I faced the same issue. The issue I had occured when OSDs came back up and peering started.
My cluster was a fresh octopus install so I think the min osd release was set to octopus.
Is it in general safe to stay with this switch at nautilus and run octopus to run a maintained release? I would guess not -- one should follow the upgrade instructions as documented. (I was merely confirming that Manuel had indeed followed that procedure).
IMHO this issue, as described by Manuel, is not understood.
Manuel could you create a tracker and upload any relevant logs from when the slow requests began?
I would be interested if the time is also spent in "waiting for readable" state.
To check this I reduced the slow ops timeout to 1 second and then fetched the info about the
slow ops from the osd daemons that were affected.
Manuel, can you check this as well?
Thanks,
Peter
Okay, I poked around a bit more and found this document: https://docs.ceph.com/en/latest/dev/osd_internals/stale_read/ I don't understand exactly what it is all about and how it works, and what the intetion is behind it. But there is one config option mentiond: "osd_pool_default_read_lease_ratio" This is defaulted to 0.8. Multiplied with the osd_hearbeat_grace (which is default 20) it sets that "read lease" to 16 seconds ?! I set this ratio to 0.2 which leads to 4 seconds lease time. With that, the problem is solved. No more slow ops. Until now, I thought that this is a problem on huge clusters. But with this setting I assumed that this should be a issue with quite small cluster as well. So I tested it with a 3 Node 12 OSD SSD Cluster on octopus with the same issues. I can't believe I am the first one, which have this problem. Manuel On Thu, 10 Jun 2021 17:45:02 +0200 Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi Peter,
your suggestion pointed me to the right spot. I didn't know about the feature, that ceph will read from replica PGs.
So on. I found two functions in the osd/PrimaryLogPG.cc: "check_laggy" and "check_laggy_requeue". On both is first a check, if the partners have the octopus features. if not, the function is skipped. This explains the beginning of the problem after about the half cluster was updated.
To verifiy this, I added "return true" in the first line of the functions. The issue is gone with it. But I don't know what problems this could trigger. I know, the root cause is not fixed with it. I think I will open a bug ticket with this knowlage.
osd_op_queue_cutoff is set to high and a icmp rate limiting should not happen
Thanks Manuel
participants (3)
-
Dan van der Ster
-
Manuel Lausch
-
Peter Lieven