Nautilus slow using "ceph tell osd.* bench"
I have 2 clusters. Cluster 1 started at Hammer and has upgraded through the versions all the way to Nautilus 14.2.10 (Luminous to Nautilus in July 2020) . Cluster 2 started as Luminous and is now Nautilus 14.2.2 (Upgraded in September 2019) The clusters are basically identical 5 OSD Nodes with 6 osd's per node. They are both using disk drives. No SSD's. Prior to upgrading Cluster 1 running "ceph tell osd.0 bench -f plain" produced similar results across both clusters. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.954819 sec at 1.0 GiB/sec 268 IOPS Now cluster 1 results are terrible, about 25% from before the upgrade. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS Ceph -s shows HEALTH_OK. Dashboard looks good. 2 pools MON Dump min_mon_release 14 (nautilus) OSD Dump require_min_compat_client luminous min_compat_client jewel require_osd_release nautilus Not sure what is causing the slow performance. Ideas?
Hi Jim, did you check system stat (e.g. iostat, top, etc.) on both osds when you ran osd bench? Those might be able to give you some clues. Moreover, did you compare both osds' configurations? ------------------ Original ------------------ From: "Jim Forde" <jimf@mninc.net>; Date: Thu, Aug 6, 2020 06:51 AM To: "ceph-users"<ceph-users@ceph.io>; Subject: [ceph-users] Nautilus slow using "ceph tell osd.* bench" I have 2 clusters. Cluster 1 started at Hammer and has upgraded through the versions all the way to Nautilus 14.2.10 (Luminous to Nautilus in July 2020) . Cluster 2 started as Luminous and is now Nautilus 14.2.2 (Upgraded in September 2019) The clusters are basically identical 5 OSD Nodes with 6 osd's per node. They are both using disk drives. No SSD's. Prior to upgrading Cluster 1 running "ceph tell osd.0 bench -f plain" produced similar results across both clusters. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.954819 sec at 1.0 GiB/sec 268 IOPS Now cluster 1 results are terrible, about 25% from before the upgrade. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS Ceph -s shows HEALTH_OK. Dashboard looks good. 2 pools MON Dump min_mon_release 14 (nautilus) OSD Dump require_min_compat_client luminous min_compat_client jewel require_osd_release nautilus Not sure what is causing the slow performance. Ideas? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I echo Jim's findings. Going from lower Nautilus versions up to .10 on 4 installations suddenly gave a huge drop in read performance, probably by about 3/4, so that my users were complaining that VMs were taking ages to boot up. Strangely, write performance was not affected so much. Kind regards, Dave ------ Original Message ------ From: "rainning" <tweetypie@qq.com> To: "Jim Forde" <jimf@mninc.net>; "ceph-users" <ceph-users@ceph.io> Sent: Thursday, 6 Aug, 2020 At 02:02 Subject: [ceph-users] Re: Nautilus slow using "ceph tell osd.* bench" Hi Jim, did you check system stat (e.g. iostat, top, etc.) on both osds when you ran osd bench? Those might be able to give you some clues. Moreover, did you compare both osds' configurations? ------------------ Original ------------------ From: "Jim Forde" <jimf@mninc.net>; Date: Thu, Aug 6, 2020 06:51 AM To: "ceph-users"<ceph-users@ceph.io>; Subject: [ceph-users] Nautilus slow using "ceph tell osd.* bench" I have 2 clusters. Cluster 1 started at Hammer and has upgraded through the versions all the way to Nautilus 14.2.10 (Luminous to Nautilus in July 2020) . Cluster 2 started as Luminous and is now Nautilus 14.2.2 (Upgraded in September 2019) The clusters are basically identical 5 OSD Nodes with 6 osd's per node. They are both using disk drives. No SSD's. Prior to upgrading Cluster 1 running "ceph tell osd.0 bench -f plain" produced similar results across both clusters. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.954819 sec at 1.0 GiB/sec 268 IOPS Now cluster 1 results are terrible, about 25% from before the upgrade. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS Ceph -s shows HEALTH_OK. Dashboard looks good. 2 pools MON Dump min_mon_release 14 (nautilus) OSD Dump require_min_compat_client luminous min_compat_client jewel require_osd_release nautilus Not sure what is causing the slow performance. Ideas? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
SOLUTION FOUND! Reweight the osd to 0, then set it back to where it belongs. ceph osd crush reweight osd.0 0.0 Original ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS After reweight of osd.0 ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 1.54555 sec at 663 MiB/sec 165 IOPS ceph tell osd.1 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 3.54652 sec at 289 MiB/sec 72 IOPS After reweight of osd.1 ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.948457 sec at 1.1 GiB/sec 269 IOPS ceph tell osd.1 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.949384 sec at 1.1 GiB/sec 269 IOPS ceph tell osd.2 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 3.56726 sec at 287 MiB/sec 71 IOPS I have finished reweight proceedure on osd node 1 and all 6 osd's are back where they belong, but have 4 more nodes to go. Looks like this should fix it. If anyone has an alternative method for getting around this I am all ears. Dave, would be interested to hear if this works for you. -Jim
Hi Jim, when you do reweighting, balancing will be triggered, how did you set it back? Setting back immediately or waiting for balancing to complete? I did try both on my cluster and couldn't see osd bench changed significantly like yours (actually no changes), however, my cluster is 12.2.12, not sure if that is the reason. Moreover, I really can't figure out why flipping the reweight can make such difference. Hope experts can explain that. ------------------ Original ------------------ From: "Jim Forde";<jimf@mninc.net>; Date: Aug 7, 2020 To: "ceph-users"<ceph-users@ceph.io>; Subject: [ceph-users] Re: Nautilus slow using "ceph tell osd.* bench" SOLUTION FOUND! Reweight the osd to 0, then set it back to where it belongs. ceph osd crush reweight osd.0 0.0 Original ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS After reweight of osd.0 ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 1.54555 sec at 663 MiB/sec 165 IOPS ceph tell osd.1 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 3.54652 sec at 289 MiB/sec 72 IOPS After reweight of osd.1 ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.948457 sec at 1.1 GiB/sec 269 IOPS ceph tell osd.1 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.949384 sec at 1.1 GiB/sec 269 IOPS ceph tell osd.2 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 3.56726 sec at 287 MiB/sec 71 IOPS I have finished reweight proceedure on osd node 1 and all 6 osd's are back where they belong, but have 4 more nodes to go. Looks like this should fix it. If anyone has an alternative method for getting around this I am all ears. Dave, would be interested to hear if this works for you. -Jim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have set it to 0.0 and let it re-balance. Then I set it back and let it re-balance again. I have a fairly small cluster, and while in production, it is not getting much use because of pandemic. So a good time to do some of these things. Because of that I have been re-balancing the osd's in groups of 3. Finished 2 nodes, with a little luck I will finish all 5 by tomorrow. Will post at completion. Off the top of my head I think it may have to do with "ceph osd crush set-all-straw-buckets-to-straw2" since that is supposed to cause a re-balance.
Solution Failed! I rebalanced all osd's to 0.0 and then back to their original weight, and started getting back to my original ~269 IOPS. It has been about 5 days since I completed the re-balance and performance is degrading again! There is a bit of improvement but not to where it was in Mimic. ceph tell osd.* bench -f plain osd.0: bench: wrote 1 GiB in blocks of 4 MiB in 2.92178 sec at 350 MiB/sec 87 IOPS osd.1: bench: wrote 1 GiB in blocks of 4 MiB in 2.46215 sec at 416 MiB/sec 103 IOPS osd.2: bench: wrote 1 GiB in blocks of 4 MiB in 1.6428 sec at 623 MiB/sec 155 IOPS osd.3: bench: wrote 1 GiB in blocks of 4 MiB in 1.78536 sec at 574 MiB/sec 143 IOPS osd.4: bench: wrote 1 GiB in blocks of 4 MiB in 2.49857 sec at 410 MiB/sec 102 IOPS osd.5: bench: wrote 1 GiB in blocks of 4 MiB in 2.68055 sec at 382 MiB/sec 95 IOPS Ceph version is 14.2.10 I realize that ceph tell osd bench is not the only metric, but it is confusing. It should be giving me an idea of which hardware is performing better. My second cluster on version 14.2.2 does have the IOPs fluctuate a little but the results of this test on cluster #2 are consistently twice the speed that I am getting on cluster #1. Still looking for an answer.
participants (3)
-
david.neal
-
Jim Forde
-
rainning