mclock scheduler kills clients IOs
Hi guys, we have Ceph cluster with Quincy 17.2.7 where we are draining 3 hosts (each one is from one failure domain). Each host has 13 HDDs and there are another 38 hosts with same size in each failure domain. There is a lot of free space. I've set up primary-affinity on drained OSDs to 0 and set OSDs out mclock profile is: osd_mclock_profile = high_client_ops but soon after that I got slow ops on some of the OSDs and clients getting high iowaits until I stop recovering. It's strange to me that client is still trying to use OSDs where primary-affinity is set to 0 Here is the case of osd.15 with primary-affinity 0: REQUESTS 4 homeless 0 8337634 osd15 16.35734e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5278.00000004 0x400024 1 write 8358040 osd15 16.ace50e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5a4e.00000007 0x400024 1 write 8438861 osd15 16.a7b80e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7c6a.0000001f 0x400024 1 write 8443242 osd15 16.de504e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7d76.00000014 0x400024 1 write LINGER REQUESTS BACKOFFS Recovery speed is also not high (about 4GiB) and network is not saturated. I've tried also following custom profile with other custom settings: osd advanced osd_mclock_override_recovery_settings true osd advanced osd_async_recovery_min_cost 100 osd advanced osd_max_backfills 1 osd advanced osd_mclock_profile custom osd advanced osd_mclock_scheduler_background_recovery_res 0.100000 osd advanced osd_mclock_scheduler_client_res 0.900000 osd advanced osd_mclock_scheduler_client_wgt 6 osd advanced osd_recovery_max_active 1 osd advanced osd_recovery_max_chunk 8388608 osd advanced osd_recovery_op_priority 1 but result is same. Immediately clients get high iowaits and can't do any IOs. This didn't happen on this cluster before when we were using wpq scheduler. My question is - are there some other options I need to set? With wpq I was able to set osd_recovery_sleep but this option looks like not possble to change with mclock. Any ideas, please?
Hi Denis, We have had the same issue with MClock, and now switch back to WPQ when draining nodes. I couldn’t identify the cause of the slow-ops. Even with custom mclock tuning and 1 backfill per osd, it was still causing client io issues. From: Denis Polom <denispolom@gmail.com> Date: Tuesday, 17 September 2024 at 8:16 pm To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [EXT] [ceph-users] mclock scheduler kills clients IOs External email: Please exercise caution Hi guys, we have Ceph cluster with Quincy 17.2.7 where we are draining 3 hosts (each one is from one failure domain). Each host has 13 HDDs and there are another 38 hosts with same size in each failure domain. There is a lot of free space. I've set up primary-affinity on drained OSDs to 0 and set OSDs out mclock profile is: osd_mclock_profile = high_client_ops but soon after that I got slow ops on some of the OSDs and clients getting high iowaits until I stop recovering. It's strange to me that client is still trying to use OSDs where primary-affinity is set to 0 Here is the case of osd.15 with primary-affinity 0: REQUESTS 4 homeless 0 8337634 osd15 16.35734e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5278.00000004 0x400024 1 write 8358040 osd15 16.ace50e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5a4e.00000007 0x400024 1 write 8438861 osd15 16.a7b80e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7c6a.0000001f 0x400024 1 write 8443242 osd15 16.de504e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7d76.00000014 0x400024 1 write LINGER REQUESTS BACKOFFS Recovery speed is also not high (about 4GiB) and network is not saturated. I've tried also following custom profile with other custom settings: osd advanced osd_mclock_override_recovery_settings true osd advanced osd_async_recovery_min_cost 100 osd advanced osd_max_backfills 1 osd advanced osd_mclock_profile custom osd advanced osd_mclock_scheduler_background_recovery_res 0.100000 osd advanced osd_mclock_scheduler_client_res 0.900000 osd advanced osd_mclock_scheduler_client_wgt 6 osd advanced osd_recovery_max_active 1 osd advanced osd_recovery_max_chunk 8388608 osd advanced osd_recovery_op_priority 1 but result is same. Immediately clients get high iowaits and can't do any IOs. This didn't happen on this cluster before when we were using wpq scheduler. My question is - are there some other options I need to set? With wpq I was able to set osd_recovery_sleep but this option looks like not possble to change with mclock. Any ideas, please? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, yes mclock scheduler doesn't looks like stable and ready for production Ceph cluster. I just switched back to wpq and everything goes smoothly. thx! On 17. 09. 24 13:14, Justin Mammarella wrote:
Hi Denis,
We have had the same issue with MClock, and now switch back to WPQ when draining nodes.
I couldn’t identify the cause of the slow-ops. Even with custom mclock tuning and
1 backfill per osd, it was still causing client io issues.
*From: *Denis Polom <denispolom@gmail.com> *Date: *Tuesday, 17 September 2024 at 8:16 pm *To: *ceph-users@ceph.io <ceph-users@ceph.io> *Subject: *[EXT] [ceph-users] mclock scheduler kills clients IOs
External email: Please exercise caution
Hi guys,
we have Ceph cluster with Quincy 17.2.7 where we are draining 3 hosts (each one is from one failure domain). Each host has 13 HDDs and there are another 38 hosts with same size in each failure domain. There is a lot of free space.
I've set up
primary-affinity on drained OSDs to 0
and set OSDs out
mclock profile is:
osd_mclock_profile = high_client_ops
but soon after that I got slow ops on some of the OSDs and clients getting high iowaits until I stop recovering.
It's strange to me that client is still trying to use OSDs where primary-affinity is set to 0
Here is the case of osd.15 with primary-affinity 0:
REQUESTS 4 homeless 0 8337634 osd15 16.35734e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5278.00000004 0x400024 1 write 8358040 osd15 16.ace50e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5a4e.00000007 0x400024 1 write 8438861 osd15 16.a7b80e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7c6a.0000001f 0x400024 1 write 8443242 osd15 16.de504e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7d76.00000014 0x400024 1 write LINGER REQUESTS BACKOFFS
Recovery speed is also not high (about 4GiB) and network is not saturated.
I've tried also following custom profile with other custom settings:
osd advanced osd_mclock_override_recovery_settings true
osd advanced osd_async_recovery_min_cost 100 osd advanced osd_max_backfills 1 osd advanced osd_mclock_profile custom osd advanced osd_mclock_scheduler_background_recovery_res 0.100000 osd advanced osd_mclock_scheduler_client_res 0.900000 osd advanced osd_mclock_scheduler_client_wgt 6 osd advanced osd_recovery_max_active 1 osd advanced osd_recovery_max_chunk 8388608 osd advanced osd_recovery_op_priority 1
but result is same. Immediately clients get high iowaits and can't do any IOs.
This didn't happen on this cluster before when we were using wpq scheduler.
My question is - are there some other options I need to set?
With wpq I was able to set osd_recovery_sleep but this option looks like not possble to change with mclock.
Any ideas, please?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Sep 17, 2024 at 04:22:40PM +0200, Denis Polom wrote:
Hi,
yes mclock scheduler doesn't looks like stable and ready for production Ceph cluster. I just switched back to wpq and everything goes smoothly.
In our cluster all IO stopped when I set 3 OSD to out when running Mclock. After switching to WPQ and had run deep-scrub on all PG the result was 698 corrupted objects that Ceph could not fix. So no, I would not say Mclock i production ready. We have set all out cluster to WPQ. -- Kai Stian Olstad
Hi Denis, we observed the same behaviour here. The cause was that the number of iops discovered at OSD startup was way too high. In our setup the rocksdb is on flash. When I set osd_mclock_max_capacity_iops_hdd to a value that the HDDs could handle, the situation was resolved, clients got ther fair share of IO. Hope this helps, Daniel -- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444
Hi, the problem comes from older ceph releases. In our case, hdd iops were benchmarked in the range of 250 to 4000, which clearly makes no sense. At osd startup, the benchmark is skipped if that value is already in ceph config, so these initial benchmark values were never changed. To reset them, all osd.N osd_mclock_max_capacity_iops_hdd values should be removed and osds restarted. There is a safety mechanism (osd_mclock_iops_capacity_threshold_hdd) which prevents for the values to be overestimated. Best, Andrej On 19. 09. 24 11:33, Daniel Schreiber wrote:
Hi Denis,
we observed the same behaviour here. The cause was that the number of iops discovered at OSD startup was way too high. In our setup the rocksdb is on flash.
When I set osd_mclock_max_capacity_iops_hdd to a value that the HDDs could handle, the situation was resolved, clients got ther fair share of IO.
Hope this helps,
Daniel
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------
We’re running Quincy 17.2.7 here, And we see the iops benchmark performed on osd start: 2024-09-20T09:57:26.265+1000 7facbdc64540 1 osd.196 2879010 maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth (MiB/sec): 4.369 iops: 1118.445 elapsed_sec: 2.682 2024-09-20T09:57:26.265+1000 7facbdc64540 0 log_channel(cluster) log [WRN] : OSD bench result of 1118.445008 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.196. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. Has the accuracy of this benchmark been improved in newer releases? From: Andrej Filipcic <andrej.filipcic@ijs.si> Date: Thursday, 19 September 2024 at 9:20 pm To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [EXT] [ceph-users] Re: mclock scheduler kills clients IOs External email: Please exercise caution Hi, the problem comes from older ceph releases. In our case, hdd iops were benchmarked in the range of 250 to 4000, which clearly makes no sense. At osd startup, the benchmark is skipped if that value is already in ceph config, so these initial benchmark values were never changed. To reset them, all osd.N osd_mclock_max_capacity_iops_hdd values should be removed and osds restarted. There is a safety mechanism (osd_mclock_iops_capacity_threshold_hdd) which prevents for the values to be overestimated. Best, Andrej On 19. 09. 24 11:33, Daniel Schreiber wrote:
Hi Denis,
we observed the same behaviour here. The cause was that the number of iops discovered at OSD startup was way too high. In our setup the rocksdb is on flash.
When I set osd_mclock_max_capacity_iops_hdd to a value that the HDDs could handle, the situation was resolved, clients got ther fair share of IO.
Hope this helps,
Daniel
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 ------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Andrej Filipcic
-
Daniel Schreiber
-
Denis Polom
-
Justin Mammarella
-
Kai Stian Olstad