About ceph disk slowops effect to cluster
Hi community I'm currently facing a significant issue with my Ceph cluster. I have a cluster consisting of 10 nodes, and each node is equipped with 6 SSDs of 960GB used for block.db and 18 12TB drives used for data, network bonding 2x10Gbps for public and local networks. I am using a 4+2 erasure code for RBD in my Ceph cluster. When one node becomes unavailable, the cluster initiates the recovery process, and subsequently, slow operations (slowops) logs appear on the disk, impacting the entire cluster. Afterward, additional nodes are marked as failures. Is this phenomenon possibly due to the performance of SSDs and HDDs? When I check the I/O of the disk using the iostat command, the result shows that disk utilization has reached 80-90% Is using a combination of HDDs and SAS SSDs in Ceph a choice leading to poor performance? My Ceph cluster has a bandwidth of 1.9GB/s Thanks and hope someone can help me ---------------------------------------------------------------------------- Email: tranphong079@gmail.com Skype: tranphong079
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
Hi Yang and Anthony, I found the solution for this problem on a HDD disk 7200rpm When the cluster recovers, one or multiple disk failures because slowop appears and then affects the cluster, we can change these configurations and may reduce IOPS when recovery. osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt Vào Th 4, 10 thg 1, 2024 vào lúc 11:22 David Yang <gmydw1118@gmail.com> đã viết:
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079
I update the config osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt=6 Vào Th 6, 12 thg 1, 2024 vào lúc 15:31 Phong Tran Thanh < tranphong079@gmail.com> đã viết:
Hi Yang and Anthony,
I found the solution for this problem on a HDD disk 7200rpm
When the cluster recovers, one or multiple disk failures because slowop appears and then affects the cluster, we can change these configurations and may reduce IOPS when recovery. osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt
Vào Th 4, 10 thg 1, 2024 vào lúc 11:22 David Yang <gmydw1118@gmail.com> đã viết:
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
-- Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079
Is it better? Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ From: Phong Tran Thanh <tranphong079@gmail.com> Sent: Friday, January 12, 2024 3:32 PM To: David Yang <gmydw1118@gmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: About ceph disk slowops effect to cluster Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ I update the config osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt=6 Vào Th 6, 12 thg 1, 2024 vào lúc 15:31 Phong Tran Thanh < tranphong079@gmail.com> đã viết:
Hi Yang and Anthony,
I found the solution for this problem on a HDD disk 7200rpm
When the cluster recovers, one or multiple disk failures because slowop appears and then affects the cluster, we can change these configurations and may reduce IOPS when recovery. osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt
Vào Th 4, 10 thg 1, 2024 vào lúc 11:22 David Yang <gmydw1118@gmail.com> đã viết:
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
-- Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Yes, it's good for me, reduce recovery process from 4GB/s to 200MB/s Vào Th 6, 12 thg 1, 2024 vào lúc 15:52 Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> đã viết:
Is it better?
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
------------------------------ *From:* Phong Tran Thanh <tranphong079@gmail.com> *Sent:* Friday, January 12, 2024 3:32 PM *To:* David Yang <gmydw1118@gmail.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* [ceph-users] Re: About ceph disk slowops effect to cluster
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
I update the config osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt=6
Vào Th 6, 12 thg 1, 2024 vào lúc 15:31 Phong Tran Thanh < tranphong079@gmail.com> đã viết:
Hi Yang and Anthony,
I found the solution for this problem on a HDD disk 7200rpm
When the cluster recovers, one or multiple disk failures because slowop appears and then affects the cluster, we can change these configurations and may reduce IOPS when recovery. osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt
Vào Th 4, 10 thg 1, 2024 vào lúc 11:22 David Yang <gmydw1118@gmail.com
đã viết:
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
-- Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079
-- Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079
On Jan 12, 2024, at 03:31, Phong Tran Thanh <tranphong079@gmail.com> wrote:
Hi Yang and Anthony,
I found the solution for this problem on a HDD disk 7200rpm
When the cluster recovers, one or multiple disk failures because slowop appears and then affects the cluster, we can change these configurations and may reduce IOPS when recovery. osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt
This got cut off. What value are you using for wgt? And how are you setting these? With 17.2.5 I get [rook@rook-ceph-tools-5ff8d58445-gkl5w /]$ ceph config set osd osd_mclock_scheduler_background_recovery_res 0.2 Error EINVAL: error parsing value: strict_si_cast: unit prefix not recognized but with 17.2.6 it works. The wording isn't clear but I suspect this is a function of https://tracker.ceph.com/issues/57533
Vào Th 4, 10 thg 1, 2024 vào lúc 11:22 David Yang <gmydw1118@gmail.com> đã viết:
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
-- Trân trọng, ----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Only change it with a custom profile, no with built-in profiles, i am configuring it from ceph dashboard. osd_mclock_scheduler_client_wgt=6 -> this is my setting Vào Th 7, 13 thg 1, 2024 vào lúc 02:19 Anthony D'Atri <aad@dreamsnake.net> đã viết:
On Jan 12, 2024, at 03:31, Phong Tran Thanh <tranphong079@gmail.com> wrote:
Hi Yang and Anthony,
I found the solution for this problem on a HDD disk 7200rpm
When the cluster recovers, one or multiple disk failures because slowop appears and then affects the cluster, we can change these configurations and may reduce IOPS when recovery. osd_mclock_profile=custom osd_mclock_scheduler_background_recovery_lim=0.2 osd_mclock_scheduler_background_recovery_res=0.2 osd_mclock_scheduler_client_wgt
This got cut off. What value are you using for wgt?
And how are you setting these?
With 17.2.5 I get
[rook@rook-ceph-tools-5ff8d58445-gkl5w /]$ ceph config set osd osd_mclock_scheduler_background_recovery_res 0.2 Error EINVAL: error parsing value: strict_si_cast: unit prefix not recognized
but with 17.2.6 it works.
The wording isn't clear but I suspect this is a function of https://tracker.ceph.com/issues/57533
Vào Th 4, 10 thg 1, 2024 vào lúc 11:22 David Yang <gmydw1118@gmail.com
đã viết:
The 2*10Gbps shared network seems to be full (1.9GB/s). Is it possible to reduce part of the workload and wait for the cluster to return to a healthy state? Tip: Erasure coding needs to collect all data blocks when recovering data, so it takes up a lot of network card bandwidth and processor resources.
-- Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@gmail.com Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079
participants (4)
-
Anthony D'Atri
-
David Yang
-
Phong Tran Thanh
-
Szabo, Istvan (Agoda)