After a huge amount of snaphot delete many snaptrim+snaptrim_wait pgs
Hi, The user deleted 20-30 snapshots and clones from the cluster and it seems like slows down the whole system. I’ve set the snaptrim parameters to the lowest as possible, set bufferred_io to true so at least have some speed for the user, but I can see the objects removal from the cluster is still happening, the beginning was 45 millioms, now 19millions but what I don’t understand many osds getting more full :( ? And the snaptrim is super slow, have 195 snaptrim wait and 36 snaptrim but in every 5 hours only 1 done :/? What can I do? One of the osd has been 62%, now it is 75% in 2 days and still growing. Set back the snap options or? The cluster has 3 servers, running on luminous 12.2.8. Some paste: https://jpst.it/2vw4H Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Yes, but before I update need to have a healthy cluster, don't really want to update if it is not healthy to carry issue over. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Sunday, May 16, 2021 4:59 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Ceph Users <ceph-users@ceph.io> Subject: Re: [ceph-users] After a huge amount of snaphot delete many snaptrim+snaptrim_wait pgs Hi, On 16 May 2021, at 04:22, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: The cluster has 3 servers, running on luminous 12.2.8. Again, this is old and unsupported version of Ceph. Please, upgrade at least to 12.2.13 k ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
participants (2)
-
Konstantin Shalygin
-
Szabo, Istvan (Agoda)