(adding back the list) On Tue, Mar 21, 2023 at 11:25 AM Joachim Kraftmayer < joachim.kraftmayer@clyso.com> wrote:
i added the questions and answers below.
___________________________________ Best Regards, Joachim Kraftmayer CEO | Clyso GmbH
Clyso GmbH p: +49 89 21 55 23 91 2 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: joachim.kraftmayer@clyso.com
We are hiring: https://www.clyso.com/jobs/ --- CEO: Dipl. Inf. (FH) Joachim Kraftmayer Unternehmenssitz: Utting am Ammersee Handelsregister beim Amtsgericht: Augsburg Handelsregister-Nummer: HRB 25866 USt. ID-Nr.: DE275430677
Am 21.03.23 um 11:14 schrieb Gauvain Pocentek:
Hi Joachim,
On Tue, Mar 21, 2023 at 10:13 AM Joachim Kraftmayer < joachim.kraftmayer@clyso.com> wrote:
Which Ceph version are you running, is mclock active?
We're using Quincy (17.2.5), upgraded step by step from Luminous if I remember correctly.
did you recreate the osds? if yes, at which version?
I actually don't remember all the history, but I think we added the HDD nodes while running Pacific.
mlock seems active, set to high_client_ops profile. HDD OSDs have very different settings for max capacity iops:
osd.137 basic osd_mclock_max_capacity_iops_hdd 929.763899 osd.161 basic osd_mclock_max_capacity_iops_hdd 4754.250946 osd.222 basic osd_mclock_max_capacity_iops_hdd 540.016984 osd.281 basic osd_mclock_max_capacity_iops_hdd 1029.193945 osd.282 basic osd_mclock_max_capacity_iops_hdd 1061.762870 osd.283 basic osd_mclock_max_capacity_iops_hdd 462.984562
We haven't set those explicitly, could they be the reason of the slow recovery?
i recommend to disable mclock for now, and yes we have seen slow recovery caused by mclock.
Stupid question: how do you do that? I've looked through the docs but could only find information about changing the settings.
Bonus question: does ceph set that itself?
yes and if you have a setup with HDD + SSD (db & wal) the discovery works not in the right way.
Good to know! Gauvain
Thanks!
Gauvain
Joachim
___________________________________ Clyso GmbH - Ceph Foundation Member
Am 21.03.23 um 06:53 schrieb Gauvain Pocentek:
Hello all,
We have an EC (4+2) pool for RGW data, with HDDs + SSDs for WAL/DB. This pool has 9 servers with each 12 disks of 16TBs. About 10 days ago we lost a server and we've removed its OSDs from the cluster. Ceph has started to remap and backfill as expected, but the process has been getting slower and slower. Today the recovery rate is around 12 MiB/s and 10 objects/s. All the remaining unclean PGs are backfilling:
data: volumes: 1/1 healthy pools: 14 pools, 14497 pgs objects: 192.38M objects, 380 TiB usage: 764 TiB used, 1.3 PiB / 2.1 PiB avail pgs: 771559/1065561630 objects degraded (0.072%) 1215899/1065561630 objects misplaced (0.114%) 14428 active+clean 50 active+undersized+degraded+remapped+backfilling 18 active+remapped+backfilling 1 active+clean+scrubbing+deep
We've checked the health of the remaining servers, and everything looks like (CPU/RAM/network/disks).
Any hints on what could be happening?
Thank you, Gauvain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io