Hello Joffrey, I am not sure why my previous reply did not go through. I am replying on this thread again. The slow backfills could also be due to the mclock_scheduler throttling the recovery operations aggressively and this issue is currently being fixed. To confirm if mclock_scheduler is causing this, can you please run a couple of commands and post the output on this thread? 1. ceph versions 2. ceph config show osd.<id> | grep osd_max_backfills 3. ceph config show osd.<id> | grep osd_recovery_max_active 4. ceph config show osd.<id> | grep osd_mclock With mclock_scheduler enabled, and with 17.2.5, it is not possible to override the recovery options like osd_max_backfills, osd_recovery_max_active and other related options. Based on the result of the above commands, I can confirm if mclock_scheduler is causing the issue and suggest next steps. -Sridhar On Fri, Mar 3, 2023 at 9:35 PM Joffrey <joff.au@gmail.com> wrote:
Ok, Thanks. You mean that the autoscale feature is... stupid ? I'm going to change pgp_num and use the legacy formula OSDs * 100 / pool size.
Le jeu. 2 mars 2023 à 17:04, Curt <lightspd@gmail.com> a écrit :
I see autoscale_mode on all pools and I'm guessing this is your largest pool bkp365-ncy.rgw.buckets.data, with 32 pg. I would definitely turn off autoscale and increase pg_num/pgp_num. Someone with more experience than I can chime in, but I would think something like 2048 would be much better.
On Thu, Mar 2, 2023 at 6:12 PM Joffrey <joff.au@gmail.com> wrote:
root@hbgt-ceph1-mon3:/# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 11 KiB 23 GiB 11 TiB 36.17 1.39 17 up 3 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 3.7 GiB 17 GiB 12 TiB 28.47 1.09 11 up 5 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 3.2 GiB 12 GiB 14 TiB 20.89 0.80 13 up 7 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 3.2 GiB 6.9 GiB 15 TiB 13.32 0.51 19 up 9 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 68 MiB 18 GiB 12 TiB 28.53 1.09 18 up 11 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 403 MiB 23 GiB 11 TiB 36.13 1.38 17 up 13 hdd 17.34140 1.00000 17 TiB 1001 GiB 7.1 GiB 9.9 MiB 1.1 GiB 16 TiB 5.64 0.22 18 up 15 hdd 17.34140 1.00000 17 TiB 8.9 TiB 7.9 TiB 842 KiB 34 GiB 8.4 TiB 51.41 1.97 18 up 17 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 24 KiB 12 GiB 14 TiB 20.90 0.80 17 up 19 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 4.1 GiB 6.2 GiB 15 TiB 13.31 0.51 18 up 21 hdd 17.34140 1.00000 17 TiB 5.0 TiB 4.0 TiB 206 MiB 17 GiB 12 TiB 28.55 1.09 23 up 23 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 4.2 GiB 17 GiB 12 TiB 28.54 1.09 14 up 0 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 7.2 GiB 12 GiB 14 TiB 20.94 0.80 18 up 2 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 18 KiB 12 GiB 14 TiB 20.93 0.80 13 up 4 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 3.0 GiB 12 GiB 14 TiB 20.95 0.80 20 up 6 hdd 17.34140 1.00000 17 TiB 8.9 TiB 7.9 TiB 4.4 MiB 34 GiB 8.4 TiB 51.36 1.97 17 up 8 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 965 KiB 6.5 GiB 15 TiB 13.26 0.51 14 up 10 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 18 KiB 6.5 GiB 15 TiB 13.25 0.51 13 up 12 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 98 MiB 17 GiB 12 TiB 28.49 1.09 16 up 14 hdd 17.34140 1.00000 17 TiB 5.0 TiB 4.0 TiB 4.2 GiB 17 GiB 12 TiB 28.55 1.09 20 up 16 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 24 KiB 12 GiB 14 TiB 20.94 0.80 20 up 18 hdd 17.34140 1.00000 17 TiB 8.9 TiB 7.9 TiB 17 MiB 34 GiB 8.4 TiB 51.42 1.97 19 up 20 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 3.2 GiB 17 GiB 12 TiB 28.50 1.09 18 up 22 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 2.7 GiB 6.2 GiB 15 TiB 13.25 0.51 11 up 24 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 70 MiB 17 GiB 12 TiB 28.50 1.09 18 up 25 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 3.0 GiB 17 GiB 12 TiB 28.51 1.09 16 up 26 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 3.0 GiB 23 GiB 11 TiB 36.13 1.38 15 up 27 hdd 17.34140 1.00000 17 TiB 5.0 TiB 4.0 TiB 205 MiB 17 GiB 12 TiB 28.59 1.10 16 up 28 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 1.0 MiB 6.3 GiB 15 TiB 13.27 0.51 12 up 29 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 1.3 MiB 17 GiB 12 TiB 28.50 1.09 4 up 30 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 379 KiB 23 GiB 11 TiB 36.14 1.38 16 up 31 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 2.5 MiB 12 GiB 14 TiB 20.92 0.80 19 up 32 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 11 MiB 12 GiB 14 TiB 20.93 0.80 16 up 33 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 18 KiB 12 GiB 14 TiB 20.91 0.80 17 up 34 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 71 MiB 23 GiB 11 TiB 36.15 1.38 19 up 35 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 3.3 GiB 6.3 GiB 15 TiB 13.28 0.51 14 up 36 hdd 17.34140 1.00000 17 TiB 5.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 28.59 1.09 13 up 37 hdd 17.34140 1.00000 17 TiB 4.9 TiB 4.0 TiB 69 MiB 17 GiB 12 TiB 28.54 1.09 12 up 38 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 2.9 GiB 6.7 GiB 15 TiB 13.26 0.51 22 up 39 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 205 MiB 23 GiB 11 TiB 36.19 1.39 25 up 40 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 9 KiB 12 GiB 14 TiB 20.88 0.80 14 up 41 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 8.2 GiB 23 GiB 11 TiB 36.11 1.38 20 up 42 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 55 KiB 12 GiB 14 TiB 20.91 0.80 16 up 43 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 70 MiB 23 GiB 11 TiB 36.17 1.39 21 up 44 hdd 17.34140 1.00000 17 TiB 7.6 TiB 6.6 TiB 18 KiB 28 GiB 9.8 TiB 43.75 1.68 16 up 45 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 141 MiB 6.5 GiB 15 TiB 13.29 0.51 17 up 46 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 1.7 MiB 6.4 GiB 15 TiB 13.27 0.51 15 up 47 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 3.5 GiB 11 GiB 14 TiB 20.89 0.80 22 up 48 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 9 KiB 6.3 GiB 15 TiB 13.25 0.51 10 up 49 hdd 17.34140 1.00000 17 TiB 8.9 TiB 7.9 TiB 4 KiB 33 GiB 8.4 TiB 51.41 1.97 18 up 50 hdd 17.34140 1.00000 17 TiB 7.6 TiB 6.6 TiB 212 MiB 31 GiB 9.7 TiB 43.81 1.68 20 up 51 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.6 TiB 85 MiB 13 GiB 14 TiB 20.87 0.80 19 up 52 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 5.4 GiB 6.0 GiB 15 TiB 13.34 0.51 18 up 53 hdd 17.34140 1.00000 17 TiB 5.0 TiB 4.0 TiB 25 MiB 19 GiB 12 TiB 28.55 1.09 16 up 54 hdd 17.34140 1.00000 17 TiB 6.2 TiB 5.3 TiB 198 MiB 23 GiB 11 TiB 35.99 1.38 14 up 55 hdd 17.34140 1.00000 17 TiB 5.0 TiB 4.0 TiB 10 GiB 18 GiB 12 TiB 28.59 1.09 26 up 56 hdd 17.34140 1.00000 17 TiB 6.3 TiB 5.3 TiB 153 MiB 24 GiB 11 TiB 36.14 1.38 22 up 57 hdd 17.34140 1.00000 17 TiB 3.6 TiB 2.7 TiB 58 KiB 12 GiB 14 TiB 20.91 0.80 13 up 58 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 3.3 GiB 6.4 GiB 15 TiB 13.23 0.51 11 up 59 hdd 17.34140 1.00000 17 TiB 2.3 TiB 1.3 TiB 19 KiB 6.3 GiB 15 TiB 13.27 0.51 11 up TOTAL 1.0 PiB 272 TiB 213 TiB 84 GiB 942 GiB 769 TiB 26.11
root@hbgt-ceph1-mon3:/# ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 15503 lfor 0/8533/8531 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8321 lfor 0/8321/8319 flags hashpspool stripe_width 0 application rgw pool 3 'bkp365-ncy.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8297 lfor 0/8297/8295 flags hashpspool stripe_width 0 application rgw pool 4 'bkp365-ncy.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8054 lfor 0/8054/8052 flags hashpspool stripe_width 0 application rgw pool 5 'bkp365-ncy.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 3412 lfor 0/3412/3410 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 6 'bkp365-ncy.rgw.buckets.data' erasure profile EC32 size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3500 lfor 0/0/2720 flags hashpspool stripe_width 12288 application rgw pool 7 'bkp365-ncy.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 3436 lfor 0/3436/3434 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 9 'ncy.rgw.buckets.data' erasure profile EC32 size 5 min_size 4 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 14975 lfor 0/0/14973 flags hashpspool stripe_width 12288 application rgw pool 10 'ncy.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 14979 flags hashpspool stripe_width 0 application rgw pool 11 'ncy.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 14981 flags hashpspool stripe_width 0 application rgw pool 12 'ncy.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15105 lfor 0/15105/15103 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 13 'ncy.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15236 lfor 0/15236/15234 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 14 'ncy.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 15241 flags hashpspool stripe_width 0 application rgw
(EC32 is a erasure coding with 3 datas and 2 codes)
No output with "ceph osd pool autoscale-status"
Le jeu. 2 mars 2023 à 15:02, Curt <lightspd@gmail.com> a écrit :
Forgot to do a reply all.
What does
ceph osd df ceph osd dump | grep pool return?
Are you using auto scaling? 289pg with 272tb of data and 60 osds, that seems like 3-4 pg per osd at almost 1TB each. Unless I'm thinking of this wrong.
On Thu, Mar 2, 2023, 17:37 Joffrey <joff.au@gmail.com> wrote:
My Ceph Version is 17.2.5 and all configuration about osd_scrub* are defaults. I tried some updates on osd-max-backfills but no change. I have many HDD with NVME for db and all are connected in a 25G network.
Yes, it's the same PG since 4 days.
I got a failure on a HDD and get many days of recovery+backfilling last 2 weeks. Perhaps the 'not in time' warning is related to this.
'Jof
Le jeu. 2 mars 2023 à 14:25, Anthony D'Atri <aad@dreamsnake.net> a écrit :
Run `ceph health detail`.
Is it the same PG backfilling for a long time, or a different one over time?
That it’s remapped makes me think that what you’re seeing is the balancer doing its job.
As far as the scrubbing, do you limit the times when scrubbing can happen? Are these HDDs? EC?
> On Mar 2, 2023, at 07:20, Joffrey <joff.au@gmail.com> wrote: > > Hi, > > I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling > and I don't understand why this backfilling is taking so long. > > root@hbgt-ceph1-mon3:/# ceph -s > cluster: > id: c300532c-51fa-11ec-9a41-0050569c3b55 > health: HEALTH_WARN > 15 pgs not deep-scrubbed in time > 13 pgs not scrubbed in time > > services: > mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 > (age 36h) > mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: > hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm > osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs > rgw: 3 daemons active (3 hosts, 2 zones) > > data: > pools: 13 pools, 289 pgs > objects: 67.74M objects, 127 TiB > usage: 272 TiB used, 769 TiB / 1.0 PiB avail > pgs: 288 active+clean > 1 active+remapped+backfilling > > io: > client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr > recovery: 790 KiB/s, 0 objects/s > > > What can I do to understand this slow recovery (is it the backfill action ?) > > Thanks you > > 'Jof > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sridhar Seshasayee Partner Engineer Red Hat <https://www.redhat.com> <https://www.redhat.com>