Forgot to do a reply all. What does ceph osd df ceph osd dump | grep pool return? Are you using auto scaling? 289pg with 272tb of data and 60 osds, that seems like 3-4 pg per osd at almost 1TB each. Unless I'm thinking of this wrong. On Thu, Mar 2, 2023, 17:37 Joffrey <joff.au@gmail.com> wrote:
My Ceph Version is 17.2.5 and all configuration about osd_scrub* are defaults. I tried some updates on osd-max-backfills but no change. I have many HDD with NVME for db and all are connected in a 25G network.
Yes, it's the same PG since 4 days.
I got a failure on a HDD and get many days of recovery+backfilling last 2 weeks. Perhaps the 'not in time' warning is related to this.
'Jof
Le jeu. 2 mars 2023 à 14:25, Anthony D'Atri <aad@dreamsnake.net> a écrit :
Run `ceph health detail`.
Is it the same PG backfilling for a long time, or a different one over time?
That it’s remapped makes me think that what you’re seeing is the balancer doing its job.
As far as the scrubbing, do you limit the times when scrubbing can happen? Are these HDDs? EC?
On Mar 2, 2023, at 07:20, Joffrey <joff.au@gmail.com> wrote:
Hi,
I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling and I don't understand why this backfilling is taking so long.
root@hbgt-ceph1-mon3:/# ceph -s cluster: id: c300532c-51fa-11ec-9a41-0050569c3b55 health: HEALTH_WARN 15 pgs not deep-scrubbed in time 13 pgs not scrubbed in time
services: mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 (age 36h) mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs rgw: 3 daemons active (3 hosts, 2 zones)
data: pools: 13 pools, 289 pgs objects: 67.74M objects, 127 TiB usage: 272 TiB used, 769 TiB / 1.0 PiB avail pgs: 288 active+clean 1 active+remapped+backfilling
io: client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr recovery: 790 KiB/s, 0 objects/s
What can I do to understand this slow recovery (is it the backfill action ?)
Thanks you
'Jof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io