Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ``` I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ``` I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ``` 1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again? Thank you! Bruno
Hello Bruno, Interesting case, few observations. What’s the average size of your PGs? Judging from the ceph status you have 1394 pls in total and 696TiB of used storage, that’s roughly 500GB per pg if I’m not mistaken. With the backfilling limits this results in a lot of time spent per single pg due to its size. You could try increasing their number in the pools to have lighter placement groups. Are you using mclock? If yes, you can try setting the profile to prioritise recovery operations with 'ceph config set osd osd_mclock_profile high_recovery_ops' The max backfills configuration is an interesting one - it should persist. What happens if you set it through the Ceph UI? In general it looks like the balancer might be “fighting” with the manual OSD balancing. You could try turning it off and do the balancing yourself (this might be helpful: https://github.com/laimis9133/plankton-swarm). Also probably known already but keep in mind erasure coded pools are known to be on the slower side when it comes to any data movement due to additional operations needed. Best, Laimis J.
On 4 Jan 2025, at 13:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
One more question: What’s the output of 'ceph config get osd osd_max_backfills’ after setting osd_max_backfills? Looks like ceph-config might be showing the wrong configurations. Best, Laimis J.
On 4 Jan 2025, at 18:05, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
Hello Bruno,
Interesting case, few observations.
What’s the average size of your PGs? Judging from the ceph status you have 1394 pls in total and 696TiB of used storage, that’s roughly 500GB per pg if I’m not mistaken. With the backfilling limits this results in a lot of time spent per single pg due to its size. You could try increasing their number in the pools to have lighter placement groups.
Are you using mclock? If yes, you can try setting the profile to prioritise recovery operations with 'ceph config set osd osd_mclock_profile high_recovery_ops'
The max backfills configuration is an interesting one - it should persist. What happens if you set it through the Ceph UI?
In general it looks like the balancer might be “fighting” with the manual OSD balancing. You could try turning it off and do the balancing yourself (this might be helpful: https://github.com/laimis9133/plankton-swarm).
Also probably known already but keep in mind erasure coded pools are known to be on the slower side when it comes to any data movement due to additional operations needed.
Best, Laimis J.
On 4 Jan 2025, at 13:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry for the mail spam, but last question: What reweighs have been set for the top OSDs (ceph osd df tree)? Just a guess but they might have been a bit too aggressive and caused a lot of backfilling operations. Best, Laimis J.
On 4 Jan 2025, at 18:05, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
Hello Bruno,
Interesting case, few observations.
What’s the average size of your PGs? Judging from the ceph status you have 1394 pls in total and 696TiB of used storage, that’s roughly 500GB per pg if I’m not mistaken. With the backfilling limits this results in a lot of time spent per single pg due to its size. You could try increasing their number in the pools to have lighter placement groups.
Are you using mclock? If yes, you can try setting the profile to prioritise recovery operations with 'ceph config set osd osd_mclock_profile high_recovery_ops'
The max backfills configuration is an interesting one - it should persist. What happens if you set it through the Ceph UI?
In general it looks like the balancer might be “fighting” with the manual OSD balancing. You could try turning it off and do the balancing yourself (this might be helpful: https://github.com/laimis9133/plankton-swarm).
Also probably known already but keep in mind erasure coded pools are known to be on the slower side when it comes to any data movement due to additional operations needed.
Best, Laimis J.
On 4 Jan 2025, at 13:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, your cephfs.cephfs01.data pool currently has 144 PGs. So this pool seems to be resizing, e.g. from 128 PGs to 256 PGs. Do you use the autoscalar or did you trigger a manual PG increment of the pool? You can check this with the output of "ceph osd pool ls detail". It shows the current and target number of PGs and PGPs for all pools. Nonetheless changing the number of PGs in a pool will always result in data movement, and this will temporarily use more data. You can ignore the not-scrubbed-in-time warnings for the moment, the PG will be scrubbed again after the pool resize is finished. You should have an eye on the free space of the OSDs (e.g. ceph osd df tree). There are already some OSDs above certain thresholds, and these will be blocking further backfilling. It is still running for 5 PGs, but it might take a lot of time at that rate. You have two options: be patient and let the cluster settle by itself, or try to speed up the backfilling. I recommend waiting and monitoring the cluster (and especially the OSD free space). If you like to live dangerous and also have a good and reliable backup of all your data, you can try two approaches to speed up backfilling: 1. What Laimis wrote in the other mails. It essentially boils down to giving backfilling a higher priority than client I/O. One addition: the osd_max_backfills setting is ignored if the cluster is using the mclock scheduler (afaik standard in ceph reef). It might be worth to change to the wpq scheduler to have a better control over backfilling, the mclock scheduler is too complicated when it comes to this matter. 2. The first approach might speed up I/O, but it does not solve the problem of OSDs blocking backfilling due to low space. This is controlled by the mon_osd_backfillfull_ratio setting, which defaults to 90%. In your case this means that 1.5 TB is still available, but backfilling is blocked. Raising the threshold might enable more OSDs to start backfilling, which will speed up the overall process. Given the size of your PGs (500GB according to Laimis), this might be a dangerous operation. I'm not sure whether a running backfilling of an individual PG is interrupted if the threshold is reached, or whether it only controls starting _new_ backfills. So if you want to change the threshold, I would propose the following steps: - set osd_max_backfills to 1 - change to wpq scheduler for all OSDs (requires OSD restart) - check the state of the cluster, free space etc. - increase mon_osd_backfillfull_ratio in a very small step (e.g. 90% -> 91%) - check the cluster state, more PGs should be backfilling now Before you start this you might want to check how many OSDs are already over the threshold and whether backfilling is moving data to or from them (check "ceph pg dump", it lists which PG is current backfilling from which set of OSDs to which other set). I assume that the data pool wants to have 256 PGs (ceph prefers powers of two), so there will be a lot more data movement. Unless your users are producing a lot of new data, it should be safe to leave the cluster at its current state and allow it to settle. On the other hand it seems to have been in this state for several weeks now, so a little push might be necessary. Best regards, Burkhard On 04.01.25 12:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Interesting. You're right: # ceph config get osd osd_max_backfills 10 ## ceph-conf --show-config | egrep osd_max_backfills osd_max_backfills = 1 I don't know why that is happening. On Sat, 4 Jan 2025 at 17:13, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
One more question: What’s the output of 'ceph config get osd osd_max_backfills’ after setting osd_max_backfills? Looks like ceph-config might be showing the wrong configurations.
Best, Laimis J.
On 4 Jan 2025, at 18:05, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
Hello Bruno,
Interesting case, few observations.
What’s the average size of your PGs? Judging from the ceph status you have 1394 pls in total and 696TiB of used storage, that’s roughly 500GB per pg if I’m not mistaken. With the backfilling limits this results in a lot of time spent per single pg due to its size. You could try increasing their number in the pools to have lighter placement groups.
Are you using mclock? If yes, you can try setting the profile to prioritise recovery operations with 'ceph config set osd osd_mclock_profile high_recovery_ops'
The max backfills configuration is an interesting one - it should persist. What happens if you set it through the Ceph UI?
In general it looks like the balancer might be “fighting” with the manual OSD balancing. You could try turning it off and do the balancing yourself (this might be helpful: https://github.com/laimis9133/plankton-swarm).
Also probably known already but keep in mind erasure coded pools are known to be on the slower side when it comes to any data movement due to additional operations needed.
Best, *Laimis J.*
On 4 Jan 2025, at 13:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha
Are you using mclock?
I'm not sure. I'll read about it. What happens if you set it through the Ceph UI? There are *global*,* mon*,* mgr*,* osd*,* mds *and* client* values that I changed to 10 (the default is 1): [image: image.png] ...but the output is still: # ceph-conf --show-config | egrep osd_max_backfills osd_max_backfills = 1 You could try turning it off and do the balancing yourself (this might be
helpful: https://github.com/laimis9133/plankton-swarm).
I'll definitely look into that. Thanks a bunch! On Sat, 4 Jan 2025 at 17:06, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
Hello Bruno,
Interesting case, few observations.
What’s the average size of your PGs? Judging from the ceph status you have 1394 pls in total and 696TiB of used storage, that’s roughly 500GB per pg if I’m not mistaken. With the backfilling limits this results in a lot of time spent per single pg due to its size. You could try increasing their number in the pools to have lighter placement groups.
Are you using mclock? If yes, you can try setting the profile to prioritise recovery operations with 'ceph config set osd osd_mclock_profile high_recovery_ops'
The max backfills configuration is an interesting one - it should persist. What happens if you set it through the Ceph UI?
In general it looks like the balancer might be “fighting” with the manual OSD balancing. You could try turning it off and do the balancing yourself (this might be helpful: https://github.com/laimis9133/plankton-swarm).
Also probably known already but keep in mind erasure coded pools are known to be on the slower side when it comes to any data movement due to additional operations needed.
Best, *Laimis J.*
On 4 Jan 2025, at 13:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha
What reweighs have been set for the top OSDs (ceph osd df tree)?
Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening? On Sat, 4 Jan 2025 at 17:18, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
Sorry for the mail spam, but last question: What reweighs have been set for the top OSDs (ceph osd df tree)? Just a guess but they might have been a bit too aggressive and caused a lot of backfilling operations.
Best, *Laimis J.*
On 4 Jan 2025, at 18:05, Laimis Juzeliūnas <laimis.juzeliunas@oxylabs.io> wrote:
Hello Bruno,
Interesting case, few observations.
What’s the average size of your PGs? Judging from the ceph status you have 1394 pls in total and 696TiB of used storage, that’s roughly 500GB per pg if I’m not mistaken. With the backfilling limits this results in a lot of time spent per single pg due to its size. You could try increasing their number in the pools to have lighter placement groups.
Are you using mclock? If yes, you can try setting the profile to prioritise recovery operations with 'ceph config set osd osd_mclock_profile high_recovery_ops'
The max backfills configuration is an interesting one - it should persist. What happens if you set it through the Ceph UI?
In general it looks like the balancer might be “fighting” with the manual OSD balancing. You could try turning it off and do the balancing yourself (this might be helpful: https://github.com/laimis9133/plankton-swarm).
Also probably known already but keep in mind erasure coded pools are known to be on the slower side when it comes to any data movement due to additional operations needed.
Best, *Laimis J.*
On 4 Jan 2025, at 13:18, bruno.pessanha@gmail.com wrote:
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha
Do you use the autoscalar or did you trigger a manual PG increment of the pool?
The pool had autoscale enabled until 2 days ago when I thought it was better to change things manually in order to have a more deterministic result. Yes, I wanted to increase from "1" to something like "1024" but it looks like it was capped to the 144 no matter what I do: # ceph osd pool get cephfs.cephfs01.data pg_num pg_num: 144 # ceph osd pool set cephfs.cephfs01.data pg_num 1024 # ceph osd pool get cephfs.cephfs01.data pg_num pg_num: 144 You can check this with the output of "ceph osd pool ls detail". It shows
the current and target number of PGs and PGPs for all pools.
That's a very useful command! Thanks. # ceph osd pool ls detail |grep cephfs.cephfs01.data pool 12 'cephfs.cephfs01.data' erasure profile 8k2m size 10 min_size 9 crush_rule 1 object_hash rjenkins pg_num 144 pgp_num 16 pg_num_target 1024 pgp_num_target 1024 autoscale_mode warn last_change 17398 lfor 0/0/16936 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 32768 pg_num_max 1024 application cephfs You can ignore the not-scrubbed-in-time warnings for the moment, the PG
will be scrubbed again after the pool resize is finished.
Will do. Your suggestion #2 makes total sense to me. I'll fine tune mon_osd_backfillfull_ratio, leave the backfills running a few more days and keep monitoring. My only concern is if users start to store a lot more data all of a sudden. I'll keep an eye on it. Is it correct to say that every PG/OSD change can potentially cause data misplacements, unbalanced osd's and long backfills? I'll be way more careful before tuning it if that's the case. Thank you both so much! It definitely helped me to understand Ceph better. It is kind of a steep curve :). On Sat, 4 Jan 2025 at 19:03, Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
your cephfs.cephfs01.data pool currently has 144 PGs. So this pool seems to be resizing, e.g. from 128 PGs to 256 PGs. Do you use the autoscalar or did you trigger a manual PG increment of the pool?
You can check this with the output of "ceph osd pool ls detail". It shows the current and target number of PGs and PGPs for all pools.
Nonetheless changing the number of PGs in a pool will always result in data movement, and this will temporarily use more data. You can ignore the not-scrubbed-in-time warnings for the moment, the PG will be scrubbed again after the pool resize is finished. You should have an eye on the free space of the OSDs (e.g. ceph osd df tree). There are already some OSDs above certain thresholds, and these will be blocking further backfilling. It is still running for 5 PGs, but it might take a lot of time at that rate.
You have two options: be patient and let the cluster settle by itself, or try to speed up the backfilling. I recommend waiting and monitoring the cluster (and especially the OSD free space). If you like to live dangerous and also have a good and reliable backup of all your data, you can try two approaches to speed up backfilling:
1. What Laimis wrote in the other mails. It essentially boils down to giving backfilling a higher priority than client I/O. One addition: the osd_max_backfills setting is ignored if the cluster is using the mclock scheduler (afaik standard in ceph reef). It might be worth to change to the wpq scheduler to have a better control over backfilling, the mclock scheduler is too complicated when it comes to this matter.
2. The first approach might speed up I/O, but it does not solve the problem of OSDs blocking backfilling due to low space. This is controlled by the mon_osd_backfillfull_ratio setting, which defaults to 90%. In your case this means that 1.5 TB is still available, but backfilling is blocked. Raising the threshold might enable more OSDs to start backfilling, which will speed up the overall process. Given the size of your PGs (500GB according to Laimis), this might be a dangerous operation. I'm not sure whether a running backfilling of an individual PG is interrupted if the threshold is reached, or whether it only controls starting _new_ backfills. So if you want to change the threshold, I would propose the following steps:
- set osd_max_backfills to 1
- change to wpq scheduler for all OSDs (requires OSD restart)
- check the state of the cluster, free space etc.
- increase mon_osd_backfillfull_ratio in a very small step (e.g. 90% -> 91%)
- check the cluster state, more PGs should be backfilling now
Before you start this you might want to check how many OSDs are already over the threshold and whether backfilling is moving data to or from them (check "ceph pg dump", it lists which PG is current backfilling from which set of OSDs to which other set).
I assume that the data pool wants to have 256 PGs (ceph prefers powers of two), so there will be a lot more data movement. Unless your users are producing a lot of new data, it should be safe to leave the cluster at its current state and allow it to settle. On the other hand it seems to have been in this state for several weeks now, so a little push might be necessary.
Best regards,
Burkhard
Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s
cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time
services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling
io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s
progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ```
I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ```
I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ```
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher
On 04.01.25 12:18, bruno.pessanha@gmail.com wrote: than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually?
3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again?
Thank you!
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha
Do you use the autoscalar or did you trigger a manual PG increment of the pool?
The pool had autoscale enabled until 2 days ago when I thought it was better to change things manually in order to have a more deterministic result. Yes, I wanted to increase from "1" to something like "1024" but it looks like it was capped to the 144 no matter what I do:
`ceph osd df`, look at the PGS column. Could be you’re hitting the limit on some OSDs. It’s odd to stop at a non power of 2.
Is it correct to say that every PG/OSD change can potentially cause data misplacements, unbalanced osd's and long backfills? I'll be way more careful before tuning it if that's the case.
The autoscaler will usually only bump pg_num for a pool when the value is <half what it thinks it should be. I suggest setting the ‘bulk’ flag one pool at a time to effectively pre-split PGs as if the pool were full of data already. Ignore .mgr. Start with the pools with the fewest , let the cluster settle between each adjustment. That way the autoscaler will only make changes if cluster topology changes.
Thank you both so much! It definitely helped me to understand Ceph better. It is kind of a steep curve :).
We’re a community! That curve is way gentler than it used to be.
What reweighs have been set for the top OSDs (ceph osd df tree)?
Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening?
Old-style legacy override reweights don’t mesh well with the balancer. Best to leave them at 1.00. 0.2 is pretty extreme, back in the day I rarely went below 0.8.
``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again late ```
That should clear. The balancer doesn’t want to stir up trouble if the cluster already has a bunch of backfill / recovery going on. Patience!
default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB
That’s odd that the data pool is that full but the others aren’t. Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool`
I also tried changing the following but it does not seem to persist:
Could be an mclock thing.
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc.
Probably a result of the autoscaler splitting PGs or of some change to CRUSH rules such that some data can’t be placed.
2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually?
I wouldn’t, that’s a symptom not the disease.
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Very solid advice here - that’s the beauty of Ceph community. Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and back) is quite extreme and the cluster won’t like it. We never go above increments/decrements of 0.02-0.04. If you have to go from 1 to 0.98 and then to 0.96 and so on leaving enough time for the cluster to settle in between. How did backfilling look when changing? Did you see a decrease in backfills after reverting back to 1? From the clients perspective Your main concern now is to keep the pools “alive” with enough space while the backfilling takes place. Even with plenty of OSDs that are not filled you might hit a single overfilled OSD and the whole pool will stop accepting new data. Clients will start getting “No more space available” errors. That happened to us with CephFS recently with a very similar scenario where the cluster got much more data than expected in a short amount of time, not fun. With the balancer not working due to too many misplaced objects that’s an increased risk so just heads up and keep that in mind. To get things working we simply balanced manually the OSDs with upmaps moving data from the most full ones to the least full ones (our builtin balancer sadly does not work). One small observation: I’ve noticed that 'ceph osd pool ls detail |grep cephfs.cephfs01.data’ has pg_num increased but the pgp_num is still the same. You will need to set it as well for data migration to new pgs to happen: https://docs.ceph.com/en/mimic/rados/operations/placement-groups/#set-the-nu... Best, Laimis J.
On 5 Jan 2025, at 16:11, Anthony D'Atri <anthony.datri@gmail.com> wrote:
What reweighs have been set for the top OSDs (ceph osd df tree)?
Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening?
Old-style legacy override reweights don’t mesh well with the balancer. Best to leave them at 1.00.
0.2 is pretty extreme, back in the day I rarely went below 0.8.
``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again late ```
That should clear. The balancer doesn’t want to stir up trouble if the cluster already has a bunch of backfill / recovery going on. Patience!
default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB
That’s odd that the data pool is that full but the others aren’t.
Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool`
I also tried changing the following but it does not seem to persist:
Could be an mclock thing.
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc.
Probably a result of the autoscaler splitting PGs or of some change to CRUSH rules such that some data can’t be placed.
2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually?
I wouldn’t, that’s a symptom not the disease.
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Very solid advice here - that’s the beauty of Ceph community.
Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and back) is quite extreme and the cluster won’t like it.
And these days with the balancer, pg-upmap entries to the same effect are a better idea.
From the clients perspective Your main concern now is to keep the pools “alive” with enough space while the backfilling takes place.
To that end, you can *temporarily* give yourself a bit more margin: ceph osd set-nearfull-ratio .85 ceph osd set-backfillfull-ratio .90 ceph osd set-full-ratio .95 Those are the default values, and Ceph (now) enforces that the values are >= (or maybe >) in that order. So you might set the full ratio to .98, backfillfull to .96. Nearfull is only cosmetic. But absolutely do not forget to revert to default values once the cluster is balanced, or to other values that you make an educated decision to choose.
Even with plenty of OSDs that are not filled you might hit a single overfilled OSD and the whole pool will stop accepting new data.
Yep, see above. Not immediately clear to me why that data pool is so full unless the CRUSH rule / device classes are wonky.
Clients will start getting “No more space available” errors. That happened to us with CephFS recently with a very similar scenario where the cluster got much more data than expected in a short amount of time, not fun. With the balancer not working due to too many misplaced objects that’s an increased risk so just heads up and keep that in mind. To get things working we simply balanced manually the OSDs with upmaps moving data from the most full ones to the least full ones (our builtin balancer sadly does not work).
One small observation: I’ve noticed that 'ceph osd pool ls detail |grep cephfs.cephfs01.data’ has pg_num increased but the pgp_num is still the same. You will need to set it as well for data migration to new pgs to happen: https://docs.ceph.com/en/mimic/rados/operations/placement-groups/#set-the-nu...
The mgr usually does that for recent Ceph releases. With older releases we had to incremental pg_num and pgp_num in lockstep, which was kind of a pain.
Best, Laimis J.
On 5 Jan 2025, at 16:11, Anthony D'Atri <anthony.datri@gmail.com> wrote:
What reweighs have been set for the top OSDs (ceph osd df tree)?
Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening?
Old-style legacy override reweights don’t mesh well with the balancer. Best to leave them at 1.00.
0.2 is pretty extreme, back in the day I rarely went below 0.8.
``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again late ```
That should clear. The balancer doesn’t want to stir up trouble if the cluster already has a bunch of backfill / recovery going on. Patience!
default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB
That’s odd that the data pool is that full but the others aren’t.
Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool`
I also tried changing the following but it does not seem to persist:
Could be an mclock thing.
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc.
Probably a result of the autoscaler splitting PGs or of some change to CRUSH rules such that some data can’t be placed.
2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually?
I wouldn’t, that’s a symptom not the disease.
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So you might set the full ratio to .98, backfillfull to .96. Nearfull is only cosmetic.
Thanks for the advice. It seems to be working with 0.92 for now. If it gets stuck I'll increase it. On Mon, 6 Jan 2025 at 00:24, Anthony D'Atri <anthony.datri@gmail.com> wrote:
Very solid advice here - that’s the beauty of Ceph community.
Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and back) is quite extreme and the cluster won’t like it.
And these days with the balancer, pg-upmap entries to the same effect are a better idea.
From the clients perspective Your main concern now is to keep the pools “alive” with enough space while the backfilling takes place.
To that end, you can *temporarily* give yourself a bit more margin:
ceph osd set-nearfull-ratio .85 ceph osd set-backfillfull-ratio .90 ceph osd set-full-ratio .95
Those are the default values, and Ceph (now) enforces that the values are
= (or maybe >) in that order.
So you might set the full ratio to .98, backfillfull to .96. Nearfull is only cosmetic.
But absolutely do not forget to revert to default values once the cluster is balanced, or to other values that you make an educated decision to choose.
Even with plenty of OSDs that are not filled you might hit a single overfilled OSD and the whole pool will stop accepting new data.
Yep, see above. Not immediately clear to me why that data pool is so full unless the CRUSH rule / device classes are wonky.
Clients will start getting “No more space available” errors. That happened to us with CephFS recently with a very similar scenario where the cluster got much more data than expected in a short amount of time, not fun. With the balancer not working due to too many misplaced objects that’s an increased risk so just heads up and keep that in mind. To get things working we simply balanced manually the OSDs with upmaps moving data from the most full ones to the least full ones (our builtin balancer sadly does not work).
One small observation: I’ve noticed that 'ceph osd pool ls detail |grep cephfs.cephfs01.data’ has pg_num increased but the pgp_num is still the same. You will need to set it as well for data migration to new pgs to happen: https://docs.ceph.com/en/mimic/rados/operations/placement-groups/#set-the-nu...
The mgr usually does that for recent Ceph releases. With older releases we had to incremental pg_num and pgp_num in lockstep, which was kind of a pain.
Best,
*Laimis J.*
On 5 Jan 2025, at 16:11, Anthony D'Atri <anthony.datri@gmail.com> wrote:
What reweighs have been set for the top OSDs (ceph osd df tree)?
Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening?
Old-style legacy override reweights don’t mesh well with the balancer. Best to leave them at 1.00.
0.2 is pretty extreme, back in the day I rarely went below 0.8.
``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again late ```
That should clear. The balancer doesn’t want to stir up trouble if the cluster already has a bunch of backfill / recovery going on. Patience!
default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB
That’s odd that the data pool is that full but the others aren’t.
Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool`
I also tried changing the following but it does not seem to persist:
Could be an mclock thing.
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc.
Probably a result of the autoscaler splitting PGs or of some change to CRUSH rules such that some data can’t be placed.
2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually?
I wouldn’t, that’s a symptom not the disease.
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha
Hi everyone. Yes. All the tips definitely helped! Now I have more free space in the pools, the number of misplaced PG's decreased a lot and lower std deviation of the usage of OSD's. The storage looks way healthier now. Thanks a bunch! I'm only confused by the number of misplaced PG's which never goes below 5%. Every time it hits 5% it goes up and down like shown in this quite interesting graph: [image: image.png] Any idea why that might be? I had the impression that it might be related to the autobalancer that kicks in and pg's are misplaced again. Or am I missing something? Bruno On Mon, 6 Jan 2025 at 16:00, Bruno Gomes Pessanha <bruno.pessanha@gmail.com> wrote:
So you might set the full ratio to .98, backfillfull to .96. Nearfull is
only cosmetic.
Thanks for the advice. It seems to be working with 0.92 for now. If it gets stuck I'll increase it.
On Mon, 6 Jan 2025 at 00:24, Anthony D'Atri <anthony.datri@gmail.com> wrote:
Very solid advice here - that’s the beauty of Ceph community.
Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and back) is quite extreme and the cluster won’t like it.
And these days with the balancer, pg-upmap entries to the same effect are a better idea.
From the clients perspective Your main concern now is to keep the pools “alive” with enough space while the backfilling takes place.
To that end, you can *temporarily* give yourself a bit more margin:
ceph osd set-nearfull-ratio .85 ceph osd set-backfillfull-ratio .90 ceph osd set-full-ratio .95
Those are the default values, and Ceph (now) enforces that the values are
= (or maybe >) in that order.
So you might set the full ratio to .98, backfillfull to .96. Nearfull is only cosmetic.
But absolutely do not forget to revert to default values once the cluster is balanced, or to other values that you make an educated decision to choose.
Even with plenty of OSDs that are not filled you might hit a single overfilled OSD and the whole pool will stop accepting new data.
Yep, see above. Not immediately clear to me why that data pool is so full unless the CRUSH rule / device classes are wonky.
Clients will start getting “No more space available” errors. That happened to us with CephFS recently with a very similar scenario where the cluster got much more data than expected in a short amount of time, not fun. With the balancer not working due to too many misplaced objects that’s an increased risk so just heads up and keep that in mind. To get things working we simply balanced manually the OSDs with upmaps moving data from the most full ones to the least full ones (our builtin balancer sadly does not work).
One small observation: I’ve noticed that 'ceph osd pool ls detail |grep cephfs.cephfs01.data’ has pg_num increased but the pgp_num is still the same. You will need to set it as well for data migration to new pgs to happen: https://docs.ceph.com/en/mimic/rados/operations/placement-groups/#set-the-nu...
The mgr usually does that for recent Ceph releases. With older releases we had to incremental pg_num and pgp_num in lockstep, which was kind of a pain.
Best,
*Laimis J.*
On 5 Jan 2025, at 16:11, Anthony D'Atri <anthony.datri@gmail.com> wrote:
What reweighs have been set for the top OSDs (ceph osd df tree)?
Right now they are all at 1.0. I had to lower them to something close to 0.2 in order to free up space but I changed them back to 1.0. Should I lower them while the backfill is happening?
Old-style legacy override reweights don’t mesh well with the balancer. Best to leave them at 1.00.
0.2 is pretty extreme, back in the day I rarely went below 0.8.
``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again late ```
That should clear. The balancer doesn’t want to stir up trouble if the cluster already has a bunch of backfill / recovery going on. Patience!
default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB
That’s odd that the data pool is that full but the others aren’t.
Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool`
I also tried changing the following but it does not seem to persist:
Could be an mclock thing.
1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc.
Probably a result of the autoscaler splitting PGs or of some change to CRUSH rules such that some data can’t be placed.
2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually?
I wouldn’t, that’s a symptom not the disease.
Bruno _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Bruno Gomes Pessanha
-- Bruno Gomes Pessanha
Den tors 16 jan. 2025 kl 00:08 skrev Bruno Gomes Pessanha < bruno.pessanha@gmail.com>:
Hi everyone. Yes. All the tips definitely helped! Now I have more free space in the pools, the number of misplaced PG's decreased a lot and lower std deviation of the usage of OSD's. The storage looks way healthier now. Thanks a bunch!
I'm only confused by the number of misplaced PG's which never goes below 5%. Every time it hits 5% it goes up and down like shown in this quite interesting graph: [image: image.png]
Any idea why that might be?
I had the impression that it might be related to the autobalancer that kicks in and pg's are misplaced again. Or am I missing something?
Yes, this seems to be the balancer keeping 5% of your PGs moving to the "correct" places. If you run an upmap remapper (or had PGs hindered by backfill_toofull) then the balancer might have a long list it wants to move, but it "only" does 5% at most at a time. -- May the most significant bit of your life be positive.
participants (6)
-
Anthony D'Atri
-
Bruno Gomes Pessanha
-
bruno.pessanha＠gmail.com
-
Burkhard Linke
-
Janne Johansson
-
Laimis Juzeliūnas