Dear Ceph users, I'd need some help with CephFS. Originally, my cluster consisted of 12 hosts with different total raw disk size, ranging from 16 to 32 TB; there were two pools, one 3x replicated metadata and one 6+2 EC data with host failure domain, and which I access via CephFS. In total there were 512 PGs. Naively, I would expect that the available size of the data pool would be about 75% of the total available space given the EC parameters (6/(6+2) = 75%), but I noticed that it was actually lower, around 66% (as reported by df -h, while for the total raw capacity I used the one reported by the Ceph dashboard). I guessed this could be due to smaller hosts limiting the total amount of available space for the filesystem, so I started a disk upgrade campaign to make the host sizes more even. For example, for a 16 TB host with 8x2TB I replaced two disks with 8 TB ones, for a new raw size of 28 TB. I did this and similar upgrades for other hosts in steps, every time waiting for backfill to finish and checking the new filesystem size before proceeding. And I noticed that the filesystem size reported by df -h always remained at 66% of the total raw capacity, despite host sizes are now all between 24 and 32 TB. So my guess about the limiting factor for the relative fs size is evidently wrong. I thought that for some reason maybe 66% is the actual limit and also about some possible explanation (e.g. some space being reserved for the metadata pool), and even considered the possibility that the size reported by df -h being meaningless, but I'd need some advice from an expert to sort this out. Thanks in advance for any help. Nicola
Remember that `ceph df` takes into account the full ratio reserved space, and the headroom between that threshold and the most-full OSD. Run `ceph osd df` and look at the PGs and VAR columns https://www.ibm.com/docs/en/storage-ceph/7?topic=monitoring-understanding-os... If you have high variability, you may have an issue with the balancer not being enabled or a CRUSH nuance preventing it from working. If the PGs column is REALLY low you could have a bin-packing phenomenon. I suspect a balancer issue. Also, when your OSDs vary a lot in size, it’s best to balance the cumulative weights of each failure domain. In your case, that’s probably host, so ceph osd tree | grep host and compare the weights. If your failure domain is host for all CRUSH rules, then I would think that 24-32 TB host weight variance would be not a big problem since there are more than 6+2 of them. If your failure domain is rack, then do the above subbing in rack. Send your entire ceph osd tree and ceph osd df here if you like, since it’ll be short, and check your CRUSH rules. I suspect that you have at least one OSD outlier that is more full than the others.
On Jan 2, 2025, at 2:56 AM, Nicola Mori <mori@fi.infn.it> wrote:
Dear Ceph users,
I'd need some help with CephFS. Originally, my cluster consisted of 12 hosts with different total raw disk size, ranging from 16 to 32 TB; there were two pools, one 3x replicated metadata and one 6+2 EC data with host failure domain, and which I access via CephFS. In total there were 512 PGs.
Naively, I would expect that the available size of the data pool would be about 75% of the total available space given the EC parameters (6/(6+2) = 75%), but I noticed that it was actually lower, around 66% (as reported by df -h, while for the total raw capacity I used the one reported by the Ceph dashboard). I guessed this could be due to smaller hosts limiting the total amount of available space for the filesystem, so I started a disk upgrade campaign to make the host sizes more even. For example, for a 16 TB host with 8x2TB I replaced two disks with 8 TB ones, for a new raw size of 28 TB. I did this and similar upgrades for other hosts in steps, every time waiting for backfill to finish and checking the new filesystem size before proceeding. And I noticed that the filesystem size reported by df -h always remained at 66% of the total raw capacity, despite host sizes are now all between 24 and 32 TB.
So my guess about the limiting factor for the relative fs size is evidently wrong. I thought that for some reason maybe 66% is the actual limit and also about some possible explanation (e.g. some space being reserved for the metadata pool), and even considered the possibility that the size reported by df -h being meaningless, but I'd need some advice from an expert to sort this out.
Thanks in advance for any help.
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, thanks for your insights. I actually used df -h from the bash shell of a machine mounting the CephFS with the kernel module, and here's the current result: wizardfs_rootsquash@b1029256-7bb3-11ec-a8ce-ac1f6b627b45.wizardfs=/ 217T 78T 139T 36% /wizard/ceph So it seems the fs size is 217 TiB, which is about 66% of the total amount of raw disk space (320 TiB) as I wrote before. Then I tried the command you suggested: # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 320 TiB 216 TiB 104 TiB 104 TiB 32.56 TOTAL 320 TiB 216 TiB 104 TiB 104 TiB 32.56 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 242 MiB 62 726 MiB 0 62 TiB wizard_metadata 2 16 1.2 GiB 85.75k 3.5 GiB 0 62 TiB wizard_data 3 512 78 TiB 27.03M 104 TiB 36.06 138 TiB In order to find the total size of the data pool I don't understand how to interpret the "MAX AVAIL" column: should it be summed to "STORED" or to "USED"? In the first case I'd get 216 TiB which corresponds to what df -h says and thus to 66%, in the second case I'd get 242 TiB which is very close to 75%... But I guess the first option is the right one. Then I looked at the weights of my failure domain (host): # ceph osd tree | grep host -7 25.51636 host aka -3 25.51636 host balin -13 29.10950 host bifur -17 29.10950 host bofur -21 29.10371 host dwalin -23 21.83276 host fili -25 29.10950 host kili -9 25.51636 host ogion -19 25.51636 host prestno -15 29.10522 host remolo -5 25.51636 host rokanan -11 27.29063 host romolo They seem quite even and quite reflecting the actual total size of each host: # ceph orch host ls --detail HOST . . . HDD aka 9/28.3TB balin 9/28.3TB bifur 9/32.5TB bofur 8/32.0TB dwalin 16/32.0TB fili 12/24.0TB kili 8/32.0TB ogion 8/28.0TB prestno 9/28.3TB remolo 16/32.0TB rokanan 9/28.5TB romolo 16/30.0TB so I see no problem here (in fact, making these even is the idea behind the disk upgrade strategy I am pursuing). About the OSD outlier: there seems to be not such an OSD, the maximum OSD occupancy is 38% and it smoothly decreases down to a minimum of 27% with no jumps. About PGs: I have 512 PGs in the data pool and 124 OSDs in total, maybe the count is too low but I'm hesitant to increase it since my cluster is very low specs and I fear to run out of memory on the oldest machines. About CRUSH rules: I don't know exactly what to search for, so if you believe it's important then I'd need some advice. Thank you again for your precious help, Nicola
On Jan 2, 2025, at 11:18 AM, Nicola Mori <mori@fi.infn.it> wrote:
Hi Anthony, thanks for your insights. I actually used df -h from the bash shell of a machine mounting the CephFS with the kernel module, and here's the current result:
wizardfs_rootsquash@b1029256-7bb3-11ec-a8ce-ac1f6b627b45.wizardfs=/ 217T 78T 139T 36% /wizard/ceph
So it seems the fs size is 217 TiB, which is about 66% of the total amount of raw disk space (320 TiB) as I wrote before.
Then I tried the command you suggested:
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 320 TiB 216 TiB 104 TiB 104 TiB 32.56 TOTAL 320 TiB 216 TiB 104 TiB 104 TiB 32.56
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 242 MiB 62 726 MiB 0 62 TiB wizard_metadata 2 16 1.2 GiB 85.75k 3.5 GiB 0 62 TiB wizard_data 3 512 78 TiB 27.03M 104 TiB 36.06 138 TiB
In order to find the total size of the data pool I don't understand how to interpret the "MAX AVAIL" column: should it be summed to "STORED" or to "USED”?
Do you have a lot of small files?
In the first case I'd get 216 TiB which corresponds to what df -h says and thus to 66%, in the second case I'd get 242 TiB which is very close to 75%... But I guess the first option is the right one.
Then I looked at the weights of my failure domain (host):
# ceph osd tree | grep host
-7 25.51636 host aka -3 25.51636 host balin -13 29.10950 host bifur -17 29.10950 host bofur -21 29.10371 host dwalin -23 21.83276 host fili -25 29.10950 host kili -9 25.51636 host ogion -19 25.51636 host prestno -15 29.10522 host remolo -5 25.51636 host rokanan -11 27.29063 host romolo
They seem quite even and quite reflecting the actual total size of each host:
# ceph orch host ls --detail HOST . . . HDD aka 9/28.3TB balin 9/28.3TB bifur 9/32.5TB bofur 8/32.0TB dwalin 16/32.0TB fili 12/24.0TB kili 8/32.0TB ogion 8/28.0TB prestno 9/28.3TB remolo 16/32.0TB rokanan 9/28.5TB romolo 16/30.0TB
so I see no problem here (in fact, making these even is the idea behind the disk upgrade strategy I am pursuing).
About the OSD outlier: there seems to be not such an OSD, the maximum OSD occupancy is 38% and it smoothly decreases down to a minimum of 27% with no jumps.
That’s a very high variance. If the balancer is working it should be like +/- 1-2%. Available space in the cluster will be reported as though all OSDs are 38%.
About PGs: I have 512 PGs in the data pool and 124 OSDs in total, maybe the count is too low but I'm hesitant to increase it since my cluster is very low specs and I fear to run out of memory on the oldest machines.
About CRUSH rules: I don't know exactly what to search for, so if you believe it's important then I'd need some advice.
Thank you again for your precious help,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have a single user producing lots of small files (currently about 4.7M with a mean size of 3 MB). The total number of files is about 7M. About the occupancy: in 1.8 TiB disks I see the PG count ranging from 27 (-> 38% occupancy) to 20 (-> 27% occupancy) at the same OSD weight (1.819). I guess these fluctuations of the number of PGs are due to the small number of PGs coupled to the inefficiency of the balancer, do you agree? If it's correct then I see only two ways: a manual rebalancing (tried in the past with much effort and little results) or an increase in PG count (risky because of old hardware), do you see any other possibility? Cheers, Nicola On 02/01/25 5:30 PM, Anthony D'Atri wrote:
On Jan 2, 2025, at 11:18 AM, Nicola Mori <mori@fi.infn.it> wrote:
Hi Anthony, thanks for your insights. I actually used df -h from the bash shell of a machine mounting the CephFS with the kernel module, and here's the current result:
wizardfs_rootsquash@b1029256-7bb3-11ec-a8ce-ac1f6b627b45.wizardfs=/ 217T 78T 139T 36% /wizard/ceph
So it seems the fs size is 217 TiB, which is about 66% of the total amount of raw disk space (320 TiB) as I wrote before.
Then I tried the command you suggested:
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 320 TiB 216 TiB 104 TiB 104 TiB 32.56 TOTAL 320 TiB 216 TiB 104 TiB 104 TiB 32.56
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 242 MiB 62 726 MiB 0 62 TiB wizard_metadata 2 16 1.2 GiB 85.75k 3.5 GiB 0 62 TiB wizard_data 3 512 78 TiB 27.03M 104 TiB 36.06 138 TiB
In order to find the total size of the data pool I don't understand how to interpret the "MAX AVAIL" column: should it be summed to "STORED" or to "USED”?
Do you have a lot of small files?
In the first case I'd get 216 TiB which corresponds to what df -h says and thus to 66%, in the second case I'd get 242 TiB which is very close to 75%... But I guess the first option is the right one.
Then I looked at the weights of my failure domain (host):
# ceph osd tree | grep host
-7 25.51636 host aka -3 25.51636 host balin -13 29.10950 host bifur -17 29.10950 host bofur -21 29.10371 host dwalin -23 21.83276 host fili -25 29.10950 host kili -9 25.51636 host ogion -19 25.51636 host prestno -15 29.10522 host remolo -5 25.51636 host rokanan -11 27.29063 host romolo
They seem quite even and quite reflecting the actual total size of each host:
# ceph orch host ls --detail HOST . . . HDD aka 9/28.3TB balin 9/28.3TB bifur 9/32.5TB bofur 8/32.0TB dwalin 16/32.0TB fili 12/24.0TB kili 8/32.0TB ogion 8/28.0TB prestno 9/28.3TB remolo 16/32.0TB rokanan 9/28.5TB romolo 16/30.0TB
so I see no problem here (in fact, making these even is the idea behind the disk upgrade strategy I am pursuing).
About the OSD outlier: there seems to be not such an OSD, the maximum OSD occupancy is 38% and it smoothly decreases down to a minimum of 27% with no jumps.
That’s a very high variance. If the balancer is working it should be like +/- 1-2%. Available space in the cluster will be reported as though all OSDs are 38%.
About PGs: I have 512 PGs in the data pool and 124 OSDs in total, maybe the count is too low but I'm hesitant to increase it since my cluster is very low specs and I fear to run out of memory on the oldest machines.
About CRUSH rules: I don't know exactly what to search for, so if you believe it's important then I'd need some advice.
Thank you again for your precious help,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it
They seem quite even
Indeed. Assuming that your failure domain is host, that shouldn’t be a factor in stranded capacity. We mostly see that happen with say a rack failure domain cluster with 3 racks and replicated pools, or with your 6,2 pool 6 racks. Having failure domains > replication eases those concerns.
About CRUSH rules: I don't know exactly what to search for, so if you believe it's important then I'd need some advice.
ceph osd crush rule dump
I have a single user producing lots of small files (currently about 4.7M with a mean size of 3 MB). The total number of files is about 7M.
That could contribute to stored vs used disparity, since Ceph (currently) writes full stripes. The data pool at EC 6,2 will allocate underlying storage in multiples of 8*4=32KB. So if there are a substantial number of objects smaller than, say, 128KB, they will strand some percentage of capacity. This visualization shows that as the object size increases, the potential space amp quickly falls into the background noise. https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToPQjN8bDI/edit?gid=358760253#gid=358760253 Bluestore Space Amplification Cheat Sheet docs.google.com Now, I didn’t think to ask before which Ceph release you’re running, and more importantly which was running when the OSDs were built. Around the Octopus / Pacific timeframe the defaults for min_alloc_size were reduced from 64KB to 4KB to minimize space amp, especially for small RGW objects. If you’re running a recent release now `ceph osd metadata` will show you the value baked into each OSD.
About the occupancy: in 1.8 TiB disks I see the PG count ranging from 27 (-> 38% occupancy) to 20 (-> 27% occupancy) at the same OSD weight (1.819). I guess these fluctuations of the number of PGs are due to the small number of PGs
I think we haven’t seen your PG count. `ceph osd df` please. And for completeness `ceph osd dump | grep pool`
coupled to the inefficiency of the balancer, do you agree?
If the balancer were working it would do better than a 27-38 spread.
If it's correct then I see only two ways: a manual rebalancing (tried in the past with much effort and little results)
Did you use `reweight by utilization`? If `ceph osd tree` shows OSDs with a REWEIGHT value < 1.0000 that could be a factor. Mixing old-style override reweighs with new-style pg-upmap can confuse the balancer. If you do have OSDs with REWEIGHT values set, try resetting them to 1.000.
or an increase in PG count (risky because of old hardware)
I don’t think increasing PGs would intersect with old hardware, unless perhaps you’re riiiight at the edge with respect to RAM. More PGs will use a bit more RAM within OSD processes, but at your scale I doubt that will be significant. Send the above and we’ll be able to pass judgement on your pg_nums.
do you see any other possibility?
https://www.syfy.com/sites/syfy/files/styles/hero_image__large__computer__alt/public/wire/legacy/itsaliens.jpg itsaliens JPEG Image · 73 KB
Cheers,
Nicola
On 02/01/25 5:30 PM, Anthony D'Atri wrote:
On Jan 2, 2025, at 11:18 AM, Nicola Mori <mori@fi.infn.it> wrote:
Hi Anthony, thanks for your insights. I actually used df -h from the bash shell of a machine mounting the CephFS with the kernel module, and here's the current result:
wizardfs_rootsquash@b1029256-7bb3-11ec-a8ce-ac1f6b627b45.wizardfs=/ 217T 78T 139T 36% /wizard/ceph
So it seems the fs size is 217 TiB, which is about 66% of the total amount of raw disk space (320 TiB) as I wrote before.
Then I tried the command you suggested:
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 320 TiB 216 TiB 104 TiB 104 TiB 32.56 TOTAL 320 TiB 216 TiB 104 TiB 104 TiB 32.56
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 242 MiB 62 726 MiB 0 62 TiB wizard_metadata 2 16 1.2 GiB 85.75k 3.5 GiB 0 62 TiB wizard_data 3 512 78 TiB 27.03M 104 TiB 36.06 138 TiB
In order to find the total size of the data pool I don't understand how to interpret the "MAX AVAIL" column: should it be summed to "STORED" or to "USED”? Do you have a lot of small files? In the first case I'd get 216 TiB which corresponds to what df -h says and thus to 66%, in the second case I'd get 242 TiB which is very close to 75%... But I guess the first option is the right one.
Then I looked at the weights of my failure domain (host):
# ceph osd tree | grep host
-7 25.51636 host aka -3 25.51636 host balin -13 29.10950 host bifur -17 29.10950 host bofur -21 29.10371 host dwalin -23 21.83276 host fili -25 29.10950 host kili -9 25.51636 host ogion -19 25.51636 host prestno -15 29.10522 host remolo -5 25.51636 host rokanan -11 27.29063 host romolo
They seem quite even and quite reflecting the actual total size of each host:
# ceph orch host ls --detail HOST . . . HDD aka 9/28.3TB balin 9/28.3TB bifur 9/32.5TB bofur 8/32.0TB dwalin 16/32.0TB fili 12/24.0TB kili 8/32.0TB ogion 8/28.0TB prestno 9/28.3TB remolo 16/32.0TB rokanan 9/28.5TB romolo 16/30.0TB
so I see no problem here (in fact, making these even is the idea behind the disk upgrade strategy I am pursuing).
About the OSD outlier: there seems to be not such an OSD, the maximum OSD occupancy is 38% and it smoothly decreases down to a minimum of 27% with no jumps. That’s a very high variance. If the balancer is working it should be like +/- 1-2%. Available space in the cluster will be reported as though all OSDs are 38%.
About PGs: I have 512 PGs in the data pool and 124 OSDs in total, maybe the count is too low but I'm hesitant to increase it since my cluster is very low specs and I fear to run out of memory on the oldest machines.
About CRUSH rules: I don't know exactly what to search for, so if you believe it's important then I'd need some advice.
Thank you again for your precious help,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I’m not sure where the doubts about old hardware and pg splits come from. We observed the opposite of what you seem to fear (increasing memory usage) after a pg split on a customer’s cluster last year. According to their Prometheus data the memory usage dropped after the split had finished. I don’t have too many data sources available, but we’ve never seen memory issues during pg splits. I would strongly recommend to consider increasing your pg count. Zitat von Nicola Mori <mori@fi.infn.it>:
I have a single user producing lots of small files (currently about 4.7M with a mean size of 3 MB). The total number of files is about 7M.
About the occupancy: in 1.8 TiB disks I see the PG count ranging from 27 (-> 38% occupancy) to 20 (-> 27% occupancy) at the same OSD weight (1.819). I guess these fluctuations of the number of PGs are due to the small number of PGs coupled to the inefficiency of the balancer, do you agree? If it's correct then I see only two ways: a manual rebalancing (tried in the past with much effort and little results) or an increase in PG count (risky because of old hardware), do you see any other possibility?
Cheers,
Nicola
On 02/01/25 5:30 PM, Anthony D'Atri wrote:
On Jan 2, 2025, at 11:18 AM, Nicola Mori <mori@fi.infn.it> wrote:
Hi Anthony, thanks for your insights. I actually used df -h from the bash shell of a machine mounting the CephFS with the kernel module, and here's the current result:
wizardfs_rootsquash@b1029256-7bb3-11ec-a8ce-ac1f6b627b45.wizardfs=/ 217T 78T 139T 36% /wizard/ceph
So it seems the fs size is 217 TiB, which is about 66% of the total amount of raw disk space (320 TiB) as I wrote before.
Then I tried the command you suggested:
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 320 TiB 216 TiB 104 TiB 104 TiB 32.56 TOTAL 320 TiB 216 TiB 104 TiB 104 TiB 32.56
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 242 MiB 62 726 MiB 0 62 TiB wizard_metadata 2 16 1.2 GiB 85.75k 3.5 GiB 0 62 TiB wizard_data 3 512 78 TiB 27.03M 104 TiB 36.06 138 TiB
In order to find the total size of the data pool I don't understand how to interpret the "MAX AVAIL" column: should it be summed to "STORED" or to "USED”?
Do you have a lot of small files?
In the first case I'd get 216 TiB which corresponds to what df -h says and thus to 66%, in the second case I'd get 242 TiB which is very close to 75%... But I guess the first option is the right one.
Then I looked at the weights of my failure domain (host):
# ceph osd tree | grep host
-7 25.51636 host aka -3 25.51636 host balin -13 29.10950 host bifur -17 29.10950 host bofur -21 29.10371 host dwalin -23 21.83276 host fili -25 29.10950 host kili -9 25.51636 host ogion -19 25.51636 host prestno -15 29.10522 host remolo -5 25.51636 host rokanan -11 27.29063 host romolo
They seem quite even and quite reflecting the actual total size of each host:
# ceph orch host ls --detail HOST . . . HDD aka 9/28.3TB balin 9/28.3TB bifur 9/32.5TB bofur 8/32.0TB dwalin 16/32.0TB fili 12/24.0TB kili 8/32.0TB ogion 8/28.0TB prestno 9/28.3TB remolo 16/32.0TB rokanan 9/28.5TB romolo 16/30.0TB
so I see no problem here (in fact, making these even is the idea behind the disk upgrade strategy I am pursuing).
About the OSD outlier: there seems to be not such an OSD, the maximum OSD occupancy is 38% and it smoothly decreases down to a minimum of 27% with no jumps.
That’s a very high variance. If the balancer is working it should be like +/- 1-2%. Available space in the cluster will be reported as though all OSDs are 38%.
About PGs: I have 512 PGs in the data pool and 124 OSDs in total, maybe the count is too low but I'm hesitant to increase it since my cluster is very low specs and I fear to run out of memory on the oldest machines.
About CRUSH rules: I don't know exactly what to search for, so if you believe it's important then I'd need some advice.
Thank you again for your precious help,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@fi.infn.it
First of all, thank you so much again for the time you spend in trying to help me, it's much appreciated. Then: - here's the dump of the CRUSH rules: # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "wizard_data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 250 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] - Running `ceph osd metadata | grep min_alloc_size` shows: "bluestore_min_alloc_size": "4096" for all the OSDs - About PG count: currently I have two OSDs destroyed waiting for disk upgrade; anyway this doesn't change the overall picture. That said, here's the info you requested: # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 4 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 53 MiB 3.4 GiB 1.2 TiB 32.22 0.99 24 up 11 hdd 1.81940 1.00000 1.8 TiB 624 GiB 622 GiB 58 MiB 2.1 GiB 1.2 TiB 33.50 1.03 25 up 18 hdd 1.81940 1.00000 1.8 TiB 677 GiB 674 GiB 80 MiB 3.4 GiB 1.2 TiB 36.36 1.12 27 up 26 hdd 1.81940 1.00000 1.8 TiB 625 GiB 623 GiB 60 MiB 2.5 GiB 1.2 TiB 33.55 1.03 24 up 32 hdd 1.81940 1.00000 1.8 TiB 625 GiB 623 GiB 2 KiB 1.7 GiB 1.2 TiB 33.53 1.03 24 up 41 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 9 KiB 8.6 GiB 4.8 TiB 33.51 1.03 97 up 48 hdd 1.81940 1.00000 1.8 TiB 625 GiB 622 GiB 59 MiB 2.8 GiB 1.2 TiB 33.52 1.03 25 up 55 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 52 MiB 7.8 GiB 4.8 TiB 33.51 1.03 97 up 0 hdd 1.81940 1.00000 1.8 TiB 626 GiB 622 GiB 2 KiB 3.6 GiB 1.2 TiB 33.59 1.03 24 up 8 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 59 MiB 6.0 GiB 4.8 TiB 33.49 1.03 97 up 15 hdd 1.81940 1.00000 1.8 TiB 572 GiB 569 GiB 16 KiB 3.4 GiB 1.3 TiB 30.71 0.94 22 up 22 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 56 MiB 2.1 GiB 1.2 TiB 36.29 1.12 27 up 29 hdd 1.81940 1.00000 1.8 TiB 573 GiB 570 GiB 59 MiB 2.3 GiB 1.3 TiB 30.74 0.95 23 up 34 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 113 MiB 7.3 GiB 4.9 TiB 32.15 0.99 94 up 43 hdd 1.81940 1.00000 1.8 TiB 572 GiB 570 GiB 15 KiB 2.7 GiB 1.3 TiB 30.73 0.95 22 up 49 hdd 1.81940 1.00000 1.8 TiB 547 GiB 544 GiB 4 KiB 2.9 GiB 1.3 TiB 29.38 0.90 21 up 3 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 7 KiB 3.8 GiB 2.5 TiB 30.73 0.95 44 up 14 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 51 MiB 4.5 GiB 2.4 TiB 34.92 1.07 51 up 27 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 15 KiB 3.1 GiB 2.4 TiB 34.91 1.07 50 up 37 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 51 MiB 3.3 GiB 2.5 TiB 29.99 0.92 44 up 50 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 22 KiB 3.7 GiB 2.4 TiB 34.88 1.07 50 up 59 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 58 MiB 3.7 GiB 2.5 TiB 30.02 0.92 44 up 64 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 28 KiB 3.9 GiB 2.5 TiB 30.02 0.92 43 up 69 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 53 MiB 3.4 GiB 2.4 TiB 34.19 1.05 50 up 2 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 59 MiB 4.2 GiB 2.4 TiB 34.90 1.07 51 up 21 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 56 MiB 3.5 GiB 2.5 TiB 30.01 0.92 44 up 39 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 49 MiB 4.3 GiB 2.4 TiB 34.20 1.05 50 up 57 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 15 MiB 4.2 GiB 2.4 TiB 34.26 1.05 49 up 66 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 148 KiB 3.5 GiB 2.5 TiB 31.40 0.97 45 up 72 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 50 MiB 3.9 GiB 2.5 TiB 30.75 0.95 45 up 76 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 19 KiB 4.1 GiB 2.5 TiB 30.05 0.92 43 up 79 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 47 KiB 3.3 GiB 2.5 TiB 32.10 0.99 46 up 88 hdd 1.81897 1.00000 1.8 TiB 650 GiB 648 GiB 54 KiB 2.8 GiB 1.2 TiB 34.92 1.07 25 up 89 hdd 1.81898 1.00000 1.8 TiB 573 GiB 570 GiB 54 MiB 2.4 GiB 1.3 TiB 30.76 0.95 23 up 90 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 19 KiB 1.8 GiB 1.2 TiB 34.91 1.07 25 up 91 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 24 KiB 2.3 GiB 1.2 TiB 34.92 1.07 25 up 92 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 11 KiB 1.6 GiB 1.2 TiB 34.90 1.07 25 up 93 hdd 1.81898 1.00000 1.8 TiB 548 GiB 545 GiB 129 MiB 2.8 GiB 1.3 TiB 29.42 0.91 21 up 94 hdd 1.81897 1.00000 1.8 TiB 548 GiB 545 GiB 29 MiB 3.2 GiB 1.3 TiB 29.43 0.91 21 up 95 hdd 1.81898 1.00000 1.8 TiB 677 GiB 674 GiB 7 KiB 2.6 GiB 1.2 TiB 36.33 1.12 26 up 96 hdd 1.81898 1.00000 1.8 TiB 573 GiB 570 GiB 1.8 MiB 2.8 GiB 1.3 TiB 30.75 0.95 22 up 97 hdd 1.81898 1.00000 1.8 TiB 546 GiB 544 GiB 20 KiB 1.9 GiB 1.3 TiB 29.32 0.90 21 up 98 hdd 1.81898 1.00000 1.8 TiB 547 GiB 545 GiB 24 KiB 1.9 GiB 1.3 TiB 29.34 0.90 21 up 99 hdd 1.81898 1.00000 1.8 TiB 677 GiB 674 GiB 64 MiB 2.4 GiB 1.2 TiB 36.33 1.12 27 up 100 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 13 KiB 1.7 GiB 1.2 TiB 34.90 1.07 25 up 101 hdd 1.81898 1.00000 1.8 TiB 547 GiB 545 GiB 104 MiB 1.9 GiB 1.3 TiB 29.38 0.90 23 up 102 hdd 1.81898 1.00000 1.8 TiB 599 GiB 596 GiB 315 KiB 2.0 GiB 1.2 TiB 32.13 0.99 23 up 103 hdd 1.81897 1.00000 1.8 TiB 599 GiB 596 GiB 21 KiB 2.6 GiB 1.2 TiB 32.14 0.99 23 up 80 hdd 1.81940 1.00000 1.8 TiB 677 GiB 674 GiB 9 KiB 3.0 GiB 1.2 TiB 36.32 1.12 26 up 106 hdd 1.81940 1.00000 1.8 TiB 677 GiB 674 GiB 2 KiB 3.0 GiB 1.2 TiB 36.31 1.12 26 up 107 hdd 1.81940 1.00000 1.8 TiB 547 GiB 545 GiB 27 KiB 2.5 GiB 1.3 TiB 29.38 0.90 21 up 108 hdd 1.81940 1.00000 1.8 TiB 625 GiB 622 GiB 109 MiB 3.0 GiB 1.2 TiB 33.57 1.03 25 up 109 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 4 KiB 1.8 GiB 1.2 TiB 36.27 1.12 26 up 110 hdd 1.81940 1.00000 1.8 TiB 572 GiB 571 GiB 5 KiB 1.6 GiB 1.3 TiB 30.72 0.95 22 up 111 hdd 1.81940 1.00000 1.8 TiB 651 GiB 648 GiB 59 MiB 2.7 GiB 1.2 TiB 34.93 1.07 26 up 112 hdd 1.81940 1.00000 1.8 TiB 598 GiB 595 GiB 59 MiB 2.3 GiB 1.2 TiB 32.08 0.99 24 up 113 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 4 KiB 2.4 GiB 1.2 TiB 36.29 1.12 26 up 114 hdd 1.81940 1.00000 1.8 TiB 624 GiB 622 GiB 2 KiB 1.8 GiB 1.2 TiB 33.49 1.03 24 up 115 hdd 1.81940 1.00000 1.8 TiB 521 GiB 519 GiB 4 KiB 2.3 GiB 1.3 TiB 27.96 0.86 20 up 116 hdd 1.81940 1.00000 1.8 TiB 650 GiB 647 GiB 2 KiB 2.9 GiB 1.2 TiB 34.91 1.07 25 up 62 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 103 MiB 3.7 GiB 2.4 TiB 34.17 1.05 51 up 117 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 60 MiB 4.1 GiB 2.4 TiB 34.20 1.05 50 up 118 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 139 KiB 4.3 GiB 2.5 TiB 31.41 0.97 45 up 119 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 52 MiB 4.2 GiB 2.4 TiB 34.23 1.05 50 up 120 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 2 KiB 4.2 GiB 2.4 TiB 32.83 1.01 47 up 121 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 2 KiB 3.8 GiB 2.6 TiB 29.34 0.90 42 up 122 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 10 KiB 3.9 GiB 2.5 TiB 31.41 0.97 45 up 123 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 85 MiB 3.9 GiB 2.5 TiB 30.71 0.94 45 up 7 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 10 KiB 5.5 GiB 5.0 TiB 31.02 0.95 90 up 16 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 115 MiB 6.0 GiB 5.0 TiB 31.05 0.96 91 up 23 hdd 1.81940 1.00000 1.8 TiB 598 GiB 596 GiB 9 KiB 2.3 GiB 1.2 TiB 32.11 0.99 23 up 33 hdd 1.81940 1.00000 1.8 TiB 652 GiB 650 GiB 14 KiB 2.8 GiB 1.2 TiB 35.01 1.08 25 up 40 hdd 1.81940 1.00000 1.8 TiB 600 GiB 598 GiB 1.0 MiB 2.4 GiB 1.2 TiB 32.21 0.99 23 up 47 hdd 1.81940 1.00000 1.8 TiB 521 GiB 519 GiB 14 KiB 2.4 GiB 1.3 TiB 27.98 0.86 20 up 54 hdd 1.81940 1.00000 1.8 TiB 522 GiB 519 GiB 4.5 MiB 2.8 GiB 1.3 TiB 28.00 0.86 21 up 61 hdd 1.81940 1.00000 1.8 TiB 548 GiB 545 GiB 12 KiB 2.9 GiB 1.3 TiB 29.39 0.90 21 up 81 hdd 1.81940 1.00000 1.8 TiB 522 GiB 518 GiB 24 KiB 3.6 GiB 1.3 TiB 28.02 0.86 20 up 82 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 7 KiB 6.7 GiB 4.8 TiB 33.49 1.03 96 up 83 hdd 1.81940 1.00000 1.8 TiB 521 GiB 518 GiB 2 KiB 2.8 GiB 1.3 TiB 27.98 0.86 20 up 84 hdd 1.81940 1.00000 1.8 TiB 625 GiB 621 GiB 229 KiB 3.7 GiB 1.2 TiB 33.54 1.03 24 up 85 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 56 MiB 7.5 GiB 5.0 TiB 31.45 0.97 91 up 86 hdd 1.81940 1.00000 1.8 TiB 650 GiB 648 GiB 3 KiB 2.6 GiB 1.2 TiB 34.91 1.07 25 up 87 hdd 1.81940 1.00000 1.8 TiB 522 GiB 519 GiB 55 MiB 2.7 GiB 1.3 TiB 27.99 0.86 21 up 104 hdd 1.81940 1.00000 1.8 TiB 520 GiB 518 GiB 1.0 MiB 1.7 GiB 1.3 TiB 27.89 0.86 20 up 6 hdd 1.81897 1.00000 1.8 TiB 598 GiB 595 GiB 26 KiB 2.3 GiB 1.2 TiB 32.08 0.99 23 up 12 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 54 MiB 2.8 GiB 1.2 TiB 32.22 0.99 24 up 19 hdd 1.81897 1.00000 1.8 TiB 599 GiB 597 GiB 4 KiB 2.1 GiB 1.2 TiB 32.15 0.99 23 up 28 hdd 1.81897 1.00000 1.8 TiB 678 GiB 675 GiB 51 MiB 2.9 GiB 1.2 TiB 36.37 1.12 27 up 35 hdd 1.81897 1.00000 1.8 TiB 572 GiB 571 GiB 10 KiB 1.5 GiB 1.3 TiB 30.72 0.95 22 up 44 hdd 1.81897 1.00000 1.8 TiB 651 GiB 649 GiB 55 MiB 2.1 GiB 1.2 TiB 34.94 1.07 26 up 52 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 21 KiB 2.2 GiB 1.2 TiB 36.29 1.12 26 up 58 hdd 1.81897 1.00000 1.8 TiB 677 GiB 674 GiB 10 KiB 2.9 GiB 1.2 TiB 36.36 1.12 26 up 63 hdd 1.81897 1.00000 1.8 TiB 572 GiB 571 GiB 17 KiB 1.7 GiB 1.3 TiB 30.73 0.95 22 up 67 hdd 1.81897 1.00000 1.8 TiB 652 GiB 649 GiB 55 MiB 2.8 GiB 1.2 TiB 34.99 1.08 26 up 71 hdd 1.81897 1.00000 1.8 TiB 625 GiB 623 GiB 2 KiB 2.0 GiB 1.2 TiB 33.55 1.03 24 up 73 hdd 1.81940 1.00000 1.8 TiB 650 GiB 648 GiB 14 KiB 2.8 GiB 1.2 TiB 34.92 1.07 25 up 74 hdd 1.81897 1.00000 1.8 TiB 600 GiB 597 GiB 20 KiB 3.0 GiB 1.2 TiB 32.20 0.99 23 up 75 hdd 1.81897 1.00000 1.8 TiB 572 GiB 570 GiB 29 KiB 2.1 GiB 1.3 TiB 30.71 0.94 22 up 77 hdd 1.81897 1.00000 1.8 TiB 600 GiB 597 GiB 51 MiB 3.0 GiB 1.2 TiB 32.20 0.99 24 up 78 hdd 1.81940 1.00000 1.8 TiB 650 GiB 648 GiB 21 KiB 2.1 GiB 1.2 TiB 34.90 1.07 25 up 1 hdd 1.81940 1.00000 1.8 TiB 678 GiB 675 GiB 58 MiB 3.3 GiB 1.2 TiB 36.39 1.12 27 up 10 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 52 MiB 7.1 GiB 4.9 TiB 33.15 1.02 97 up 17 hdd 1.81940 1.00000 1.8 TiB 495 GiB 493 GiB 51 MiB 2.0 GiB 1.3 TiB 26.55 0.82 20 up 24 hdd 1.81940 1.00000 1.8 TiB 548 GiB 544 GiB 3.4 MiB 3.8 GiB 1.3 TiB 29.40 0.90 21 up 31 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 60 MiB 8.4 GiB 5.0 TiB 31.44 0.97 91 up 38 hdd 1.81940 1.00000 1.8 TiB 571 GiB 570 GiB 26 KiB 1.6 GiB 1.3 TiB 30.67 0.94 22 up 46 hdd 1.81940 1.00000 1.8 TiB 651 GiB 649 GiB 2 KiB 2.3 GiB 1.2 TiB 34.95 1.08 25 up 53 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 9 KiB 2.8 GiB 1.2 TiB 32.18 0.99 23 up 5 hdd 1.81940 1.00000 1.8 TiB 676 GiB 673 GiB 55 MiB 2.3 GiB 1.2 TiB 36.27 1.12 27 up 9 hdd 1.81999 1.00000 1.8 TiB 677 GiB 674 GiB 15 KiB 3.0 GiB 1.2 TiB 36.31 1.12 26 up 13 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 59 MiB 2.5 GiB 1.2 TiB 36.30 1.12 27 up 20 hdd 1.81940 1.00000 1.8 TiB 573 GiB 571 GiB 9 KiB 2.7 GiB 1.3 TiB 30.77 0.95 22 up 25 hdd 1.81999 1.00000 1.8 TiB 677 GiB 674 GiB 2 KiB 3.0 GiB 1.2 TiB 36.33 1.12 26 up 30 hdd 1.81940 1.00000 1.8 TiB 573 GiB 571 GiB 49 MiB 1.5 GiB 1.3 TiB 30.76 0.95 23 up 36 hdd 1.81999 1.00000 1.8 TiB 677 GiB 674 GiB 15 KiB 2.4 GiB 1.2 TiB 36.32 1.12 26 up 42 hdd 1.81940 1.00000 1.8 TiB 598 GiB 597 GiB 2 KiB 1.4 GiB 1.2 TiB 32.09 0.99 23 up 45 hdd 1.81999 1.00000 1.8 TiB 599 GiB 597 GiB 14 KiB 2.2 GiB 1.2 TiB 32.15 0.99 23 up 51 hdd 0.90970 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 destroyed 56 hdd 1.81940 1.00000 1.8 TiB 574 GiB 572 GiB 9 KiB 1.7 GiB 1.3 TiB 30.79 0.95 22 up 60 hdd 1.81940 1.00000 1.8 TiB 572 GiB 570 GiB 12 KiB 1.4 GiB 1.3 TiB 30.69 0.94 22 up 65 hdd 1.81940 1.00000 1.8 TiB 675 GiB 673 GiB 61 MiB 2.3 GiB 1.2 TiB 36.26 1.12 27 up 68 hdd 1.81940 1.00000 1.8 TiB 571 GiB 570 GiB 16 KiB 1.4 GiB 1.3 TiB 30.68 0.94 22 up 70 hdd 1.81940 1.00000 1.8 TiB 571 GiB 569 GiB 14 KiB 1.3 GiB 1.3 TiB 30.63 0.94 22 up 105 hdd 0.90700 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 destroyed TOTAL 320 TiB 104 TiB 104 TiB 2.9 GiB 378 GiB 216 TiB 32.51 MIN/MAX VAR: 0.82/1.12 STDDEV: 2.53 # ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 191543 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 150.00 pool 2 'wizard_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 191543 lfor 0/8092/8090 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 7.86 pool 3 'wizard_data' erasure profile k6_m2_host size 8 min_size 7 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 191543 lfor 0/0/41881 flags hashpspool,ec_overwrites stripe_width 24576 application cephfs - All the REWEIGHT entries of `ceph osd tree` are currently set to 1 (except for the two destroyed OSDs which are at 0) Thanks again, Nicola
First of all, thank you so much again for the time you spend in trying to help me, it's much appreciated.
Prego. In my book I assert that the community is a core Ceph component, and I tell people all the time that it’s one of many reasons to choose Ceph over alternatives. Like Red Green said, we’re all in this together.
Then: - here's the dump of the CRUSH rules:
Ok, that looks fine. If you had some rules specifying a device class but not others that would break the pg autoscaler.
- Running `ceph osd metadata | grep min_alloc_size` shows:
"bluestore_min_alloc_size": "4096"
for all the OSDs
Groovy. That this shows up is an indication that your Ceph is recent, I think that went out in Reef. Maybe Quincy.
- About PG count: currently I have two OSDs destroyed waiting for disk upgrade; anyway this doesn't change the overall picture. That said, here's the info you requested:
# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 4 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 53 MiB 3.4 GiB 1.2 TiB 32.22 0.99 24 up
Oh yeah, way too low.
11 TOTAL 320 TiB 104 TiB 104 TiB 2.9 GiB 378 GiB 216 TiB 32.51 MIN/MAX VAR: 0.82/1.12 STDDEV: 2.53
# ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 191543 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 150.00 pool 2 'wizard_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 191543 lfor 0/8092/8090 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 7.86 pool 3 'wizard_data' erasure profile k6_m2_host size 8 min_size 7 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 191543 lfor 0/0/41881 flags hashpspool,ec_overwrites stripe_width 24576 application cephfs
Ok. You have the autoscaler on for some pools but not others. I suggest setting mon_target_pg_per_osd to 200 then let’s see where we are when the dust settles.
- All the REWEIGHT entries of `ceph osd tree` are currently set to 1 (except for the two destroyed OSDs which are at 0)
Groovy.
Thanks again,
Nicola
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So you suggest to give this command: ceph config set global mon_target_pg_per_osd 200 right? If I understood the meaning of this parameter then it is meaningful when automated PG scaling is on, but it is currently off for the data partition: # ceph osd pool get wizard_data pg_autoscale_mode pg_autoscale_mode: off So should I proceed anyway? Sorry to bother you but I'm not sure I understood your suggestion and I fear I could make a mistake at this point.
I wouldn’t decrease mon_target_pg_per_osd below the default (250), Anthony is usually someone who recommends the opposite and wants to increase the default. So I’m not sure what exactly he’s aiming for… 😉 Zitat von Nicola Mori <mori@fi.infn.it>:
So you suggest to give this command:
ceph config set global 200
right? If I understood the meaning of this parameter then it is meaningful when automated PG scaling is on, but it is currently off for the data partition:
# ceph osd pool get wizard_data pg_autoscale_mode pg_autoscale_mode: off
So should I proceed anyway? Sorry to bother you but I'm not sure I understood your suggestion and I fear I could make a mistake at this point.
Default is 100, no?. I have a pr open to double it. The data pool disables the autoscaler so you would need to either enable it or increase pg_num manually
On Jan 3, 2025, at 11:03 AM, Eugen Block <eblock@nde.ag> wrote:
I wouldn’t decrease mon_target_pg_per_osd below the default (250), Anthony is usually someone who recommends the opposite and wants to increase the default. So I’m not sure what exactly he’s aiming for… 😉
Zitat von Nicola Mori <mori@fi.infn.it>:
So you suggest to give this command:
ceph config set global 200
right? If I understood the meaning of this parameter then it is meaningful when automated PG scaling is on, but it is currently off for the data partition:
# ceph osd pool get wizard_data pg_autoscale_mode pg_autoscale_mode: off
So should I proceed anyway? Sorry to bother you but I'm not sure I understood your suggestion and I fear I could make a mistake at this point.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Oh wait, I got confused, I thought you meant the max_pg_per_osd setting, please ignore my last comment. 😁 Zitat von Anthony D'Atri <anthony.datri@gmail.com>:
Default is 100, no?. I have a pr open to double it.
The data pool disables the autoscaler so you would need to either enable it or increase pg_num manually
On Jan 3, 2025, at 11:03 AM, Eugen Block <eblock@nde.ag> wrote:
I wouldn’t decrease mon_target_pg_per_osd below the default (250), Anthony is usually someone who recommends the opposite and wants to increase the default. So I’m not sure what exactly he’s aiming for… 😉
Zitat von Nicola Mori <mori@fi.infn.it>:
So you suggest to give this command:
ceph config set global 200
right? If I understood the meaning of this parameter then it is meaningful when automated PG scaling is on, but it is currently off for the data partition:
# ceph osd pool get wizard_data pg_autoscale_mode pg_autoscale_mode: off
So should I proceed anyway? Sorry to bother you but I'm not sure I understood your suggestion and I fear I could make a mistake at this point.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So you suggest to give this command:
ceph config set global mon_target_pg_per_osd 200
right? If I understood the meaning of this parameter then it is meaningful when automated PG scaling is on, but it is currently off for the data partition:
# ceph osd pool get wizard_data pg_autoscale_mode pg_autoscale_mode: off
So should I proceed anyway? Sorry to bother you but I'm not sure I understood your suggestion and I fear I could make a mistake at this point.
Yes. Had you turned off autoscaling for that pool so you could set it manually? I might suggest enabling or disabling for all pools, vs having a mix. If the data pool doesn’t get enough PGs with the autoscaler on, I might increase its BIAS value.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, I set disabled the autoscaler manually because I feared it could increase the count to much and that my old machines could go OOM. Anyway I issued `ceph config set global mon_target_pg_per_osd 200` but nothing happened. I guess this is expected given the autoscaler is off, Should I enable it for the data pool?
I would enable it, yes. Historically the autoscaler has arguably aimed too low not too high, and I think with BlueStore OSDs the OOM potential is very low. Did I suggest enabling the bulk flag for all pools (the .mgr pool doesn’t matter)? ceph osd pool set <pool-name> bulk true
On Jan 3, 2025, at 1:59 PM, Nicola Mori <mori@fi.infn.it> wrote:
Yes, I set disabled the autoscaler manually because I feared it could increase the count to much and that my old machines could go OOM.
Anyway I issued `ceph config set global mon_target_pg_per_osd 200` but nothing happened. I guess this is expected given the autoscaler is off, Should I enable it for the data pool?
Ok, will do it after upgrading the two disks of the currently destroyed OSDs on Tuesday. In the meantime let me ask another question, to better understand the situation. To the best of my knowledge, the autoscaler and the balancer are two different entities, the former taking care of setting the number off PGs and the latter of distributing them to the OSDs according to the CRUSH rules. If this is correct, then I'd say my issue is with the balancer and not with the autoscaler, unless the problem is that with too few PGs the balancer is unable to distribute them uniformly. Is this correct? If yes, then should I expect that after turning the autoscaler on the number of PGs will be increased and my problem hopefully resolved?
On Jan 4, 2025, at 10:56 AM, Nicola Mori <mori@fi.infn.it> wrote:
Ok, will do it after upgrading the two disks of the currently destroyed OSDs on Tuesday. In the meantime let me ask another question, to better understand the situation. To the best of my knowledge, the autoscaler and the balancer are two different entities, the former taking care of setting the number off PGs and the latter of distributing them to the OSDs according to the CRUSH rules. If this is correct, then I'd say my issue is with the balancer and not with the autoscaler, unless the problem is that with too few PGs the balancer is unable to distribute them uniformly. Is this correct? If yes, then should I expect that after turning the autoscaler on the number of PGs will be increased and my problem hopefully resolved?
Yes to all of the above. With too few PGs, the balancer can’t do a very uniform job. Think of an oil drum. Pack it full of basketballs, or golf balls.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, I did all you suggested: ceph osd pool set wizard_data pg_autoscale_mode on ceph config set global mon_target_pg_per_osd 200 ceph osd pool set wizard_metadata bulk true ceph osd pool set wizard_data bulk true but this didn't change anything. The PG count is unchanged and also the relative size of the filesystem which is still around 67%. Should I disable the autoscaler again and manually increase the PG count to 1024? Thank you, Nicola
I've been too impatient: after some minutes the autoscaler kicked in and now the situation is the following: # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 246.1M 3.0 323.8T 0.0000 1.0 1 on False wizard_metadata 1176M 3.0 323.8T 0.0000 4.0 16 16384 on True wizard_data 80443G 1.3333333730697632 323.8T 0.3235 1.0 2048 on True So it seems that the data pool is increasing the number of PGs from 512 to 2048 (currently there are 711 PG in total for the three pools). I'll report back after the backfill operations finish. Nicola
On Jan 8, 2025, at 4:32 AM, Nicola Mori <mori@fi.infn.it> wrote:
Hi Anthony,
I did all you suggested:
ceph osd pool set wizard_data pg_autoscale_mode on ceph config set global mon_target_pg_per_osd 200 ceph osd pool set wizard_metadata bulk true ceph osd pool set wizard_data bulk true
but this didn't change anything.
That’s weird. What does “autoscale status” show?
The PG count is unchanged and also the relative size of the filesystem which is still around 67%. Should I disable the autoscaler again and manually increase the PG count to 1024? Thank you,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Anthony, the autoscaler has finished its work, no OOM disaster happened, and the current situation is: # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 248.4M 3.0 323.8T 0.0000 1.0 1 on False wizard_metadata 1171M 3.0 323.8T 0.0000 4.0 16 16384 on True wizard_data 79648G 1.3333333730697632 323.8T 0.3203 1.0 2048 on True The min and max OSD occupancy are now much more similar: 30.34% and 33.66%, which seems reasonable if I correctly understand. The available space of the filesystem however is at 227 TiB which is just 70% of the 324 TiB raw space. It is quite improved but still far from the theoretical maximum of 75% for my 6+2 EC data pool. Neglecting the .mgr and metadata pools, I'd really like to recover the missing 5% which is 16 TB (i.e. about half host), is there something more that can be attempted? One more question: on the web UI keep receiving notifications like "PG autoscaler increasing pool 2 PGs from 16 to 16384", once per minute, but no change in PG number is actually happening. From the autoscale-status output posted above I see that for the metadata pool the value for NEW PG_NUM is 16384, but I don't understand what it means. Thank you, Nicola
On Jan 14, 2025, at 10:51 AM, Nicola Mori <mori@fi.infn.it> wrote:
Dear Anthony,
the autoscaler has finished its work, no OOM disaster happened
:D
, and the current situation is:
# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 248.4M 3.0 323.8T 0.0000 1.0 1 on False wizard_metadata 1171M 3.0 323.8T 0.0000 4.0 16 16384 on True wizard_data 79648G 1.3333333730697632 323.8T 0.3203 1.0 2048 on True
The min and max OSD occupancy are now much more similar: 30.34% and 33.66%, which seems reasonable if I correctly understand.
Yes, absolutely. You could squeeze that tighter if you try hard enough, but IMHO you’re probably fine where you are.
The available space of the filesystem however is at 227 TiB which is just 70% of the 324 TiB raw space. It is quite improved but still far from the theoretical maximum of 75% for my 6+2 EC data pool. Neglecting the .mgr and metadata pools, I'd really like to recover the missing 5% which is 16 TB (i.e. about half host), is there something more that can be attempted?
Those calculations are a bit subtle, I fear that I have no additional wisdom on a Monday for that gap.
One more question: on the web UI keep receiving notifications like "PG autoscaler increasing pool 2 PGs from 16 to 16384", once per minute, but no change in PG number is actually happening. From the autoscale-status output posted above I see that for the metadata pool the value for NEW PG_NUM is 16384, but I don't understand what it means.
Use the source, Luke ;) def update(self, module: MgrModule, progress: float) -> None: desc = 'increasing' if self.pg_num < self.pg_num_target else 'decreasing' module.remote('progress', 'update', self.ev_id, ev_msg="PG autoscaler %s pool %d PGs from %d to %d" % (desc, self.pool_id, self.pg_num, self.pg_num_target), ev_progress=progress, refs=[("pool", self.pool_id)]) I think NEW PG_NUM is the target and PG_NUM is the current value, though I wouldn’t expect the metadata pool to want to be that large. I would expect PG_NUM to be increasing. What does `ceph osd dump | grep pool` show for pg_num and pgp_num?
Thank you,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Here is it: # ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 191543 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 150.00 pool 2 'wizard_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 254279 lfor 0/8092/8090 flags hashpspool,bulk stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 7.86 pool 3 'wizard_data' erasure profile k6_m2_host size 8 min_size 7 crush_rule 1 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode on last_change 266071 lfor 0/0/265366 flags hashpspool,ec_overwrites,bulk stripe_width 24576 application cephfs According to the documentation: NEW PG_NUM (if present) is the value that the system recommends that the pg_num of the pool should be. It is always a power of two, and it is present only if the recommended value varies from the current value by more than the default factor of 3 So it's not the target, but just a recommendation if I correctly understand. I find it quite strange that 16384 PGs are recommended for a pool hosting ~ 1 GB of data. Nicola
Indeed I have no explanation for that. I personally would target pg_num for the metadata pool being the number of OSDs rounded to the next power of two. The way I believe this normally works is that the autoscaler bumps pg_num for a pool, and the mon/mgr gradually increases pgp_num to match.
On Jan 14, 2025, at 11:53 AM, Nicola Mori <mori@fi.infn.it> wrote:
Here is it:
# ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 191543 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 150.00 pool 2 'wizard_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 254279 lfor 0/8092/8090 flags hashpspool,bulk stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 7.86 pool 3 'wizard_data' erasure profile k6_m2_host size 8 min_size 7 crush_rule 1 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode on last_change 266071 lfor 0/0/265366 flags hashpspool,ec_overwrites,bulk stripe_width 24576 application cephfs
According to the documentation:
NEW PG_NUM (if present) is the value that the system recommends that the pg_num of the pool should be. It is always a power of two, and it is present only if the recommended value varies from the current value by more than the default factor of 3
So it's not the target, but just a recommendation if I correctly understand. I find it quite strange that 16384 PGs are recommended for a pool hosting ~ 1 GB of data.
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I removed the bulk flag for the metadata pool and the NEW PG_NUM has vanished. Now everything seems ok. Thank you very much again for your great help, Nicola
Interesting. I’ll have to experiment with that. Glad to have been of service. We’re a community!
On Jan 14, 2025, at 12:01 PM, Nicola Mori <mori@fi.infn.it> wrote:
I removed the bulk flag for the metadata pool and the NEW PG_NUM has vanished. Now everything seems ok. Thank you very much again for your great help,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
bulk is about tuning the pool assuming it will be closeish to full, not based on what utilization currently is. That way, as you add a bunch of data, it isn't constantly adding more pg's to the pool. ________________________________________ From: Anthony D'Atri <aad@dreamsnake.net> Sent: Tuesday, January 14, 2025 9:13 AM To: Nicola Mori Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Understanding filesystem size Check twice before you click! This email originated from outside PNNL. Interesting. I’ll have to experiment with that. Glad to have been of service. We’re a community!
On Jan 14, 2025, at 12:01 PM, Nicola Mori <mori@fi.infn.it> wrote:
I removed the bulk flag for the metadata pool and the NEW PG_NUM has vanished. Now everything seems ok. Thank you very much again for your great help,
Nicola _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Anthony D'Atri
-
Eugen Block
-
Fox, Kevin M
-
Nicola Mori