Disk Utilisation Graph
Hello All I need to understand what below graph expalining and which disk utilisation it is? I have 8 NVME and 16 SSD and their weight is 1 with 20% disk utilised as per osd df. If keeping the weight of NVME and reweight SSD to some .8 and .9 will worth anything to lower this graph utilisation? Below is the graph under each host performance on CEPH dashboard. Regards Dev
Would you consider increasing the pgs on your larger pool(s)? You might find things balance better. Then you could look at weighting to favor NVMe if needed. On Wed, Jul 2, 2025 at 10:54 AM, Devender Singh wrote: Hello All I need to understand what below graph expalining and which disk utilisation it is? I have 8 NVME and 16 SSD and their weight is 1 with 20% disk utilised as per osd df. If keeping the weight of NVME and reweight SSD to some .8 and .9 will worth anything to lower this graph utilisation? Below is the graph under each host performance on CEPH dashboard. RegardsDev
We need to see `ceph osd crush rule dump` and `ceph osd pool ls detail` to see which pools are using which CRUSH rule. Since there are two device classes, *every* pool should specify a CRUSH rule that constrains to one or the other device class. If this is not done, e.g. root@cmigsdsc-m18-33:~# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": “default” <——————————————==<<<< }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, then pools whose rule specifies an item_name “default” will be placed on both the “nvme” and “ssd” device classes. Which sort of works, but is usually not the best approach, and may result in the balancer and pg autoscaler not working properly if at all. Segregating the NVMe and SAS/SATA SSDs to separate pools is usually the better option since the former are usually faster. Below is an example CRUSH rule that will constrain pools specifying it to only the `nvme` device class. { "rule_id": 6, "rule_name": "ssd_nvme_replicated", "type": 1, "steps": [ { "op": "take", "item": -33, "item_name": “default~nvme" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, If the pools are using only one or the other device class, weighting to favor the NVMe SSDs as such isn’t in scope. Messing with CRUSH or legacy override reweights is more likely to just result in OSDs filling up prematurely. I’ve counseled the OP re PGs, but I think the underlying concern here is that “utilization” is being interpreted as two different things, one of which is not meaningfully represented for SSDs by that graph that was shared.
On Jul 2, 2025, at 11:59 AM, Peter Eisch <peter@boku.net> wrote:
Would you consider increasing the pgs on your larger pool(s)? You might find things balance better. Then you could look at weighting to favor NVMe if needed.
Hello Peter I hope you are not keeping images on EC Pool? In my case my all data and images majorly on ec pools Regards Dev On Wed, 2 Jul 2025 at 1:29 PM, Anthony D'Atri <aad@dreamsnake.net> wrote:
We need to see `ceph osd crush rule dump` and `ceph osd pool ls detail` to see which pools are using which CRUSH rule.
Since there are two device classes, *every* pool should specify a CRUSH rule that constrains to one or the other device class.
If this is not done, e.g.
root@cmigsdsc-m18-33:~# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": “default” <——————————————==<<<< }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] },
then pools whose rule specifies an item_name “default” will be placed on both the “nvme” and “ssd” device classes. Which sort of works, but is usually not the best approach, and may result in the balancer and pg autoscaler not working properly if at all. Segregating the NVMe and SAS/SATA SSDs to separate pools is usually the better option since the former are usually faster.
Below is an example CRUSH rule that will constrain pools specifying it to only the `nvme` device class.
{ "rule_id": 6, "rule_name": "ssd_nvme_replicated", "type": 1, "steps": [ { "op": "take", "item": -33, "item_name": “default~nvme" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] },
If the pools are using only one or the other device class, weighting to favor the NVMe SSDs as such isn’t in scope. Messing with CRUSH or legacy override reweights is more likely to just result in OSDs filling up prematurely.
I’ve counseled the OP re PGs, but I think the underlying concern here is that “utilization” is being interpreted as two different things, one of which is not meaningfully represented for SSDs by that graph that was shared.
On Jul 2, 2025, at 11:59 AM, Peter Eisch <peter@boku.net> wrote:
Would you consider increasing the pgs on your larger pool(s)? You might find things balance better. Then you could look at weighting to favor NVMe if needed.
Hi Devender, If I'm not wrong, that graph shows the disk utilization in terms of I/O, not the utilization in terms of capacity. Basically the value you get while doing iostat -x. https://github.com/ceph/ceph/blob/bf1302169e8a78630feed1cc84d30305b648889a/m... Best ----------------------------------------------------------------- Roberto VALVERDE CAMESELLE IT Storage And Data Management CERN, European Organization for Nuclear Research Esplanade des Particules 1, Geneve (Switzerland) ________________________________ From: Devender Singh <devender@netskrt.io> Sent: Wednesday, July 2, 2025 5:25 PM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Disk Utilisation Graph Hello All I need to understand what below graph expalining and which disk utilisation it is? I have 8 NVME and 16 SSD and their weight is 1 with 20% disk utilised as per osd df. If keeping the weight of NVME and reweight SSD to some .8 and .9 will worth anything to lower this graph utilisation? Below is the graph under each host performance on CEPH dashboard. [cid:ii_197cbbc465f21a975f91] Regards Dev
participants (4)
-
Anthony D'Atri
-
Devender Singh
-
Peter Eisch
-
Roberto Valverde Cameselle