squid 19.2.2 - osd_memory_target_autotune - best practices when host has lots of RAM
Hi What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ? My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD Should I disable autotune and allocate more RAM? I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD Many thanks Steven
IMHO the autotuner is awesome. 1TB of RAM is an embarrassment of riches -- are these hosts perhaps converged compute+storage?
On Jul 31, 2025, at 10:17 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi
What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ?
My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD
Remember that NVMe devices *are* SSDs ;) I'm guessing those are used for WAL+DB offload, and thus you have 24x OSDs per host?
Should I disable autotune and allocate more RAM?
The autotuner by default will divide 70% of physmem across all the OSDs it finds on a given host, with 30% allocated for the OS and other daemons. I *think* any RGWs, mons, etc. are assumed to be part of that 30% but am not positive.
I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD
I personally have a growing sense that more RAM actually can help slower OSDs more, at least with respect to rebalancing without rampant slow ops. ymmv. This implies that your NVMe devices are standalone OSDs, so that would mean 27 OSDs per node? I'm curious what manner of chassis this is. I then would think that the autotuner would set ~~ 26TB to osd_memory_size, which is ample by any measure. ~307TB will be available for non-OSD processes. If you're running compute or other significant non-Ceph workloads on the same nodes, you can adjust the reservation factor by setting ceph config set mgr mgr/cephadm/autotune_memory_target_ratio xxx. So if you want to reserve less for non-OSD processes, something like ceph config set mgr mgr/cephadm/autotune_memory_target_ratio 0.1 If yo do have hungry compute colocated, a good value might be something like 0.25, which would give each OSD > 9GB for osd_memory_target. If you do want to allot different amounts to different device classes, you can instead set static values, using central config device class masks.
Many thanks Steven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, The servers are dedicated to CEPH Yes, it is perhaps too much but my IT philosophy is "there is always room for more RAM" as it usually helps running things faster Now, since I have it, I would like to use it as efficiently as possible The 3 NVMEs are 15TB dedicated to OSD - there are 2 more 1.6TB dedicated to DB/WAL HDD are 20TB and SSD are 7TB Is my understanding correct that autotune will dedicated 70% to OSDs indiscriminately ??? ... or there is some sort of algorithm for differentiating between the disk type and size ? If NVME is SSD from autotune perspective, it would probably make sense to tune it manually , no ? How would I check status of autotune ...other than checking individual OSD config ? Many thanks Steven On Thu, 31 Jul 2025 at 10:43, Anthony D'Atri <aad@dreamsnake.net> wrote:
IMHO the autotuner is awesome.
1TB of RAM is an embarrassment of riches -- are these hosts perhaps converged compute+storage?
On Jul 31, 2025, at 10:17 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi
What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ?
My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD
Remember that NVMe devices *are* SSDs ;) I'm guessing those are used for WAL+DB offload, and thus you have 24x OSDs per host?
Should I disable autotune and allocate more RAM?
The autotuner by default will divide 70% of physmem across all the OSDs it finds on a given host, with 30% allocated for the OS and other daemons. I *think* any RGWs, mons, etc. are assumed to be part of that 30% but am not positive.
I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD
I personally have a growing sense that more RAM actually can help slower OSDs more, at least with respect to rebalancing without rampant slow ops. ymmv.
This implies that your NVMe devices are standalone OSDs, so that would mean 27 OSDs per node? I'm curious what manner of chassis this is.
I then would think that the autotuner would set ~~ 26TB to osd_memory_size, which is ample by any measure. ~307TB will be available for non-OSD processes.
If you're running compute or other significant non-Ceph workloads on the same nodes, you can adjust the reservation factor by setting ceph config set mgr mgr/cephadm/autotune_memory_target_ratio xxx. So if you want to reserve less for non-OSD processes, something like
ceph config set mgr mgr/cephadm/autotune_memory_target_ratio 0.1
If yo do have hungry compute colocated, a good value might be something like 0.25, which would give each OSD > 9GB for osd_memory_target. If you do want to allot different amounts to different device classes, you can instead set static values, using central config device class masks.
Many thanks Steven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The servers are dedicated to Ceph Yes, it is perhaps too much but my IT philosophy is "there is always room for more RAM" as it usually helps running things faster
Unless you're a certain Sun model, but I digress... The $ spent on all that RAM would IMHO have been more effective choosing NVMe SSDs instead of HDDs. And not having to pay for the HBA.
Now, since I have it, I would like to use it as efficiently as possible
That's what the autotuner is all about.
The 3 NVMEs are 15TB dedicated to OSD - there are 2 more 1.6TB dedicated to DB/WAL HDD are 20TB and SSD are 7TB
Is my understanding correct that autotune will dedicated 70% to OSDs indiscriminately ??? ... or there is some sort of algorithm for differentiating between the disk type and size ?
There is not as far as I know, but honestly you're dramatically into the territory where you have so much that there wouldn't be much to be gained by customizing. Diminishing returns.
If NVME is SSD
NVMe is an interface not a medium. A SATA SSD and an NVMe SSD are the same NAND (at similar cost) with a different interface.
from autotune perspective, it would probably make sense to tune it manually , no ?
If you like. See the below page for setting manually on a host-by-host or per-OSD basis. Or disable it and do the math to divide it by device class as you like, though note that if you add or remove OSDs from a given host the system won't adjust without intervention. Or if you add systems less expensively with less RAM and move some from these to them to even it out. Since you don't have much else contending for that RAM, you might ceph config set mgr/cephadm/autotune_memory_target_ratio 0.930000 which will let the autotuner use even more for OSDs.
How would I check status of autotune ...other than checking individual OSD config ?
# ceph config dump | grep osd_memory_target osd host:cephab92 basic osd_memory_target 12083522051 osd host:cephac0f basic osd_memory_target 12083519715 osd host:dd13-25 basic osd_memory_target 6156072793 osd host:dd13-29 basic osd_memory_target 6235526712 osd host:dd13-33 basic osd_memory_target 6780274813 osd host:dd13-37 basic osd_memory_target 6601357610 osd host:i18-24 basic osd_memory_target 6670077087 osd host:i18-28 basic osd_memory_target 6600879861 osd host:k18-23 basic osd_memory_target 6663117330 osd host:l18-24 basic osd_memory_target 6822190406 osd host:l18-28 basic osd_memory_target 6782421978 osd host:m18-33 basic osd_memory_target 6593523272 osd advanced osd_memory_target_autotune true Here the first two hosts have much more RAM than the others, so the autotuner has more to distribute. You can game the autotuner in various ways, see https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=osds-automatically-tuni... https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#sections-and-...
Many thanks
Steven
On Thu, 31 Jul 2025 at 10:43, Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> wrote:
IMHO the autotuner is awesome.
1TB of RAM is an embarrassment of riches -- are these hosts perhaps converged compute+storage?
On Jul 31, 2025, at 10:17 AM, Steven Vacaroaia <stef97@gmail.com <mailto:stef97@gmail.com>> wrote:
Hi
What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ?
My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD
Remember that NVMe devices *are* SSDs ;) I'm guessing those are used for WAL+DB offload, and thus you have 24x OSDs per host?
Should I disable autotune and allocate more RAM?
The autotuner by default will divide 70% of physmem across all the OSDs it finds on a given host, with 30% allocated for the OS and other daemons. I *think* any RGWs, mons, etc. are assumed to be part of that 30% but am not positive.
I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD
I personally have a growing sense that more RAM actually can help slower OSDs more, at least with respect to rebalancing without rampant slow ops. ymmv.
This implies that your NVMe devices are standalone OSDs, so that would mean 27 OSDs per node? I'm curious what manner of chassis this is.
I then would think that the autotuner would set ~~ 26TB to osd_memory_size, which is ample by any measure. ~307TB will be available for non-OSD processes.
If you're running compute or other significant non-Ceph workloads on the same nodes, you can adjust the reservation factor by setting ceph config set mgr mgr/cephadm/autotune_memory_target_ratio xxx. So if you want to reserve less for non-OSD processes, something like
ceph config set mgr mgr/cephadm/autotune_memory_target_ratio 0.1
If yo do have hungry compute colocated, a good value might be something like 0.25, which would give each OSD > 9GB for osd_memory_target. If you do want to allot different amounts to different device classes, you can instead set static values, using central config device class masks.
Many thanks Steven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Excellent info Anthony Many thanks Steven On Fri, 1 Aug 2025 at 09:29, Anthony D'Atri <aad@dreamsnake.net> wrote:
The servers are dedicated to Ceph Yes, it is perhaps too much but my IT philosophy is "there is always room for more RAM" as it usually helps running things faster
Unless you're a certain Sun model, but I digress...
The $ spent on all that RAM would IMHO have been more effective choosing NVMe SSDs instead of HDDs. And not having to pay for the HBA.
Now, since I have it, I would like to use it as efficiently as possible
That's what the autotuner is all about.
The 3 NVMEs are 15TB dedicated to OSD - there are 2 more 1.6TB dedicated to DB/WAL HDD are 20TB and SSD are 7TB
Is my understanding correct that autotune will dedicated 70% to OSDs indiscriminately ??? ... or there is some sort of algorithm for differentiating between the disk type and size ?
There is not as far as I know, but honestly you're dramatically into the territory where you have so much that there wouldn't be much to be gained by customizing. Diminishing returns.
If NVME is SSD
NVMe is an interface not a medium. A SATA SSD and an NVMe SSD are the same NAND (at similar cost) with a different interface.
from autotune perspective, it would probably make sense to tune it manually , no ?
If you like. See the below page for setting manually on a host-by-host or per-OSD basis. Or disable it and do the math to divide it by device class as you like, though note that if you add or remove OSDs from a given host the system won't adjust without intervention. Or if you add systems less expensively with less RAM and move some from these to them to even it out.
Since you don't have much else contending for that RAM, you might
ceph config set mgr/cephadm/autotune_memory_target_ratio 0.930000
which will let the autotuner use even more for OSDs.
How would I check status of autotune ...other than checking individual OSD config ?
# ceph config dump | grep osd_memory_target osd host:cephab92 basic osd_memory_target 12083522051 osd host:cephac0f basic osd_memory_target 12083519715 osd host:dd13-25 basic osd_memory_target 6156072793 osd host:dd13-29 basic osd_memory_target 6235526712 osd host:dd13-33 basic osd_memory_target 6780274813 osd host:dd13-37 basic osd_memory_target 6601357610 osd host:i18-24 basic osd_memory_target 6670077087 osd host:i18-28 basic osd_memory_target 6600879861 osd host:k18-23 basic osd_memory_target 6663117330 osd host:l18-24 basic osd_memory_target 6822190406 osd host:l18-28 basic osd_memory_target 6782421978 osd host:m18-33 basic osd_memory_target 6593523272 osd advanced osd_memory_target_autotune true
Here the first two hosts have much more RAM than the others, so the autotuner has more to distribute.
You can game the autotuner in various ways, see https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=osds-automatically-tuni...
https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#sections-and-...
Many thanks
Steven
On Thu, 31 Jul 2025 at 10:43, Anthony D'Atri <aad@dreamsnake.net> wrote:
IMHO the autotuner is awesome.
1TB of RAM is an embarrassment of riches -- are these hosts perhaps converged compute+storage?
On Jul 31, 2025, at 10:17 AM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi
What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ?
My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD
Remember that NVMe devices *are* SSDs ;) I'm guessing those are used for WAL+DB offload, and thus you have 24x OSDs per host?
Should I disable autotune and allocate more RAM?
The autotuner by default will divide 70% of physmem across all the OSDs it finds on a given host, with 30% allocated for the OS and other daemons. I *think* any RGWs, mons, etc. are assumed to be part of that 30% but am not positive.
I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD
I personally have a growing sense that more RAM actually can help slower OSDs more, at least with respect to rebalancing without rampant slow ops. ymmv.
This implies that your NVMe devices are standalone OSDs, so that would mean 27 OSDs per node? I'm curious what manner of chassis this is.
I then would think that the autotuner would set ~~ 26TB to osd_memory_size, which is ample by any measure. ~307TB will be available for non-OSD processes.
If you're running compute or other significant non-Ceph workloads on the same nodes, you can adjust the reservation factor by setting ceph config set mgr mgr/cephadm/autotune_memory_target_ratio xxx. So if you want to reserve less for non-OSD processes, something like
ceph config set mgr mgr/cephadm/autotune_memory_target_ratio 0.1
If yo do have hungry compute colocated, a good value might be something like 0.25, which would give each OSD > 9GB for osd_memory_target. If you do want to allot different amounts to different device classes, you can instead set static values, using central config device class masks.
Many thanks Steven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Steven, It really depends on how large the active dataset size is that's being accessed across all of the clients. The biggest consumer of memory in the OSD typically is the onode cache. Reading onodes from disk is expensive, so keeping the onode cache hit rate high (especially on NVMe) is a pretty big performance win. The suggestions you mentioned are probably fairly reasonable as an initial rule-of-thumb, but you may want to scale up/down depending on various factors. IE 8GB memory targets might be fine for NVMe backed OSDs if the drives are smaller and if most of the data being stored is cold. On the other hand, you might want more than 6GB per OSD if you have big HDDs with many active users all storing tiny objects. Mark On 7/31/25 7:17 AM, Steven Vacaroaia wrote:
Hi
What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ?
My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD Should I disable autotune and allocate more RAM?
I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD
Many thanks Steven _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
Mark Nelson
-
Steven Vacaroaia