How does mclock work?
Hello, Could someone please explain how mclock works regarding reads and writes? Does mclock intervene on both read and write iops? Or only on reads or only on writes? And what type of underlying hardware performance is calculated and considered by mclock? Seems to be only write performance. The mclock documentation shows HDDs and SSDs specific configuration options (capacity and sequential bandwidth) but nothing regarding hybrid setups and these configuration options do not distinguish reads and writes. But read and write performance are often not in par for a single drive and even less when using hybrid setups. With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads. With HDD only setups (RocksDB+WAL+Data on HDD), if mclock only considers write performance, the OSD may not take advantage of higher read performance. Can someone please shed some light on this? Best regards, Frédéric Nass Sous-direction Infrastructures et Services Direction du Numérique Université de Lorraine Tél : +33 3 72 74 11 35
Not strictly an answer to your worthy question, but IMHO this supports my stance that hybrid OSDs aren't worth the hassle.
On Jan 9, 2024, at 06:13, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads.
With HDDs and a lot of metadata, it's tough to get away from it imho. In an alternate universe it would have been really neat if Intel could have worked with the HDD vendors to put like 16GB of user accessible optane on every HDD. Enough for the WAL and L0 (and maybe L1). Mark On 1/9/24 08:53, Anthony D'Atri wrote:
Not strictly an answer to your worthy question, but IMHO this supports my stance that hybrid OSDs aren't worth the hassle.
On Jan 9, 2024, at 06:13, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There was a client SSD sorta like that, a bit of Optane with TLC or QLC, but it didn't seem to sell well. Optane was groovy tech, but with certain challenges as well.
On Jan 9, 2024, at 14:30, Mark Nelson <mark.a.nelson@gmail.com> wrote:
With HDDs and a lot of metadata, it's tough to get away from it imho. In an alternate universe it would have been really neat if Intel could have worked with the HDD vendors to put like 16GB of user accessible optane on every HDD. Enough for the WAL and L0 (and maybe L1).
Mark
On 1/9/24 08:53, Anthony D'Atri wrote:
Not strictly an answer to your worthy question, but IMHO this supports my stance that hybrid OSDs aren't worth the hassle.
On Jan 9, 2024, at 06:13, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frédéric, Please see answers below.
Could someone please explain how mclock works regarding reads and writes? Does mclock intervene on both read and write iops? Or only on reads or only on writes?
mClock schedules both read and write ops.
And what type of underlying hardware performance is calculated and considered by mclock? Seems to be only write performance.
Random write performance is considered for setting the maximum IOPS capacity of an OSD. This along with the sequential bandwidth capability of the OSD is used to calculate the cost per IO that is internally used by mClock for scheduling Ops. In addition, the mClock profiles use the capacity information to allocate reservation and limit for different classes of service (for e.g., client, background-recovery, scrub, snaptrim etc.). The write performance is used to set a lower bound on the amount of bandwidth to be allocated for different classes of services. For e.g., the 'balanced' profile allocates 50% of the OSD's IOPS capacity to cllent ops. In other words, a minimum guarantee of 50% of the OSD's bandwidth is allocated to client ops (read or write). If you look at the 'balanced' profile, there is no upper limit set for client ops (i.e. set to MAX) which means that reads can potentially use the maximum possible bandwidth (i.e., not contrained by max IOPS capacity) if there are no other competing ops. Please see https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/#built-i... for more information about mClock profiles.
The mclock documentation shows HDDs and SSDs specific configuration options (capacity and sequential bandwidth) but nothing regarding hybrid setups and these configuration options do not distinguish reads and writes. But read and write performance are often not in par for a single drive and even less when using hybrid setups.
With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads.
With HDD only setups (RocksDB+WAL+Data on HDD), if mclock only considers write performance, the OSD may not take advantage of higher read performance.
Can someone please shed some light on this?
As mentioned above, as long as there are no competing ops, the mClock profiles ensure that there is nothing constraining client ops from using the full available bandwidth of an OSD for both reads and writes regardless of the type of setup (hybrid, HDD, SSD) being employed. The important aspect is to ensure that the set IOPS capacity for the OSD reflects a fairly accurate representation of the underlying device capability. This is because the reservation criteria based the IOPS capacity helps maintain an acceptable level of performance with other active competing ops. You could run some synthetic benchmarks to ensure that read and write performance are along expected lines with the default mClock profile to confirm the above. I hope this helps. -Sridhar
Sridhar, Thanks a lot for this explantation. It's clearer now. So at the end of the day (at least with balanced profile) it's a lower bound and no upper limit and a balanced distribution between client and cluster IOPS. Regards, Frédéric. -----Message original----- De: Sridhar <sseshasa@redhat.com> à: Frédéric <frederic.nass@univ-lorraine.fr> Cc: ceph-users <ceph-users@ceph.com> Envoyé: mercredi 10 janvier 2024 08:15 CET Sujet : Re: [ceph-users] How does mclock work? Hello Frédéric, Please see answers below. Could someone please explain how mclock works regarding reads and writes? Does mclock intervene on both read and write iops? Or only on reads or only on writes? mClock schedules both read and write ops. And what type of underlying hardware performance is calculated and considered by mclock? Seems to be only write performance. Random write performance is considered for setting the maximum IOPS capacity of an OSD. This along with the sequential bandwidth capability of the OSD is used to calculate the cost per IO that is internally used by mClock for scheduling Ops. In addition, the mClock profiles use the capacity information to allocate reservation and limit for different classes of service (for e.g., client, background-recovery, scrub, snaptrim etc.). The write performance is used to set a lower bound on the amount of bandwidth to be allocated for different classes of services. For e.g., the 'balanced' profile allocates 50% of the OSD's IOPS capacity to cllent ops. In other words, a minimum guarantee of 50% of the OSD's bandwidth is allocated to client ops (read or write). If you look at the 'balanced' profile, there is no upper limit set for client ops (i.e. set to MAX) which means that reads can potentially use the maximum possible bandwidth (i.e., not contrained by max IOPS capacity) if there are no other competing ops. Please see https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/#built-i... for more information about mClock profiles. The mclock documentation shows HDDs and SSDs specific configuration options (capacity and sequential bandwidth) but nothing regarding hybrid setups and these configuration options do not distinguish reads and writes. But read and write performance are often not in par for a single drive and even less when using hybrid setups. With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads. With HDD only setups (RocksDB+WAL+Data on HDD), if mclock only considers write performance, the OSD may not take advantage of higher read performance. Can someone please shed some light on this? As mentioned above, as long as there are no competing ops, the mClock profiles ensure that there is nothing constraining client ops from using the full available bandwidth of an OSD for both reads and writes regardless of the type of setup (hybrid, HDD, SSD) being employed. The important aspect is to ensure that the set IOPS capacity for the OSD reflects a fairly accurate representation of the underlying device capability. This is because the reservation criteria based the IOPS capacity helps maintain an acceptable level of performance with other active competing ops. You could run some synthetic benchmarks to ensure that read and write performance are along expected lines with the default mClock profile to confirm the above. I hope this helps. -Sridhar
Hi folks. I had a quick search but found nothing concrete on this so thought I would ask. We currently have a 4 host CEPH cluster with an NVMe pool (1 OSD per host) and an HDD Pool (1 OSD per host). Both OSD's use a separate NVMe for DB/WAL. These machines are identical (Homogenous) and are Ryzen 7 5800X machines with 64GB DDR3200 RAM. The NVMe's are 1TB Seagate Ironwolfs and the HDD's are 16TB Seagate IronWolfs. We are wanting to add more nodes mainly for capacity and resilience reasons. We have an old 3 node cluster of Dell R740 servers that could be added to this CEPH cluster. Instead of DDR4, they use DDR3 (although 1.5TB each!!). and instead of Ryzen 7 5800X CPUs they use old Intel Xeon CPU E5-4657L v2 (96 cores at 2.4Ghz). What would be the performance impact of adding these three nodes with the same OSD layout (i.e 1NVMe OSD and 1 HDD OSD per host with 1x NVMe DB/WAL NVMe) Would we get overall better performance or worse? Can weighting be used to mitigate performance penalties and if so is this easy to configure? On performance, I would deem it Ok for our use case currently (VM disks), as we are running on 10Gbe network (with dedicated NICs for public and cluster network). Many thanks in advance Tino This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.
On 17.01.24 11:13, Tino Todino wrote:
Hi folks.
I had a quick search but found nothing concrete on this so thought I would ask.
We currently have a 4 host CEPH cluster with an NVMe pool (1 OSD per host) and an HDD Pool (1 OSD per host). Both OSD's use a separate NVMe for DB/WAL. These machines are identical (Homogenous) and are Ryzen 7 5800X machines with 64GB DDR3200 RAM. The NVMe's are 1TB Seagate Ironwolfs and the HDD's are 16TB Seagate IronWolfs.
We are wanting to add more nodes mainly for capacity and resilience reasons. We have an old 3 node cluster of Dell R740 servers that could be added to this CEPH cluster. Instead of DDR4, they use DDR3 (although 1.5TB each!!). and instead of Ryzen 7 5800X CPUs they use old Intel Xeon CPU E5-4657L v2 (96 cores at 2.4Ghz).
What would be the performance impact of adding these three nodes with the same OSD layout (i.e 1NVMe OSD and 1 HDD OSD per host with 1x NVMe DB/WAL NVMe) Would we get overall better performance or worse? Can weighting be used to mitigate performance penalties and if so is this easy to configure?
What will happen is that Ceph will distribute PGs across your cluster uniformly by default, so for some requests PGs from the Ryzen nodes will answer, for others the Xeons. Presumably the Xeons will be slower based on slower clock speed. The net effect will be jitter in completion latencies -- it depends on your workloads if they're fine with it or not. Note the latencies are sensitive to clock speeds; the huge amount or RAM on the Xeons won't help much for OSDs, and as your cluster is small MONs/MGRs won't need that much either. Check out https://docs.ceph.com/en/reef/start/hardware-recommendations/ and see the section on memory for tuning recommendations. You can influence distribution by weighting but this will only influence the rate of jitter not the magnitude (I mean if you weight down the Xeons to 0 the jitter will cease but that's not very useful :-)). One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-performance-comparison...
On performance, I would deem it Ok for our use case currently (VM disks), as we are running on 10Gbe network (with dedicated NICs for public and cluster network).
Many thanks in advance
Tino This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-performance-comparison...
It's a little tricky. In the upstream lab we don't strictly see an IOPS or average latency advantage with heavy parallelism by running muliple OSDs per NVMe drive until per-OSD core counts get very high. There does seem to be a fairly consistent tail latency advantage even at moderately low core counts however. Results are here: https://ceph.io/en/news/blog/2023/reef-osds-per-nvme/ Specifically for jitter, there is probably an advantage to using 2 cores per OSD unless you are very CPU starved, but how much that actually helps in practice for a typical production workload is questionable imho. You do pay some overhead for running 2 OSDs per NVMe as well. Mark On 1/17/24 12:24, Anthony D'Atri wrote:
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-performance-comparison...
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Very informative article you did Mark. IMHO if you find yourself with very high per-OSD core count, it may be logical to just pack/add more nvmes per host, you'd be getting the best price per performance and capacity. /Maged On 17/01/2024 22:00, Mark Nelson wrote:
It's a little tricky. In the upstream lab we don't strictly see an IOPS or average latency advantage with heavy parallelism by running muliple OSDs per NVMe drive until per-OSD core counts get very high. There does seem to be a fairly consistent tail latency advantage even at moderately low core counts however. Results are here:
https://ceph.io/en/news/blog/2023/reef-osds-per-nvme/
Specifically for jitter, there is probably an advantage to using 2 cores per OSD unless you are very CPU starved, but how much that actually helps in practice for a typical production workload is questionable imho. You do pay some overhead for running 2 OSDs per NVMe as well.
Mark
On 1/17/24 12:24, Anthony D'Atri wrote:
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-performance-comparison...
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
+1 to this, great article and great research. Something we've been keeping a very close eye on ourselves. Overall we've mostly settled on the old keep it simple stupid methodology with good results. Especially as the benefits have gotten less beneficial the more recent your ceph version, and have been rocking with single OSD/NVMe, but as always everything is workload dependant and there is sometimes a need for doubling up 😊 Regards, Bailey
-----Original Message----- From: Maged Mokhtar <mmokhtar@petasan.org> Sent: January 17, 2024 4:59 PM To: Mark Nelson <mark.nelson@clyso.com>; ceph-users@ceph.io Subject: [ceph-users] Re: Performance impact of Heterogeneous environment
Very informative article you did Mark.
IMHO if you find yourself with very high per-OSD core count, it may be logical to just pack/add more nvmes per host, you'd be getting the best price per performance and capacity.
/Maged
It's a little tricky. In the upstream lab we don't strictly see an IOPS or average latency advantage with heavy parallelism by running muliple OSDs per NVMe drive until per-OSD core counts get very high. There does seem to be a fairly consistent tail latency advantage even at moderately low core counts however. Results are here:
https://ceph.io/en/news/blog/2023/reef-osds-per-nvme/
Specifically for jitter, there is probably an advantage to using 2 cores per OSD unless you are very CPU starved, but how much that actually helps in practice for a typical production workload is questionable imho. You do pay some overhead for running 2 OSDs per NVMe as well.
Mark
On 1/17/24 12:24, Anthony D'Atri wrote:
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-
On 17/01/2024 22:00, Mark Nelson wrote: performance-co
mparison/
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks kindly Maged/Bailey! As always it's a bit of a moving target. New hardware comes out that reveals bottlenecks in our code. Doubling up the OSDs sometimes improves things. We figure out how to make the OSDs faster and the old assumptions stop being correct. Even newer hardware comes out, etc etc. Mark On 1/17/24 17:36, Bailey Allison wrote:
+1 to this, great article and great research. Something we've been keeping a very close eye on ourselves.
Overall we've mostly settled on the old keep it simple stupid methodology with good results. Especially as the benefits have gotten less beneficial the more recent your ceph version, and have been rocking with single OSD/NVMe, but as always everything is workload dependant and there is sometimes a need for doubling up 😊
Regards,
Bailey
-----Original Message----- From: Maged Mokhtar <mmokhtar@petasan.org> Sent: January 17, 2024 4:59 PM To: Mark Nelson <mark.nelson@clyso.com>; ceph-users@ceph.io Subject: [ceph-users] Re: Performance impact of Heterogeneous environment
Very informative article you did Mark.
IMHO if you find yourself with very high per-OSD core count, it may be logical to just pack/add more nvmes per host, you'd be getting the best price per performance and capacity.
/Maged
It's a little tricky. In the upstream lab we don't strictly see an IOPS or average latency advantage with heavy parallelism by running muliple OSDs per NVMe drive until per-OSD core counts get very high. There does seem to be a fairly consistent tail latency advantage even at moderately low core counts however. Results are here:
https://ceph.io/en/news/blog/2023/reef-osds-per-nvme/
Specifically for jitter, there is probably an advantage to using 2 cores per OSD unless you are very CPU starved, but how much that actually helps in practice for a typical production workload is questionable imho. You do pay some overhead for running 2 OSDs per NVMe as well.
Mark
On 1/17/24 12:24, Anthony D'Atri wrote:
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-
On 17/01/2024 22:00, Mark Nelson wrote: performance-co
mparison/
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
For multi- vs. single-OSD per flash drive decision the following test might be useful: We found dramatic improvements using multiple OSDs per flash drive with octopus *if* the bottleneck is the kv_sync_thread. Apparently, each OSD has only one and this thread is effectively sequentializing otherwise async IO if saturated. There was a dev discussion about having more kv_sync_threads per OSD daemon by splitting up rocks-dbs for PGs, but I don't know if this ever materialized. My guess is that for good NVMe drives it is possible that a single kv_sync_thread can saturate the device and there will be no advantage of having more OSDs/device. On not so good drives (SATA/SAS flash) multi-OSD deployments usually are better, because the on-disk controller requires concurrency to saturate the drive. Its not possible to saturate usual SAS-/SATA- SSDs with iodepth=1. With good NVME drives I have seen fio-tests with direct IO saturate the drive with 4K random IO and iodepth=1. You need enough PCI-lanes per drive for that and I could imagine that here 1 OSD/drive is sufficient. For such drives, storage access quickly becomes CPU bound, so some benchmarking taking all system properties into account is required. If you are already CPU bound (too many NVMe drives per core, many standard servers with 24+ NVMe drives have that property) there is no point adding extra CPU load with more OSD daemons. Don't just look at single disks, look at the whole system. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Thursday, January 18, 2024 12:36 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Performance impact of Heterogeneous environment +1 to this, great article and great research. Something we've been keeping a very close eye on ourselves. Overall we've mostly settled on the old keep it simple stupid methodology with good results. Especially as the benefits have gotten less beneficial the more recent your ceph version, and have been rocking with single OSD/NVMe, but as always everything is workload dependant and there is sometimes a need for doubling up 😊 Regards, Bailey
-----Original Message----- From: Maged Mokhtar <mmokhtar@petasan.org> Sent: January 17, 2024 4:59 PM To: Mark Nelson <mark.nelson@clyso.com>; ceph-users@ceph.io Subject: [ceph-users] Re: Performance impact of Heterogeneous environment
Very informative article you did Mark.
IMHO if you find yourself with very high per-OSD core count, it may be logical to just pack/add more nvmes per host, you'd be getting the best price per performance and capacity.
/Maged
It's a little tricky. In the upstream lab we don't strictly see an IOPS or average latency advantage with heavy parallelism by running muliple OSDs per NVMe drive until per-OSD core counts get very high. There does seem to be a fairly consistent tail latency advantage even at moderately low core counts however. Results are here:
https://ceph.io/en/news/blog/2023/reef-osds-per-nvme/
Specifically for jitter, there is probably an advantage to using 2 cores per OSD unless you are very CPU starved, but how much that actually helps in practice for a typical production workload is questionable imho. You do pay some overhead for running 2 OSDs per NVMe as well.
Mark
On 1/17/24 12:24, Anthony D'Atri wrote:
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-
On 17/01/2024 22:00, Mark Nelson wrote: performance-co
mparison/
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 1/18/24 03:40, Frank Schilder wrote:
For multi- vs. single-OSD per flash drive decision the following test might be useful:
We found dramatic improvements using multiple OSDs per flash drive with octopus *if* the bottleneck is the kv_sync_thread. Apparently, each OSD has only one and this thread is effectively sequentializing otherwise async IO if saturated.
There was a dev discussion about having more kv_sync_threads per OSD daemon by splitting up rocks-dbs for PGs, but I don't know if this ever materialized.
I advocated for it at one point, but Sage was pretty concerned about how much we'd be disturbing Bluestore's write path. Instead, Adam ended up implementing the column family sharding inside RocksDB which got us some (but not all) of the benefit. A lot of the work that has gone into refactoring the RocksDB settings in Reef has been to help mitigate some of the overhead in the kv sync thread. The gist of it is that we are trying to balance keeping the memtables large enough to avoid letting short lived items like pg log entries from regularly leaking into the DB, while simultaneously keeping the memtables as small as possible to reduce the number of comparisons RocksDB needs to do to keep them in sorted order during inserts (which is a significant overhead in the kv sync thread during heavy small random writes). This is also part of the reason that Igor was experimenting with implementing a native bluestore WAL rather than relying on the one in RocksDB.
My guess is that for good NVMe drives it is possible that a single kv_sync_thread can saturate the device and there will be no advantage of having more OSDs/device. On not so good drives (SATA/SAS flash) multi-OSD deployments usually are better, because the on-disk controller requires concurrency to saturate the drive. Its not possible to saturate usual SAS-/SATA- SSDs with iodepth=1.
Oddly enough, we do see some effect on small random reads. The way that the async msgr / shards / threads interact doesn't scale well past about 14-16 cpu threads (and increasing the shard/thread counts has complicated effects, it may not always help). If you look at that 1 vs 2 NVMe article on the ceph.io page, you'll see that once you hit the 14 CPU threads the 2 OSD/NVMe configuration keeps scaling but the single OSD configuration tops out.
With good NVME drives I have seen fio-tests with direct IO saturate the drive with 4K random IO and iodepth=1. You need enough PCI-lanes per drive for that and I could imagine that here 1 OSD/drive is sufficient. For such drives, storage access quickly becomes CPU bound, so some benchmarking taking all system properties into account is required. If you are already CPU bound (too many NVMe drives per core, many standard servers with 24+ NVMe drives have that property) there is no point adding extra CPU load with more OSD daemons.
Don't just look at single disks, look at the whole system.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Bailey Allison <ballison@45drives.com> Sent: Thursday, January 18, 2024 12:36 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Performance impact of Heterogeneous environment
+1 to this, great article and great research. Something we've been keeping a very close eye on ourselves.
Overall we've mostly settled on the old keep it simple stupid methodology with good results. Especially as the benefits have gotten less beneficial the more recent your ceph version, and have been rocking with single OSD/NVMe, but as always everything is workload dependant and there is sometimes a need for doubling up 😊
Regards,
Bailey
-----Original Message----- From: Maged Mokhtar <mmokhtar@petasan.org> Sent: January 17, 2024 4:59 PM To: Mark Nelson <mark.nelson@clyso.com>; ceph-users@ceph.io Subject: [ceph-users] Re: Performance impact of Heterogeneous environment
Very informative article you did Mark.
IMHO if you find yourself with very high per-OSD core count, it may be logical to just pack/add more nvmes per host, you'd be getting the best price per performance and capacity.
/Maged
It's a little tricky. In the upstream lab we don't strictly see an IOPS or average latency advantage with heavy parallelism by running muliple OSDs per NVMe drive until per-OSD core counts get very high. There does seem to be a fairly consistent tail latency advantage even at moderately low core counts however. Results are here:
https://ceph.io/en/news/blog/2023/reef-osds-per-nvme/
Specifically for jitter, there is probably an advantage to using 2 cores per OSD unless you are very CPU starved, but how much that actually helps in practice for a typical production workload is questionable imho. You do pay some overhead for running 2 OSDs per NVMe as well.
Mark
On 1/17/24 12:24, Anthony D'Atri wrote:
Conventional wisdom is that with recent Ceph releases there is no longer a clear advantage to this.
On Jan 17, 2024, at 11:56, Peter Sabaini <peter@sabaini.at> wrote:
One thing that I've heard people do but haven't done personally with fast NVMes (not familiar with the IronWolf so not sure if they qualify) is partition them up so that they run more than one OSD (say 2 to 4) on a single NVMe to better utilize the NVMe bandwidth. See https://ceph.com/community/bluestore-default-vs-tuned-
On 17/01/2024 22:00, Mark Nelson wrote: performance-co
mparison/
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
participants (11)
-
Anthony D'Atri
-
Anthony D'Atri
-
Bailey Allison
-
Frank Schilder
-
Frédéric Nass
-
Maged Mokhtar
-
Mark Nelson
-
Mark Nelson
-
Peter Sabaini
-
Sridhar Seshasayee
-
Tino Todino