We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ). Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense? 2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off. 4. Any feedback on max OSDs? 5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ). Regards, -Brent Existing Clusters: Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways
On 2020-09-23 07:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
As always ... it depends :-). But I would not recommend it. For NVMe you want to use more like 10 GB per OSD (osd memory target) and have some spare RAM for buffer cache. The amount of CPU would be sufficient for normal use, but might not be enough when in a recovery situation / RocksDB housekeeping etc. But it depends on what Ceph features you want to use (RBD won't use much OMAP/META, so you would be OK with that use case).
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job.
Yeah, let Ceph handle to redundancy. You don't want to use hardware raid controllers.
3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off.
Should be OK, especially for new clusters. Test, test, test.
4. Any feedback on max OSDs?
I would recommend like 10 NVMe per server. More nodes is always better than more dense nodes from a performance perspective, has less impact when one node fails. The more nodes the less impact when one node fails, faster recovery / backfill, etc.
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ).
Do you want to be able to fully utilize all NVMe regarding throughput? That will be an issue. You will be limited by bandwith to backfill those OSDs (especially if you need to backfill a whole node at once).
I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ).
Most probably, yes, plus increased latency. My standpoint is to not use hardware raidcontrollers for NVMe storage. Gr. Stefan
Thanks for the feedback everyone! It seems we have more to look into regarding NVMe enterprise storage solutions. The workload doesn’t demand NVMe performance, so SSD seems to be the most cost effective way to handle this. The performance discussion is very interesting! Regards, Brent -----Original Message----- From: Stefan Kooman <stefan@bit.nl> Sent: Wednesday, September 23, 2020 3:49 AM To: Brent Kennedy <bkennedy@cfl.rr.com>; 'ceph-users' <ceph-users@ceph.io> Subject: Re: [ceph-users] NVMe's On 2020-09-23 07:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
As always ... it depends :-). But I would not recommend it. For NVMe you want to use more like 10 GB per OSD (osd memory target) and have some spare RAM for buffer cache. The amount of CPU would be sufficient for normal use, but might not be enough when in a recovery situation / RocksDB housekeeping etc. But it depends on what Ceph features you want to use (RBD won't use much OMAP/META, so you would be OK with that use case).
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job.
Yeah, let Ceph handle to redundancy. You don't want to use hardware raid controllers.
3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off.
Should be OK, especially for new clusters. Test, test, test.
4. Any feedback on max OSDs?
I would recommend like 10 NVMe per server. More nodes is always better than more dense nodes from a performance perspective, has less impact when one node fails. The more nodes the less impact when one node fails, faster recovery / backfill, etc.
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ).
Do you want to be able to fully utilize all NVMe regarding throughput? That will be an issue. You will be limited by bandwith to backfill those OSDs (especially if you need to backfill a whole node at once).
I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ).
Most probably, yes, plus increased latency. My standpoint is to not use hardware raidcontrollers for NVMe storage. Gr. Stefan
Depends on your expected load not? I already read here numerous of times that osd's can not keep up with nvme's, that is why people put 2 osd's on a single nvme. So on a busy node, you probably run out of cores? (But better verify this with someone that has an nvme cluster ;)) -----Original Message----- To: 'ceph-users' Subject: [ceph-users] NVMe's We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ). Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense? 2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off. 4. Any feedback on max OSDs? 5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ). Regards, -Brent Existing Clusters: Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 23/09/2020 10:54, Marc Roos wrote:
Depends on your expected load not? I already read here numerous of times that osd's can not keep up with nvme's, that is why people put 2 osd's on a single nvme. So on a busy node, you probably run out of cores? (But better verify this with someone that has an nvme cluster ;))
Did you? I just start to though about this idea too, as some devices can deliver about twice of the own ceph-osd performance. How they did it? I have an idea to create a new bucket type under host, and put two LV from each ceph osd VG into that new bucket. Rules are the same (different host), so redundancy won't be affected, but doubling number of ceph-osd daemons can squeeze a bit more iops from backend devices at expense of doubling Rocksdb size (reducing payload size) and using more cores. And I really want to hear all bad things about this setup before trying it.
I don't think you need a bucket under host for the two LVs. It's unnecessary. September 23, 2020 6:45 AM, "George Shuklin" <george.shuklin@gmail.com> wrote:
On 23/09/2020 10:54, Marc Roos wrote:
Depends on your expected load not? I already read here numerous of times that osd's can not keep up with nvme's, that is why people put 2 osd's on a single nvme. So on a busy node, you probably run out of cores? (But better verify this with someone that has an nvme cluster ;))
Did you? I just start to though about this idea too, as some devices can deliver about twice of the own ceph-osd performance.
How they did it?
I have an idea to create a new bucket type under host, and put two LV from each ceph osd VG into that new bucket. Rules are the same (different host), so redundancy won't be affected, but doubling number of ceph-osd daemons can squeeze a bit more iops from backend devices at expense of doubling Rocksdb size (reducing payload size) and using more cores.
And I really want to hear all bad things about this setup before trying it.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sounds like you just want to create 2 OSDs per drive? It's OK, everyone does that :) I tested Ceph with 2 OSDs per SATA SSD when comparing it to my Vitastor, Micron also tested Ceph with 2 OSDs per SSD in their PDF and so on.
On 23/09/2020 10:54, Marc Roos wrote:
Depends on your expected load not? I already read here numerous of times that osd's can not keep up with nvme's, that is why people put 2 osd's on a single nvme. So on a busy node, you probably run out of cores? (But better verify this with someone that has an nvme cluster ;))
Did you? I just start to though about this idea too, as some devices can deliver about twice of the own ceph-osd performance.
How they did it?
I have an idea to create a new bucket type under host, and put two LV from each ceph osd VG into that new bucket. Rules are the same (different host), so redundancy won't be affected, but doubling number of ceph-osd daemons can squeeze a bit more iops from backend devices at expense of doubling Rocksdb size (reducing payload size) and using more cores.
And I really want to hear all bad things about this setup before trying it.
How they did it?
You can create partitions / LVs by hand and build OSDs on them, or you can use ceph-volume lvm batch –osds-per-device
I have an idea to create a new bucket type under host, and put two LV from each ceph osd VG into that new bucket. Rules are the same (different host), so redundancy won't be affected
CRUSH lets you do that, but to what end? It would visually show you a bit more clearly which OSDs share a device when you run `ceph osd tree`, maybe some operational convenience with `ceph osd ls-tree`, but for placement anti-affinity it wouldn’t get you anything you don’t already have.
Hi Brent,
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
Do you seriously mean 48 NVMes per server? How would you even come remotely close to supporting them with connection (to board) and network bandwidth? Regarding some of your points there are some valuable comments by Mark Nelson in the archives, hopefully he is okay with me quoting them here, but of course better look them up in the archives for full context. RAM with NVMe OSDs:
So basically the answer is that how much memory you need depends largely on how much you care about performance, how many objects are present on an OSD, and how many objects (and how much data) you have in your active data set. 4GB is sort of our current default memory target per OSD, but as someone else mentioned bumping that up to 8-12GB per OSD might make sense for OSDs on large NVMe drives. You can also lower that down to about 2GB before you start having real issues, but it definitely can have an impact on OSD performance.
CPUs with NVMe OSDs:
With 10 NVMe drives per node, I'm guessing that a single EPYC 7451 is going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but will be network bound for large IO workloads unless you are sticking 2x100GbE in. You might want to consider jumping up to the 7601. That would get you closer to where you want to be for 10 NVMe drives (3.2c/6.4t per OSD).
Vitaly Filipov (I think) has also compiled interesting information in his wiki here: https://yourcmc.ru/wiki/Ceph_performance Best Greetings André ----- Am 23. Sep 2020 um 7:39 schrieb Brent Kennedy bkennedy@cfl.rr.com:
We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ).
-- Dipl.-Inf. André Gemünd, Leiter IT / Head of IT Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemuend@scai.fraunhofer.de Tel: +49 2241 14-2193 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend
Apologies for not consolidating these replys. My UMA is not my friend today.
With 10 NVMe drives per node, I'm guessing that a single EPYC 7451 is going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but will be network bound for large IO workloads unless you are sticking 2x100GbE in. You might want to consider jumping up to the 7601. That would get you closer to where you want to be for 10 NVMe drives (3.2c/6.4t per OSD).
A note about this worthy excerpt — I believe it dates to 2019/03/09 ; the models described are the first-gen (Naples) procs and these numbers I think assume one OSD / drive. Today we have the second-gen (Rome) procs that deliver more bang/core and there are also more options up to 64c / 128t. Tying into the discussion favoring smaller, easier to manage nodes: with both generations there are XXXXP models that are only single-socket capable — possibly binned dies but that’s only speculation on my part. By going with a single-socket architecture one can choose one of these models and save considerably compared to a dual-socket-capable model with comparable performance. So today one might consider say the 7502P, stepping to the 7702P if needed. When comparing benchmarks, let’s remember that “performance” exists along multiple axes and one needs to be specific about your use-case. Eg. for RBD vols attached to VMs for block storage, I’ve found that users tend to be more concerned with latency than with bandwidth, so sure you can pump mad GB/s out of a system, but if the average write latency climbs to 100ms, that might not be compatible with the use-case. Similarly with high queue depths that we see in benchmarks, those aren’t compatible with some common RBD workloads — but might make a killer RGW OSD box. — aad
Hi
We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
What do you want to achieve? NVMes aren't much better than good SATA SSDs in Ceph for random workloads. They're only better for linear workloads.
Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
In my opinion, no. 1 NVMe OSD = 3-6 CPU cores in Ceph. And JBODs of multiple NVMe drives... O_o why would you want that? You won't get NVMe speeds from such JBODs.
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job.
Ceph over RAID 5 is a bad idea.
3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off.
4. Any feedback on max OSDs?
min(CPU cores / 3..6, network bandwidth / 1..2 gigabyte per second, free PCIe lanes / 4, free memory / at least 4 GB) Otherwise you have a bottleneck and you're wasting money. :-) Also I think you should forget about NVMe RAID controllers. 1 NVMe = 4 PCIe lanes, that's all.
I've just finishing doing our own benchmarking, and I can say, you want to do something very unbalanced and CPU bounded. 1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per ceph-osd at top-performance (see the recent thread on 'ceph on brd') with more realistic numbers around 300-400% CPU per device. 2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a little more with top-tier low-core high-frequency CPU, but not much). So, super-duper-nvme wont make difference. (btw, I have a stupid idea to try to run two ceph-osd from the same LV with a single PV underneath VG, but it not tested). 3. You wll find that any given client performance is heavily limited by sum of all RTT in the network, plus own latencies of ceph, so very fast NVME give a diminishing return. 4. CPU bounded ceph-osd completely wipe any differences for underlying devices (except for desktop-class crawlers). You can run your own tests, even without fancy 48-nvme boxes - just run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on anything else (ramdisk is the fastest), so numbers you get from brd is supremum (upper bound) for theoretical performance. Given max 400-500% CPU per ceph-osd I'd say you need to keep number of NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation). In my opinion less fancy boxes with smaller number of drives per server (but larger number of servers) would make your (or your operation team's) life much less stressful. NEVER ever use raid with ceph. On 23/09/2020 08:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ).
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways, 2 iscsi gateways
UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways
US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 9/23/20 5:41 AM, George Shuklin wrote:
I've just finishing doing our own benchmarking, and I can say, you want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per ceph-osd at top-performance (see the recent thread on 'ceph on brd') with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for community ceph R&D we've pushed a single OSD to consume around 1400% CPU at 80K write IOPS! :) I agree though, we typical see a peak of about 500-600% CPU per OSD on multi-node clusters with a correspondingly lower write throughput. I do believe that in some cases the mix of IO we are doing is causing us to at least be partially bound by disk write latency with the single writer thread in the rocksdb WAL though.
2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a little more with top-tier low-core high-frequency CPU, but not much). So, super-duper-nvme wont make difference. (btw, I have a stupid idea to try to run two ceph-osd from the same LV with a single PV underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's caches which internally has proven to help quite a bit with latency bound workloads as it reduces lock contention in onode cache shards and the impact of cache trimming (no more single trimming trim thread constantly grabbing the lock for long periods of time!). In a 64 NVMe drive setup (P4510s), we were able to do a little north of 400K write IOPS with 3x replication, so about 19K IOPs per OSD once you factor rep in. Also, in Nautilus you can see real benefits wtih running multiple OSDs on a single device but with Octopus and master we've pretty much closed the gap on our test setup: https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror... Generally speaking using the latency-performance or latency-network tuned profiles helps (mostly due to avoid C state CPU transitions) as does higher clock speeds. Not using replication helps but that's obviously not a realistic solution for most people. :)
3. You wll find that any given client performance is heavily limited by sum of all RTT in the network, plus own latencies of ceph, so very fast NVME give a diminishing return. 4. CPU bounded ceph-osd completely wipe any differences for underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on anything else (ramdisk is the fastest), so numbers you get from brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation).
In my opinion less fancy boxes with smaller number of drives per server (but larger number of servers) would make your (or your operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the Inktank days. It costs more and is lower density, but the design is simpler, you are less likely to under provision CPU, less likely to run into memory bandwidth bottlenecks, and you have less recovery to do when a node fails. Especially now with how many NVMe drives you can fit in a single 1U server!
NEVER ever use raid with ceph.
NEVER is a strong word. There are some specialized products other there that do raid behind the scenes fairly quickly. In very specific cases you might consider a solution with very fast RAID6 backed OSDs and 2X replication, but generally speaking I agree that simpler is better especially if you are doing it yourself. Mark
On 23/09/2020 08:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ).
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways, 2 iscsi gateways
UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways
US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm curious if you've tried octopus+ yet?
Why don't you publish results of your test cluster? You cannot expect all new users to buy 4 servers with 40 disks, and try if the performance is ok. Get a basic cluster and start publishing results, and document changes to the test cluster.
On 9/23/20 8:05 AM, Marc Roos wrote:
I'm curious if you've tried octopus+ yet? Why don't you publish results of your test cluster? You cannot expect all new users to buy 4 servers with 40 disks, and try if the performance is ok.
Get a basic cluster and start publishing results, and document changes to the test cluster.
By publish do you mean write up a report based on the spreadsheet I linked? I periodically do that (both internally and externally) but there's not enough time in the day to do it for everything I look at if I also want to actually fix anything ala https://github.com/ceph/ceph/pull/28597. If you mean more reference architecture validation type stuff we have a team at Red Hat that does that with RHCS and OCS. It's one of the benefits you get when you go down the RH support path. Mark
I would put that data on the ceph.com website. Eg. A performance/test page with every release and compared to the previous release. Some default fio tests like you now have in the spreadsheet. And maybe some io patterns that relate to real world use cases like databases. Like eg how these guys from storagereview publish stuff[1] I think it is bad that someone needs to ask here on the mailing list, whether or not he can put mariadb on a rbd image. I also had to search the mailing list to discover info about the impact of using encrypted osd's. Would have been great to look at charts of a reference cluster from before and after such a change. I assume your choice for aes-xts-plain64 was done on the basis of some default tests. If performed on the reference cluster, they could be published straight away on the internet. (And nobody needed to ask here) [1] https://www.storagereview.com/review/hgst-4tb-deskstar-nas-hdd-review -----Original Message----- Subject: Re: [ceph-users] Re: NVMe's On 9/23/20 8:05 AM, Marc Roos wrote:
I'm curious if you've tried octopus+ yet? Why don't you publish results of your test cluster? You cannot expect all new users to buy 4 servers with 40 disks, and try if the performance is ok.
Get a basic cluster and start publishing results, and document changes
to the test cluster.
By publish do you mean write up a report based on the spreadsheet I linked? I periodically do that (both internally and externally) but there's not enough time in the day to do it for everything I look at if I also want to actually fix anything ala https://github.com/ceph/ceph/pull/28597. If you mean more reference architecture validation type stuff we have a team at Red Hat that does that with RHCS and OCS. It's one of the benefits you get when you go down the RH support path. Mark
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror...
I see that in your tests Octopus delivers more than twice iops with 1 OSD. Can I ask you what's my problem then? :-) I have a 4-node Ceph cluster with 14 NVMe drives and fast CPUs (Threadrippers), I only get about 30k iops in total! With size=2. And I can't say that anything improved when I upgraded from Nautilus to Octopus...
I've just finishing doing our own benchmarking, and I can say, you want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per ceph-osd at top-performance (see the recent thread on 'ceph on brd') with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for community ceph R&D we've pushed a single OSD to consume around 1400% CPU at 80K write IOPS! :) I agree though, we typical see a peak of about 500-600% CPU per OSD on multi-node clusters with a correspondingly lower write throughput. I do believe that in some cases the mix of IO we are doing is causing us to at least be partially bound by disk write latency with the single writer thread in the rocksdb WAL though.
I'd really like to see how they done this without offloading (their configuration).
2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a little more with top-tier low-core high-frequency CPU, but not much). So, super-duper-nvme wont make difference. (btw, I have a stupid idea to try to run two ceph-osd from the same LV with a single PV underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's caches which internally has proven to help quite a bit with latency bound workloads as it reduces lock contention in onode cache shards and the impact of cache trimming (no more single trimming trim thread constantly grabbing the lock for long periods of time!). In a 64 NVMe drive setup (P4510s), we were able to do a little north of 400K write IOPS with 3x replication, so about 19K IOPs per OSD once you factor rep in. Also, in Nautilus you can see real benefits wtih running multiple OSDs on a single device but with Octopus and master we've pretty much closed the gap on our test setup:
It's octopus. I was doing single-osd benchmark, removing all movable parts (brd instead of nvme, no network, size=1, etc). Moreover, I've focused on rados benchmark, as RBD is just a derivative from rados performance. Anyway, big thank you for input.
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror...
Generally speaking using the latency-performance or latency-network tuned profiles helps (mostly due to avoid C state CPU transitions) as does higher clock speeds. Not using replication helps but that's obviously not a realistic solution for most people. :)
I used size=1 and 'no ssd, no network' as upper bound. If allows to find limits for ceph-osd performance. Any real-life things (replication, network, real block devices) will make things worse, not better. Knowing upper performance bound is really nice when start to choose server configuration.
3. You wll find that any given client performance is heavily limited by sum of all RTT in the network, plus own latencies of ceph, so very fast NVME give a diminishing return. 4. CPU bounded ceph-osd completely wipe any differences for underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on anything else (ramdisk is the fastest), so numbers you get from brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation).
In my opinion less fancy boxes with smaller number of drives per server (but larger number of servers) would make your (or your operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the Inktank days. It costs more and is lower density, but the design is simpler, you are less likely to under provision CPU, less likely to run into memory bandwidth bottlenecks, and you have less recovery to do when a node fails. Especially now with how many NVMe drives you can fit in a single 1U server!
On 9/23/20 8:23 AM, George Shuklin wrote:
I've just finishing doing our own benchmarking, and I can say, you want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per ceph-osd at top-performance (see the recent thread on 'ceph on brd') with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for community ceph R&D we've pushed a single OSD to consume around 1400% CPU at 80K write IOPS! :) I agree though, we typical see a peak of about 500-600% CPU per OSD on multi-node clusters with a correspondingly lower write throughput. I do believe that in some cases the mix of IO we are doing is causing us to at least be partially bound by disk write latency with the single writer thread in the rocksdb WAL though.
I'd really like to see how they done this without offloading (their configuration).
I went back and looked over some of the old results. I didn't find the really high test scores (and now that I'm thinking about it they may have been from when I was ripping out pglog OMAP updates!), but here's one example I did find from earlier testing last winter that at least got into roughly the right ballpark with stock master from last December (~66K IOPS): Avg 4K FIO randwrite IOPS: 65841.7 - 1 p4510 NVMe backed OSD - 8GB osd memory target - 4K min alloc size - 4 clients, 1 128GB RBD volume per client, io_depth=128, time=300s - 128 PGs (fixed) - latency-network tuned profile - bluestore_rocksdb_options = "compression=kNoCompression,max_total_wal_size=1073741824,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=4,write_buffer_size=67108864,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,compaction_style=kCompactionStyleUniversal" - bluestore_default_buffered_write = true - bluestore_default_buffered_read = true - rbd cache = false Beyond that general stuff like background scrubbing and pg autoscaling was disabled. I should note that these results are using universal compaction in rocksdb which you probably don't want to do in production because it can require 2x the total DB space to perform a compaction. It might actually be feasible now that we are doing column family sharding thanks to Adam's PR because you will only need 2x the space of any individual column family for compaction rather than the whole DB, but it's still unsupported for now. Mark
2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a little more with top-tier low-core high-frequency CPU, but not much). So, super-duper-nvme wont make difference. (btw, I have a stupid idea to try to run two ceph-osd from the same LV with a single PV underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's caches which internally has proven to help quite a bit with latency bound workloads as it reduces lock contention in onode cache shards and the impact of cache trimming (no more single trimming trim thread constantly grabbing the lock for long periods of time!). In a 64 NVMe drive setup (P4510s), we were able to do a little north of 400K write IOPS with 3x replication, so about 19K IOPs per OSD once you factor rep in. Also, in Nautilus you can see real benefits wtih running multiple OSDs on a single device but with Octopus and master we've pretty much closed the gap on our test setup:
It's octopus. I was doing single-osd benchmark, removing all movable parts (brd instead of nvme, no network, size=1, etc). Moreover, I've focused on rados benchmark, as RBD is just a derivative from rados performance.
Anyway, big thank you for input.
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror...
Generally speaking using the latency-performance or latency-network tuned profiles helps (mostly due to avoid C state CPU transitions) as does higher clock speeds. Not using replication helps but that's obviously not a realistic solution for most people. :)
I used size=1 and 'no ssd, no network' as upper bound. If allows to find limits for ceph-osd performance. Any real-life things (replication, network, real block devices) will make things worse, not better. Knowing upper performance bound is really nice when start to choose server configuration.
3. You wll find that any given client performance is heavily limited by sum of all RTT in the network, plus own latencies of ceph, so very fast NVME give a diminishing return. 4. CPU bounded ceph-osd completely wipe any differences for underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on anything else (ramdisk is the fastest), so numbers you get from brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation).
In my opinion less fancy boxes with smaller number of drives per server (but larger number of servers) would make your (or your operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the Inktank days. It costs more and is lower density, but the design is simpler, you are less likely to under provision CPU, less likely to run into memory bandwidth bottlenecks, and you have less recovery to do when a node fails. Especially now with how many NVMe drives you can fit in a single 1U server!
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have no idea how you get 66k write iops with one OSD ) I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images. I got 17k iops. OK, in fact that's not the worst result for Ceph, but problem is that I only get 30k write iops when benchmarking 4 RBD images spread over all OSDs _in_the_same_cluster_. And there are 14 of them.
I've just finishing doing our own benchmarking, and I can say, you want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per ceph-osd at top-performance (see the recent thread on 'ceph on brd') with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for community ceph R&D we've pushed a single OSD to consume around 1400% CPU at 80K write IOPS! :) I agree though, we typical see a peak of about 500-600% CPU per OSD on multi-node clusters with a correspondingly lower write throughput. I do believe that in some cases the mix of IO we are doing is causing us to at least be partially bound by disk write latency with the single writer thread in the rocksdb WAL though.
I'd really like to see how they done this without offloading (their configuration).
I went back and looked over some of the old results. I didn't find the really high test scores (and now that I'm thinking about it they may have been from when I was ripping out pglog OMAP updates!), but here's one example I did find from earlier testing last winter that at least got into roughly the right ballpark with stock master from last December (~66K IOPS):
Avg 4K FIO randwrite IOPS: 65841.7
- 1 p4510 NVMe backed OSD
- 8GB osd memory target
- 4K min alloc size
- 4 clients, 1 128GB RBD volume per client, io_depth=128, time=300s
- 128 PGs (fixed)
- latency-network tuned profile
- bluestore_rocksdb_options = "compression=kNoCompression,max_total_wal_size=1073741824,max_write_buffer_number=16,min_write_buffe _number_to_merge=3,recycle_log_file_num=4,write_buffer_size=67108864,writable_file_max_buffer_size=0 compaction_readahead_size=2097152,max_background_compactions=2,compaction_style=kCompactionStyleUniv rsal"
- bluestore_default_buffered_write = true
- bluestore_default_buffered_read = true
- rbd cache = false
Beyond that general stuff like background scrubbing and pg autoscaling was disabled. I should note that these results are using universal compaction in rocksdb which you probably don't want to do in production because it can require 2x the total DB space to perform a compaction. It might actually be feasible now that we are doing column family sharding thanks to Adam's PR because you will only need 2x the space of any individual column family for compaction rather than the whole DB, but it's still unsupported for now.
Mark
2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a little more with top-tier low-core high-frequency CPU, but not much). So, super-duper-nvme wont make difference. (btw, I have a stupid idea to try to run two ceph-osd from the same LV with a single PV underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's caches which internally has proven to help quite a bit with latency bound workloads as it reduces lock contention in onode cache shards and the impact of cache trimming (no more single trimming trim thread constantly grabbing the lock for long periods of time!). In a 64 NVMe drive setup (P4510s), we were able to do a little north of 400K write IOPS with 3x replication, so about 19K IOPs per OSD once you factor rep in. Also, in Nautilus you can see real benefits wtih running multiple OSDs on a single device but with Octopus and master we've pretty much closed the gap on our test setup:
It's octopus. I was doing single-osd benchmark, removing all movable parts (brd instead of nvme, no network, size=1, etc). Moreover, I've focused on rados benchmark, as RBD is just a derivative from rados performance.
Anyway, big thank you for input.
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror...
Generally speaking using the latency-performance or latency-network tuned profiles helps (mostly due to avoid C state CPU transitions) as does higher clock speeds. Not using replication helps but that's obviously not a realistic solution for most people. :)
I used size=1 and 'no ssd, no network' as upper bound. If allows to find limits for ceph-osd performance. Any real-life things (replication, network, real block devices) will make things worse, not better. Knowing upper performance bound is really nice when start to choose server configuration.
3. You wll find that any given client performance is heavily limited by sum of all RTT in the network, plus own latencies of ceph, so very fast NVME give a diminishing return. 4. CPU bounded ceph-osd completely wipe any differences for underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on anything else (ramdisk is the fastest), so numbers you get from brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation).
In my opinion less fancy boxes with smaller number of drives per server (but larger number of servers) would make your (or your operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the Inktank days. It costs more and is lower density, but the design is simpler, you are less likely to under provision CPU, less likely to run into memory bandwidth bottlenecks, and you have less recovery to do when a node fails. Especially now with how many NVMe drives you can fit in a single 1U server!
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 9/23/20 10:58 AM, vitalif@yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images. I got 17k iops.
OK, in fact that's not the worst result for Ceph, but problem is that I only get 30k write iops when benchmarking 4 RBD images spread over all OSDs _in_the_same_cluster_. And there are 14 of them.
The key parts are the single OSD, the tuning, the fast HW, the massive aggregate io depth, and the fact that it's a fresh OSD with only this dataset sitting on it. Like I said, you can't get that when you go to a multi-node setup with replication. The same hardware is only achieving about 19K IOPS per OSD before replication in an 8 node, 64 NVMe, 3x rep configuration. Feel free to look at the data from the run yourself though (2.5MB tgz file): https://drive.google.com/file/d/1SuFoJtxAauKTBjgrV9XB00_M3Yr9LwW1/view?usp=s... In the results directory you'll see the 4 separate copies of fio doing 4k random writes each to it's own 128GB dataset on it's own RBD volume via the librbd engine with iodepth=128: [nhm@o10 id-f7e4b44f]$ grep "randwrite," output.*.localhost output.0.localhost:cbt-librbd/o10.front.sepia.ceph.com-0-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 output.1.localhost:cbt-librbd/o10.front.sepia.ceph.com-1-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 output.2.localhost:cbt-librbd/o10.front.sepia.ceph.com-2-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 output.3.localhost:cbt-librbd/o10.front.sepia.ceph.com-3-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 [nhm@o10 id-f7e4b44f]$ grep rbdname output.*.localhost output.0.localhost: "rbdname" : "o10.front.sepia.ceph.com-0", output.1.localhost: "rbdname" : "o10.front.sepia.ceph.com-1", output.2.localhost: "rbdname" : "o10.front.sepia.ceph.com-2", output.3.localhost: "rbdname" : "o10.front.sepia.ceph.com-3", [nhm@o10 id-f7e4b44f]$ grep size output.*.localhost output.0.localhost: "size" : "131072M", output.1.localhost: "size" : "131072M", output.2.localhost: "size" : "131072M", output.3.localhost: "size" : "131072M", [nhm@o10 id-f7e4b44f]$ grep IOPS output.*.localhost output.0.localhost: write: IOPS=15.5k, BW=60.5MiB/s (63.4MB/s)(17.7GiB/300003msec); 0 zone resets output.1.localhost: write: IOPS=15.7k, BW=61.2MiB/s (64.2MB/s)(17.9GiB/300005msec); 0 zone resets output.2.localhost: write: IOPS=15.6k, BW=60.9MiB/s (63.9MB/s)(17.8GiB/300005msec); 0 zone resets output.3.localhost: write: IOPS=19.1k, BW=74.5MiB/s (78.2MB/s)(21.8GiB/300006msec); 0 zone resets timestamp is virtually identical: [nhm@o10 id-f7e4b44f]$ grep timestamp_ms output.*.localhost output.0.localhost: "timestamp_ms" : 1570222502962, output.1.localhost: "timestamp_ms" : 1570222502965, output.2.localhost: "timestamp_ms" : 1570222502965, output.3.localhost: "timestamp_ms" : 1570222502959, as is job runtime: output.0.localhost: "job_runtime" : 300002, output.1.localhost: "job_runtime" : 300004, output.2.localhost: "job_runtime" : 300004, output.3.localhost: "job_runtime" : 300005, We can perform some sanity checks by looking at collectl data from the run: [nhm@o10 id-f7e4b44f]$ collectl -sD -oT -p collectl.localhost/o10-20191004-204957.raw.gz | grep -E "nvme0n1p1|#" | tail -n 21 # DISK STATISTICS (/sec) # <---------reads---------------><---------writes--------------><--------averages--------> Pct #Time Name KBytes Merged IOs Size Wait KBytes Merged IOs Size Wait RWSize QLen Wait SvcTim Util 20:54:45 nvme0n1p1 0 0 0 0 0 227780 55001 1944 117 1 117 7 1 0 39 20:54:46 nvme0n1p1 0 0 0 0 0 485056 117319 3945 123 4 122 38 4 0 45 20:54:47 nvme0n1p1 0 0 0 0 0 409144 98947 3339 123 7 122 52 7 0 42 20:54:48 nvme0n1p1 0 0 0 0 0 384940 93056 3179 121 2 121 14 2 0 51 20:54:49 nvme0n1p1 0 0 0 0 0 499152 120724 4064 123 4 122 28 4 0 55 20:54:50 nvme0n1p1 0 0 0 0 0 388560 93931 3209 121 4 121 26 4 0 52 20:54:51 nvme0n1p1 0 0 0 0 0 316304 76415 2661 119 5 118 26 5 0 51 20:54:52 nvme0n1p1 0 0 0 0 0 418876 107305 3443 122 5 121 30 5 0 52 20:54:53 nvme0n1p1 0 0 0 0 0 367420 82728 3098 119 4 118 26 4 0 50 20:54:54 nvme0n1p1 0 0 0 0 0 293444 70912 2449 120 5 119 33 5 0 41 20:54:55 nvme0n1p1 0 0 0 0 0 300680 72687 2483 121 6 121 39 6 0 38 20:54:56 nvme0n1p1 0 0 0 0 0 274404 66311 2290 120 2 119 10 2 0 40 20:54:57 nvme0n1p1 0 0 0 0 0 244220 58996 2059 119 5 118 28 5 0 39 20:54:58 nvme0n1p1 0 0 0 0 0 242116 58491 2038 119 6 118 31 6 0 37 20:54:59 nvme0n1p1 0 0 0 0 0 436208 105512 3540 123 5 123 40 5 0 44 20:55:00 nvme0n1p1 0 0 0 0 0 427420 103459 3474 123 7 123 54 7 0 42 20:55:01 nvme0n1p1 0 0 0 0 0 152428 36677 1352 113 0 112 1 0 0 36 20:55:02 nvme0n1p1 0 0 0 0 0 448740 108542 3643 123 5 123 45 5 0 44 So in this case the block layer in the kernel was able to play some tricks and merge a bunch of IOs helping us get that nice juicy benchmark number (though these P4510 NVMe drives may have been able to handle more randomly scattered non-merged 4K writes relatively well too). in this case we actually have a number of IOs waiting in the device queue meaning we are pushing the NVMe at least moderately hard. Next we look at OSD process data over time: [nhm@o10 id-f7e4b44f]$ collectl -sZ -oT -p collectl.localhost/o10-20191004-204957.raw.gz | awk 'NR==5 || /ceph-osd/' | head -n 10 #Time PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command 20:50:08 7964 root 20 1 71 S 7G 6G 84 10.13 49.34 594 09:03.66 0 0 0 75K /usr/local/bin/ceph-osd 20:50:18 7964 root 20 1 71 S 7G 6G 84 15.26 84.28 995 10:43.21 0 0 0 42K /usr/local/bin/ceph-osd 20:50:28 7964 root 20 1 71 S 7G 6G 84 15.87 89.10 1049 12:28.18 0 0 0 59K /usr/local/bin/ceph-osd 20:50:38 7964 root 20 1 71 S 7G 6G 84 15.46 94.56 1100 14:18.20 0 0 0 66K /usr/local/bin/ceph-osd 20:50:48 7964 root 20 1 71 S 7G 6G 84 15.19 94.82 1100 16:08.21 0 0 0 76K /usr/local/bin/ceph-osd 20:50:58 7964 root 20 1 71 S 7G 6G 84 14.64 92.22 1068 17:55.07 0 0 0 52K /usr/local/bin/ceph-osd 20:51:08 7964 root 20 1 71 S 7G 6G 84 14.91 95.36 1102 19:45.34 0 0 0 55K /usr/local/bin/ceph-osd 20:51:18 7964 root 20 1 71 S 7G 6G 84 15.48 100.92 1164 21:41.74 0 0 0 64K /usr/local/bin/ceph-osd 20:51:28 7964 root 20 1 71 S 7G 6G 84 14.91 90.63 1055 23:27.28 0 0 0 53K /usr/local/bin/ceph-osd The OSD process is consuming a little under 8GB which is reasonably on target. PCT used by the OSD stabilizes between about 10-12 cores. The important point is that the OSD/bluestore machinery is perfectly happy consuming a ton of cores and driving ~66K IOPs worth of work at the underlying device assuming the block layer and the device itself can handle it. It used to be that the kv sync and onode cache triming threads were much bigger bottlenecks but we've improved both over the last couple of years. Having said that we're getting pretty close to the current limits of bluestore here. Also once you start involving the network and replication we introduce enough extra latency and contention in our stack that the OSD can no longer maintain these kinds of numbers even if the device itself and bluestore can handle it. That's where seastore, crimson, and future msgr work come in imho. Finally, we can use cbt's osd log parser to look at rocksdb compaction statistics. This also probably explains at least some of the large IO workload we see in collectl since we are spending nearly 1/3rd of the overall OSD uptime in compaction (and given that the test only ran for 300s of that uptime, rocksdb may be spending the majority of the actual test portion of the OSD uptime performing universal compaction): [nhm@o10 nvme_nvme_bs]$ ~/src/ceph/cbt/tools/ceph_rocksdb_log_parser.py ./osd.0.log | head -n 13 Compaction Statistics ./osd.0.log Total OSD Log Duration (seconds) 605.512 Number of Compaction Events 112 Avg Compaction Time (seconds) 1.68255873214 Total Compaction Time (seconds) 188.446578 Avg Output Size: (MB) 380.801597553 Total Output Size: (MB) 42649.7789259 Total Input Records 243648735 Total Output Records 174897784 Avg Output Throughput (MB/s) 250.561577229 Avg Input Records/second 1242038.3797 Avg Output Records/second 1059967.66123 Avg Output/Input Ratio 0.862494228971 Mark
On 9/23/20 12:18 PM, Mark Nelson wrote:
On 9/23/20 10:58 AM, vitalif@yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images. I got 17k iops.
OK, in fact that's not the worst result for Ceph, but problem is that I only get 30k write iops when benchmarking 4 RBD images spread over all OSDs _in_the_same_cluster_. And there are 14 of them.
The key parts are the single OSD, the tuning, the fast HW, the massive aggregate io depth, and the fact that it's a fresh OSD with only this dataset sitting on it. Like I said, you can't get that when you go to a multi-node setup with replication. The same hardware is only achieving about 19K IOPS per OSD before replication in an 8 node, 64 NVMe, 3x rep configuration. Feel free to look at the data from the run yourself though (2.5MB tgz file):
https://drive.google.com/file/d/1SuFoJtxAauKTBjgrV9XB00_M3Yr9LwW1/view?usp=s...
In the results directory you'll see the 4 separate copies of fio doing 4k random writes each to it's own 128GB dataset on it's own RBD volume via the librbd engine with iodepth=128:
[nhm@o10 id-f7e4b44f]$ grep "randwrite," output.*.localhost output.0.localhost:cbt-librbd/o10.front.sepia.ceph.com-0-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 output.1.localhost:cbt-librbd/o10.front.sepia.ceph.com-1-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 output.2.localhost:cbt-librbd/o10.front.sepia.ceph.com-2-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 output.3.localhost:cbt-librbd/o10.front.sepia.ceph.com-3-0: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128
[nhm@o10 id-f7e4b44f]$ grep rbdname output.*.localhost output.0.localhost: "rbdname" : "o10.front.sepia.ceph.com-0", output.1.localhost: "rbdname" : "o10.front.sepia.ceph.com-1", output.2.localhost: "rbdname" : "o10.front.sepia.ceph.com-2", output.3.localhost: "rbdname" : "o10.front.sepia.ceph.com-3",
[nhm@o10 id-f7e4b44f]$ grep size output.*.localhost output.0.localhost: "size" : "131072M", output.1.localhost: "size" : "131072M", output.2.localhost: "size" : "131072M", output.3.localhost: "size" : "131072M",
[nhm@o10 id-f7e4b44f]$ grep IOPS output.*.localhost output.0.localhost: write: IOPS=15.5k, BW=60.5MiB/s (63.4MB/s)(17.7GiB/300003msec); 0 zone resets output.1.localhost: write: IOPS=15.7k, BW=61.2MiB/s (64.2MB/s)(17.9GiB/300005msec); 0 zone resets output.2.localhost: write: IOPS=15.6k, BW=60.9MiB/s (63.9MB/s)(17.8GiB/300005msec); 0 zone resets output.3.localhost: write: IOPS=19.1k, BW=74.5MiB/s (78.2MB/s)(21.8GiB/300006msec); 0 zone resets
timestamp is virtually identical:
[nhm@o10 id-f7e4b44f]$ grep timestamp_ms output.*.localhost output.0.localhost: "timestamp_ms" : 1570222502962, output.1.localhost: "timestamp_ms" : 1570222502965, output.2.localhost: "timestamp_ms" : 1570222502965, output.3.localhost: "timestamp_ms" : 1570222502959,
as is job runtime:
output.0.localhost: "job_runtime" : 300002, output.1.localhost: "job_runtime" : 300004, output.2.localhost: "job_runtime" : 300004, output.3.localhost: "job_runtime" : 300005,
We can perform some sanity checks by looking at collectl data from the run:
[nhm@o10 id-f7e4b44f]$ collectl -sD -oT -p collectl.localhost/o10-20191004-204957.raw.gz | grep -E "nvme0n1p1|#" | tail -n 21 # DISK STATISTICS (/sec) # <---------reads---------------><---------writes--------------><--------averages--------> Pct #Time Name KBytes Merged IOs Size Wait KBytes Merged IOs Size Wait RWSize QLen Wait SvcTim Util 20:54:45 nvme0n1p1 0 0 0 0 0 227780 55001 1944 117 1 117 7 1 0 39 20:54:46 nvme0n1p1 0 0 0 0 0 485056 117319 3945 123 4 122 38 4 0 45 20:54:47 nvme0n1p1 0 0 0 0 0 409144 98947 3339 123 7 122 52 7 0 42 20:54:48 nvme0n1p1 0 0 0 0 0 384940 93056 3179 121 2 121 14 2 0 51 20:54:49 nvme0n1p1 0 0 0 0 0 499152 120724 4064 123 4 122 28 4 0 55 20:54:50 nvme0n1p1 0 0 0 0 0 388560 93931 3209 121 4 121 26 4 0 52 20:54:51 nvme0n1p1 0 0 0 0 0 316304 76415 2661 119 5 118 26 5 0 51 20:54:52 nvme0n1p1 0 0 0 0 0 418876 107305 3443 122 5 121 30 5 0 52 20:54:53 nvme0n1p1 0 0 0 0 0 367420 82728 3098 119 4 118 26 4 0 50 20:54:54 nvme0n1p1 0 0 0 0 0 293444 70912 2449 120 5 119 33 5 0 41 20:54:55 nvme0n1p1 0 0 0 0 0 300680 72687 2483 121 6 121 39 6 0 38 20:54:56 nvme0n1p1 0 0 0 0 0 274404 66311 2290 120 2 119 10 2 0 40 20:54:57 nvme0n1p1 0 0 0 0 0 244220 58996 2059 119 5 118 28 5 0 39 20:54:58 nvme0n1p1 0 0 0 0 0 242116 58491 2038 119 6 118 31 6 0 37 20:54:59 nvme0n1p1 0 0 0 0 0 436208 105512 3540 123 5 123 40 5 0 44 20:55:00 nvme0n1p1 0 0 0 0 0 427420 103459 3474 123 7 123 54 7 0 42 20:55:01 nvme0n1p1 0 0 0 0 0 152428 36677 1352 113 0 112 1 0 0 36 20:55:02 nvme0n1p1 0 0 0 0 0 448740 108542 3643 123 5 123 45 5 0 44
So in this case the block layer in the kernel was able to play some tricks and merge a bunch of IOs helping us get that nice juicy benchmark number (though these P4510 NVMe drives may have been able to handle more randomly scattered non-merged 4K writes relatively well too). in this case we actually have a number of IOs waiting in the device queue meaning we are pushing the NVMe at least moderately hard. Next we look at OSD process data over time:
[nhm@o10 id-f7e4b44f]$ collectl -sZ -oT -p collectl.localhost/o10-20191004-204957.raw.gz | awk 'NR==5 || /ceph-osd/' | head -n 10 #Time PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command 20:50:08 7964 root 20 1 71 S 7G 6G 84 10.13 49.34 594 09:03.66 0 0 0 75K /usr/local/bin/ceph-osd 20:50:18 7964 root 20 1 71 S 7G 6G 84 15.26 84.28 995 10:43.21 0 0 0 42K /usr/local/bin/ceph-osd 20:50:28 7964 root 20 1 71 S 7G 6G 84 15.87 89.10 1049 12:28.18 0 0 0 59K /usr/local/bin/ceph-osd 20:50:38 7964 root 20 1 71 S 7G 6G 84 15.46 94.56 1100 14:18.20 0 0 0 66K /usr/local/bin/ceph-osd 20:50:48 7964 root 20 1 71 S 7G 6G 84 15.19 94.82 1100 16:08.21 0 0 0 76K /usr/local/bin/ceph-osd 20:50:58 7964 root 20 1 71 S 7G 6G 84 14.64 92.22 1068 17:55.07 0 0 0 52K /usr/local/bin/ceph-osd 20:51:08 7964 root 20 1 71 S 7G 6G 84 14.91 95.36 1102 19:45.34 0 0 0 55K /usr/local/bin/ceph-osd 20:51:18 7964 root 20 1 71 S 7G 6G 84 15.48 100.92 1164 21:41.74 0 0 0 64K /usr/local/bin/ceph-osd 20:51:28 7964 root 20 1 71 S 7G 6G 84 14.91 90.63 1055 23:27.28 0 0 0 53K /usr/local/bin/ceph-osd
The OSD process is consuming a little under 8GB which is reasonably on target. PCT used by the OSD stabilizes between about 10-12 cores. The important point is that the OSD/bluestore machinery is perfectly happy consuming a ton of cores and driving ~66K IOPs worth of work at the underlying device assuming the block layer and the device itself can handle it. It used to be that the kv sync and onode cache triming threads were much bigger bottlenecks but we've improved both over the last couple of years. Having said that we're getting pretty close to the current limits of bluestore here. Also once you start involving the network and replication we introduce enough extra latency and contention in our stack that the OSD can no longer maintain these kinds of numbers even if the device itself and bluestore can handle it. That's where seastore, crimson, and future msgr work come in imho. Finally, we can use cbt's osd log parser to look at rocksdb compaction statistics. This also probably explains at least some of the large IO workload we see in collectl since we are spending nearly 1/3rd of the overall OSD uptime in compaction (and given that the test only ran for 300s of that uptime, rocksdb may be spending the majority of the actual test portion of the OSD uptime performing universal compaction):
[nhm@o10 nvme_nvme_bs]$ ~/src/ceph/cbt/tools/ceph_rocksdb_log_parser.py ./osd.0.log | head -n 13 Compaction Statistics ./osd.0.log Total OSD Log Duration (seconds) 605.512 Number of Compaction Events 112 Avg Compaction Time (seconds) 1.68255873214 Total Compaction Time (seconds) 188.446578 Avg Output Size: (MB) 380.801597553 Total Output Size: (MB) 42649.7789259 Total Input Records 243648735 Total Output Records 174897784 Avg Output Throughput (MB/s) 250.561577229 Avg Input Records/second 1242038.3797 Avg Output Records/second 1059967.66123 Avg Output/Input Ratio 0.862494228971
Mark
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Minor correction, nvme0n1p1 was a DB/WAL partition while nvme1n1p2 is the block storage device where the 4k writes are happening and unmerged. That makes way more sense since those merged writes are all just the WAL and big SST writes. Sorry, it's been a while since I looked at these numbers. IE here we are using a separate NVMe drive in this case for the DB/WAL. In reality I don't recall it actually helping that much vs just putting everything on a single P4510, but in any event I wanted to update for clarity. Actual block partition writes: [nhm@o10 id-f7e4b44f]$ collectl -sD -oT -p collectl.localhost/o10-20191004-204957.raw.gz | grep -E "nvme1n1p2|#" | tail -n 21 # DISK STATISTICS (/sec) # <---------reads---------------><---------writes--------------><--------averages--------> Pct #Time Name KBytes Merged IOs Size Wait KBytes Merged IOs Size Wait RWSize QLen Wait SvcTim Util 20:54:45 nvme1n1p2 0 0 0 0 0 261804 0 65K 4 0 3 0 0 0 89 20:54:46 nvme1n1p2 0 0 0 0 0 206916 0 51K 4 0 4 0 0 0 67 20:54:47 nvme1n1p2 0 0 0 0 0 218888 0 54K 4 0 4 0 0 0 75 20:54:48 nvme1n1p2 0 0 0 0 0 261032 0 65K 4 0 4 0 0 0 79 20:54:49 nvme1n1p2 0 0 0 0 0 237132 0 59K 4 0 4 0 0 0 70 20:54:50 nvme1n1p2 0 0 0 0 0 269404 0 67K 4 0 4 0 0 0 80 20:54:51 nvme1n1p2 0 0 0 0 0 280064 0 70K 4 0 4 0 0 0 82 20:54:52 nvme1n1p2 0 0 0 0 0 251956 0 62K 4 0 4 0 0 0 75 20:54:53 nvme1n1p2 0 0 0 0 0 267232 0 66K 4 0 4 0 0 0 82 20:54:54 nvme1n1p2 0 0 0 0 0 263884 0 65K 4 0 4 0 0 0 83 20:54:55 nvme1n1p2 0 0 0 0 0 245224 0 61K 4 0 4 0 0 0 85 20:54:56 nvme1n1p2 0 0 0 0 0 262472 0 65K 4 0 4 0 0 0 86 20:54:57 nvme1n1p2 0 0 0 0 0 270272 0 67K 4 0 4 0 0 0 89 20:54:58 nvme1n1p2 0 0 0 0 0 264256 0 66K 4 0 4 0 0 0 90 20:54:59 nvme1n1p2 0 0 0 0 0 226052 0 56K 4 0 4 0 0 0 75 20:55:00 nvme1n1p2 0 0 0 0 0 224352 0 56K 4 0 4 0 0 0 76 20:55:01 nvme1n1p2 0 0 0 0 0 287760 0 71K 4 0 4 0 0 0 96 20:55:02 nvme1n1p2 0 0 0 0 0 220128 0 55K 4 0 4 0 0 0 73 Mark
OK, I'll retry my tests several times more. But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it myself on my machine. :-)) I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658% CPU and pushing only 4138 write iops...
On 9/24/20 11:46 AM, vitalif@yourcmc.ru wrote:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it myself on my machine. :-))
It's better to see evidence with your own eyes of course!
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658% CPU and pushing only 4138 write iops...
If you have the time to investigate, you might try gdbpmp or Adam's wallclock profiler to see where the OSD is spending it's time. That might help diagnose where the OSD is bottlenecked. I would certainly be interested in the results. We have quite fast systems in the lab but very limited diversity of hardware.
I did not see 10 cores, but 7 cores per osd over a long period on pm1725a disks with around 60k IO/s according to sysstat of each disk. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif@yourcmc.ru>:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658% CPU and pushing only 4138 write iops... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Mind if I ask what size of IOs those where, what kind of IOs (reads/writes/sequential/random?) and what kind of cores? Mark On 9/24/20 1:43 PM, Martin Verges wrote:
I did not see 10 cores, but 7 cores per osd over a long period on pm1725a disks with around 60k IO/s according to sysstat of each disk.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io <mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif@yourcmc.ru <mailto:vitalif@yourcmc.ru>>:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658% CPU and pushing only 4138 write iops... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hello, It was some time ago but as far as I remember and found in the chat log, it was during backfill/recovery and high client workload and on Intel Xeon Silver 4110, 2.10GHz, 8C/16T Cpu. I found a screenshot in my chat history stating 775% and 722% cpu usage in htop for 2 OSDs (the server has 2 PCIe PM1725a NVMe OSDs and 12 HDD OSDs). Unfortunately I have no console log output that would show more details like IO pattern. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Do., 24. Sept. 2020 um 21:01 Uhr schrieb Mark Nelson <mnelson@redhat.com
:
Mind if I ask what size of IOs those where, what kind of IOs (reads/writes/sequential/random?) and what kind of cores?
Mark
On 9/24/20 1:43 PM, Martin Verges wrote:
I did not see 10 cores, but 7 cores per osd over a long period on pm1725a disks with around 60k IO/s according to sysstat of each disk.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io <mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif@yourcmc.ru <mailto:vitalif@yourcmc.ru>>:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658% CPU and pushing only 4138 write iops... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Thanks for the info! Interesting numbers. Probably not 60K client IOPs/OSD then, but the tp_osd_tp threads were probably working pretty hard under the combined client/recovery workload. Mark On 9/24/20 2:49 PM, Martin Verges wrote:
Hello,
It was some time ago but as far as I remember and found in the chat log, it was during backfill/recovery and high client workload and on Intel Xeon Silver 4110, 2.10GHz, 8C/16T Cpu. I found a screenshot in my chat history stating 775% and 722% cpu usage in htop for 2 OSDs (the server has 2 PCIe PM1725a NVMe OSDs and 12 HDD OSDs). Unfortunately I have no console log output that would show more details like IO pattern.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io <mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 21:01 Uhr schrieb Mark Nelson <mnelson@redhat.com <mailto:mnelson@redhat.com>>:
Mind if I ask what size of IOs those where, what kind of IOs (reads/writes/sequential/random?) and what kind of cores?
Mark
On 9/24/20 1:43 PM, Martin Verges wrote: > I did not see 10 cores, but 7 cores per osd over a long period on > pm1725a disks with around 60k IO/s according to sysstat of each disk. > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges@croit.io <mailto:martin.verges@croit.io> <mailto:martin.verges@croit.io <mailto:martin.verges@croit.io>> > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > > Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif@yourcmc.ru <mailto:vitalif@yourcmc.ru> > <mailto:vitalif@yourcmc.ru <mailto:vitalif@yourcmc.ru>>>: > > OK, I'll retry my tests several times more. > > But I've never seen OSD utilize 10 cores, so... I won't believe it > until I see it myself on my machine. :-)) > > I tried a fresh OSD on a block ramdisk ("brd"), for example. It > was eating 658% CPU and pushing only 4138 write iops... > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> > <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>> >
On 23/09/2020 17:58, vitalif@yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images. I got 17k iops.
8 PGs is a low number, there will be a lot of PG lock contention across your 4x 128 queue depth.
OK, in fact that's not the worst result for Ceph, but problem is that I only get 30k write iops when benchmarking 4 RBD images spread over all OSDs _in_the_same_cluster_. And there are 14 of them.
I've just finishing doing our own benchmarking, and I can say, you want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per ceph-osd at top-performance (see the recent thread on 'ceph on brd') with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for community ceph R&D we've pushed a single OSD to consume around 1400% CPU at 80K write IOPS! :) I agree though, we typical see a peak of about 500-600% CPU per OSD on multi-node clusters with a correspondingly lower write throughput. I do believe that in some cases the mix of IO we are doing is causing us to at least be partially bound by disk write latency with the single writer thread in the rocksdb WAL though. I'd really like to see how they done this without offloading (their configuration). I went back and looked over some of the old results. I didn't find the really high test scores (and now that I'm thinking about it they may have been from when I was ripping out pglog OMAP updates!), but here's one example I did find from earlier testing last winter that at least got into roughly the right ballpark with stock master from last December (~66K IOPS):
Avg 4K FIO randwrite IOPS: 65841.7
- 1 p4510 NVMe backed OSD
- 8GB osd memory target
- 4K min alloc size
- 4 clients, 1 128GB RBD volume per client, io_depth=128, time=300s
- 128 PGs (fixed)
- latency-network tuned profile
- bluestore_rocksdb_options = "compression=kNoCompression,max_total_wal_size=1073741824,max_write_buffer_number=16,min_write_buffe _number_to_merge=3,recycle_log_file_num=4,write_buffer_size=67108864,writable_file_max_buffer_size=0 compaction_readahead_size=2097152,max_background_compactions=2,compaction_style=kCompactionStyleUniv rsal"
- bluestore_default_buffered_write = true
- bluestore_default_buffered_read = true
- rbd cache = false
Beyond that general stuff like background scrubbing and pg autoscaling was disabled. I should note that these results are using universal compaction in rocksdb which you probably don't want to do in production because it can require 2x the total DB space to perform a compaction. It might actually be feasible now that we are doing column family sharding thanks to Adam's PR because you will only need 2x the space of any individual column family for compaction rather than the whole DB, but it's still unsupported for now.
Mark
I'm curious if you've tried octopus+ yet? We refactored bluestore's caches which internally has proven to help quite a bit with latency bound workloads as it reduces lock contention in onode cache shards and the impact of cache trimming (no more single trimming trim thread constantly grabbing the lock for long periods of time!). In a 64 NVMe drive setup (P4510s), we were able to do a little north of 400K write IOPS with 3x replication, so about 19K IOPs per OSD once you factor rep in. Also, in Nautilus you can see real benefits wtih running multiple OSDs on a single device but with Octopus and master we've pretty much closed the gap on our test setup: It's octopus. I was doing single-osd benchmark, removing all movable
2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a little more with top-tier low-core high-frequency CPU, but not much). So, super-duper-nvme wont make difference. (btw, I have a stupid idea to try to run two ceph-osd from the same LV with a single PV underneath VG, but it not tested). parts (brd instead of nvme, no network, size=1, etc). Moreover, I've focused on rados benchmark, as RBD is just a derivative from rados performance.
Anyway, big thank you for input.
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror...
Generally speaking using the latency-performance or latency-network tuned profiles helps (mostly due to avoid C state CPU transitions) as does higher clock speeds. Not using replication helps but that's obviously not a realistic solution for most people. :) I used size=1 and 'no ssd, no network' as upper bound. If allows to find limits for ceph-osd performance. Any real-life things (replication, network, real block devices) will make things worse, not better. Knowing upper performance bound is really nice when start to choose server configuration.
3. You wll find that any given client performance is heavily limited by sum of all RTT in the network, plus own latencies of ceph, so very fast NVME give a diminishing return. 4. CPU bounded ceph-osd completely wipe any differences for underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on anything else (ramdisk is the fastest), so numbers you get from brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation).
In my opinion less fancy boxes with smaller number of drives per server (but larger number of servers) would make your (or your operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the Inktank days. It costs more and is lower density, but the design is simpler, you are less likely to under provision CPU, less likely to run into memory bandwidth bottlenecks, and you have less recovery to do when a node fails. Especially now with how many NVMe drives you can fit in a single 1U server!
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's pretty much the advice I've been giving people since the Inktank days. It costs more and is lower density, but the design is simpler, you are less likely to under provision CPU, less likely to run into memory bandwidth bottlenecks, and you have less recovery to do when a node fails.
Agreed. I’ve seen very clever presentations extolling the benefits of pinning within, say, a 4-socket server — OSDs to cores to HBAs to NICs to NUMA domains. A lot of the diagrams start looking like 4x 1-socket servers in the same chassis, but with more work. With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI lanes. https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-c...
Especially now with how many NVMe drives you can fit in a single 1U server!
I’ve seen 10 for a conventional server, though depending on CPU choice some of those options don’t treat dual PSUs as redundant. EDSFF, the “ruler” form factor, shows a lot of promise in this space. Especially once the drives are available from more than one manufacturer. With TLC flash and the right Epyc P CPU it seems like a killer OSD node for RBD use. And for some Object / RGW use cases QLC drives start looking like a viable alternative to HDDs.
On Wed, Sep 23, 2020 at 8:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-c...
I was able to reach 35 Gb/s network traffic on each server (5 servers, with 6 NVMEs per server, one OSD per NVME) during a read benchmark from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s may be a bit too tight. -- Alexander E. Patrakov CV: http://pc.cd/PLz7
On 9/23/20 2:21 PM, Alexander E. Patrakov wrote:
On Wed, Sep 23, 2020 at 8:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-c... I was able to reach 35 Gb/s network traffic on each server (5 servers, with 6 NVMEs per server, one OSD per NVME) during a read benchmark from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s may be a bit too tight.
FWIW, in the nodes I mentioned earlier, we have 8 P4510 NVMe drives and 4x25GbE ports. We can do about 60-70GB/s (even some times closer to 80GB/s) per server depending on the exact workload and network/client setup. I suspect it should be possible to saturate 100GbE in ideal setups and maybe even semi-utilize 200GbE (especially for reads) in the future. Sadly we don't have any in-house to test right now. Mark
-- Alexander E. Patrakov CV: http://pc.cd/PLz7 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-c...
I was able to reach 35 Gb/s network traffic on each server (5 servers, with 6 NVMEs per server, one OSD per NVME) during a read benchmark from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s may be a bit too tight.
Thanks for the data point — without real-world reports, it’s all just theoretical. In the above presentation the point is made that latency is more important than throughput, but this is very, very much a function of the use-case. For DBs on RBD volumes, there’s a lot of truth to that especially for writes. For, say, object service or for things like OpenStack Glance, it may often be the other way around.
The workload doesn’t demand NVMe performance, so SSD seems to be the most cost effective way to handle this.
To be pedantic, NVMe devices *are* SSDs, but you most likely mean SATA SSDs. The thing is, with recent drives, chassis, and CPU models, it can be very possible to provision an NVMe server at a cost comparable to a conventional SATA SSD server, in which case, why not? — aad
participants (12)
-
Alexander E. Patrakov
-
André Gemünd
-
Anthony D'Atri
-
Brent Kennedy
-
George Shuklin
-
Maged Mokhtar
-
Marc Roos
-
Mark Nelson
-
Martin Verges
-
Stefan Kooman
-
tri＠postix.net
-
vitalif＠yourcmc.ru