23 Sep
2020
23 Sep
'20
10:39 a.m.
We currently run a SSD cluster and HDD clusters and are looking at possibly
creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
max recommended per osd host server was 16 OSDs ( I know it depends on the
CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less
CPU/RAM )? There is a reluctance to waste even a single drive on raid
because redundancy is basically cephs job.
3. The plan was to build this with octopus ( hopefully there are no issues
we should know about ). Though I just saw one posted today, but this is a
few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking
this would need at least 40Gb links to every node ( the hope is to use these
to speed up image processing at the application layer locally in the DC ).
I haven't spoken to the Dell engineers yet but my concern with NVMe is that
the raid controller would end up being the bottleneck ( next in line after
network connectivity ).
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4
gateways, 2 iscsi gateways
UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways
US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways,
2 iscsi gateways
23 Sep
23 Sep
12:48 p.m.
On 2020-09-23 07:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and
are looking at possibly
creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
max recommended per osd host server was 16 OSDs ( I know it depends on the
CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
As always ... it depends :-). But I would not recommend it. For NVMe you
want to use more like 10 GB per OSD (osd memory target) and have some
spare RAM for buffer cache. The amount of CPU would be sufficient for
normal use, but might not be enough when in a recovery situation /
RocksDB housekeeping etc. But it depends on what Ceph features you want
to use (RBD won't use much OMAP/META, so you would be OK with that use
case).
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less
CPU/RAM )? There is a reluctance to waste even a single drive on raid
because redundancy is basically cephs job.
Yeah, let Ceph handle to redundancy. You don't want to use hardware raid
controllers.
3. The plan was to build this with octopus (
hopefully there are no issues
we should know about ). Though I just saw one posted today, but this is a
few months off.
Should be OK, especially for new clusters. Test, test, test.
4. Any feedback on max OSDs?
I would recommend like 10 NVMe per server. More nodes is always better
than more dense nodes from a performance perspective, has less impact
when one node fails. The more nodes the less impact when one node fails,
faster recovery / backfill, etc.
5. Right now they run 10Gb everywhere with 80Gb
uplinks, I was thinking
this would need at least 40Gb links to every node ( the hope is to use these
to speed up image processing at the application layer locally in the DC ).
Do you want to be able to fully utilize all NVMe regarding throughput?
That will be an issue. You will be limited by bandwith to backfill those
OSDs (especially if you need to backfill a whole node at once).
I haven't spoken to the Dell engineers yet but my
concern with NVMe is that
the raid controller would end up being the bottleneck ( next in line after
network connectivity ).
Most probably, yes, plus increased latency. My standpoint is to not use
hardware raidcontrollers for NVMe storage.
Gr. Stefan
10:59 p.m.
Thanks for the feedback everyone! It seems we have more to look into regarding NVMe
enterprise storage solutions. The workload doesn’t demand NVMe performance, so SSD seems
to be the most cost effective way to handle this. The performance discussion is very
interesting!
Regards,
Brent
-----Original Message-----
From: Stefan Kooman <stefan(a)bit.nl>
Sent: Wednesday, September 23, 2020 3:49 AM
To: Brent Kennedy <bkennedy(a)cfl.rr.com>om>; 'ceph-users'
<ceph-users(a)ceph.io>
Subject: Re: [ceph-users] NVMe's
On 2020-09-23 07:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and
are looking at
possibly creating a cluster for NVMe storage. For spinners and SSDs,
it seemed the max recommended per osd host server was 16 OSDs ( I know
it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the
servers were bought with 48 cores and 100+ GB of RAM, would this make sense?
As always ... it depends :-). But I would not recommend it. For NVMe you want to use more
like 10 GB per OSD (osd memory target) and have some spare RAM for buffer cache. The
amount of CPU would be sufficient for normal use, but might not be enough when in a
recovery situation / RocksDB housekeeping etc. But it depends on what Ceph features you
want to use (RBD won't use much OMAP/META, so you would be OK with that use case).
2. Should we just raid 5 by groups of NVMe drives instead ( and buy
less CPU/RAM )? There is a reluctance to waste even a single drive on
raid because redundancy is basically cephs job.
Yeah, let Ceph handle to redundancy. You don't want to use hardware raid controllers.
3. The plan was to build this with octopus (
hopefully there are no
issues we should know about ). Though I just saw one posted today,
but this is a few months off.
Should be OK, especially for new clusters. Test, test, test.
4. Any feedback on max OSDs?
I would recommend like 10 NVMe per server. More nodes is always better than more dense
nodes from a performance perspective, has less impact when one node fails. The more nodes
the less impact when one node fails, faster recovery / backfill, etc.
5. Right now they run 10Gb everywhere with 80Gb
uplinks, I was
thinking this would need at least 40Gb links to every node ( the hope
is to use these to speed up image processing at the application layer locally in the DC
).
Do you want to be able to fully utilize all NVMe regarding throughput?
That will be an issue. You will be limited by bandwith to backfill those OSDs (especially
if you need to backfill a whole node at once).
I haven't spoken to the Dell engineers yet but my
concern with NVMe is
that the raid controller would end up being the bottleneck ( next in
line after network connectivity ).
Most probably, yes, plus increased latency. My standpoint is to not use hardware
raidcontrollers for NVMe storage.
Gr. Stefan
12:54 p.m.
Depends on your expected load not? I already read here numerous of times
that osd's can not keep up with nvme's, that is why people put 2 osd's
on a single nvme. So on a busy node, you probably run out of cores? (But
better verify this with someone that has an nvme cluster ;))
-----Original Message-----
To: 'ceph-users'
Subject: [ceph-users] NVMe's
We currently run a SSD cluster and HDD clusters and are looking at
possibly creating a cluster for NVMe storage. For spinners and SSDs, it
seemed the max recommended per osd host server was 16 OSDs ( I know it
depends on the CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy
less CPU/RAM )? There is a reluctance to waste even a single drive on
raid because redundancy is basically cephs job.
3. The plan was to build this with octopus ( hopefully there are no
issues we should know about ). Though I just saw one posted today, but
this is a few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking
this would need at least 40Gb links to every node ( the hope is to use
these to speed up image processing at the application layer locally in
the DC ).
I haven't spoken to the Dell engineers yet but my concern with NVMe is
that the raid controller would end up being the bottleneck ( next in
line after network connectivity ).
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4
gateways, 2 iscsi gateways
UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4
gateways
US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3
gateways,
2 iscsi gateways
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
3:45 p.m.
On 23/09/2020 10:54, Marc Roos wrote:
Depends on your expected load not? I already read here numerous of times
that osd's can not keep up with nvme's, that is why people put 2 osd's
on a single nvme. So on a busy node, you probably run out of cores? (But
better verify this with someone that has an nvme cluster ;))
Did you? I just start to though about this idea too, as some devices can
deliver about twice of the own ceph-osd performance.
How they did it?
I have an idea to create a new bucket type under host, and put two LV
from each ceph osd VG into that new bucket. Rules are the same
(different host), so redundancy won't be affected, but doubling number
of ceph-osd daemons can squeeze a bit more iops from backend devices at
expense of doubling Rocksdb size (reducing payload size) and using more
cores.
And I really want to hear all bad things about this setup before trying it.
4:45 p.m.
I don't think you need a bucket under host for the two LVs. It's unnecessary.
September 23, 2020 6:45 AM, "George Shuklin" <george.shuklin(a)gmail.com>
wrote:
> On 23/09/2020 10:54, Marc Roos wrote:
>
>>> Depends on your expected load not? I already read here numerous of times
>> that osd's can not keep up with nvme's, that is why people put 2
osd's
>> on a single nvme. So on a busy node, you probably run out of cores? (But
>> better verify this with someone that has an nvme cluster ;))
>
> Did you? I just start to though about this idea too, as some devices can deliver
about twice of the
> own ceph-osd performance.
>
> How they did it?
>
> I have an idea to create a new bucket type under host, and put two LV from each ceph
osd VG into
> that new bucket. Rules are the same (different host), so redundancy won't be
affected, but doubling
> number of ceph-osd daemons can squeeze a bit more iops from backend devices at
expense of doubling
> Rocksdb size (reducing payload size) and using more cores.
>
> And I really want to hear all bad things about this setup before trying it.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:15 p.m.
Sounds like you just want to create 2 OSDs per drive? It's OK, everyone does that :) I
tested Ceph with 2 OSDs per SATA SSD when comparing it to my Vitastor, Micron also tested
Ceph with 2 OSDs per SSD in their PDF and so on.
> On 23/09/2020 10:54, Marc Roos wrote:
>
>>> Depends on your expected load not? I already read here numerous of times
>> that osd's can not keep up with nvme's, that is why people put 2
osd's
>> on a single nvme. So on a busy node, you probably run out of cores? (But
>> better verify this with someone that has an nvme cluster ;))
>
> Did you? I just start to though about this idea too, as some devices can deliver
about twice of the
> own ceph-osd performance.
>
> How they did it?
>
> I have an idea to create a new bucket type under host, and put two LV from each ceph
osd VG into
> that new bucket. Rules are the same (different host), so redundancy won't be
affected, but doubling
> number of ceph-osd daemons can squeeze a bit more iops from backend devices at
expense of doubling
> Rocksdb size (reducing payload size) and using more cores.
>
> And I really want to hear all bad things about this setup before trying it.
8:27 p.m.
How they did it?
You can create partitions / LVs by hand and build OSDs on them, or you can use
ceph-volume lvm batch –osds-per-device
I have an idea to create a new bucket type under host,
and put two LV from each ceph osd VG into that new bucket. Rules are the same (different
host), so redundancy won't be affected
CRUSH lets you do that, but to what end? It would visually show you a bit more clearly
which OSDs share a device when you run `ceph osd tree`, maybe some operational convenience
with `ceph osd ls-tree`, but for placement anti-affinity it wouldn’t get you anything you
don’t already have.
1:47 p.m.
Hi Brent,
1. If we do a jbod setup, the servers can hold 48
NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
Do you seriously mean 48 NVMes per server? How would you even come remotely close to
supporting them with connection (to board) and network bandwidth?
Regarding some of your points there are some valuable comments by Mark Nelson in the
archives, hopefully he is okay with me quoting them here, but of course better look them
up in the archives for full context.
RAM with NVMe OSDs:
So basically the answer is that how much memory you
need depends largely
on how much you care about performance, how many objects are present on
an OSD, and how many objects (and how much data) you have in your active
data set. 4GB is sort of our current default memory target per OSD, but
as someone else mentioned bumping that up to 8-12GB per OSD might make
sense for OSDs on large NVMe drives. You can also lower that down to
about 2GB before you start having real issues, but it definitely can
have an impact on OSD performance.
CPUs with NVMe OSDs:
With 10 NVMe drives per node, I'm guessing that a
single EPYC 7451 is
going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but
will be network bound for large IO workloads unless you are sticking
2x100GbE in. You might want to consider jumping up to the 7601. That
would get you closer to where you want to be for 10 NVMe drives
(3.2c/6.4t per OSD).
Vitaly Filipov (I think) has also compiled interesting information in his wiki here:
https://yourcmc.ru/wiki/Ceph_performance
Best Greetings
André
----- Am 23. Sep 2020 um 7:39 schrieb Brent Kennedy bkennedy(a)cfl.rr.com:
> We currently run a SSD cluster and HDD clusters and are looking at possibly
> creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
> max recommended per osd host server was 16 OSDs ( I know it depends on the
> CPUs and RAM, like 1 cpu core and 2GB memory ).
>
>
>
> Questions:
1. If we do a jbod setup, the servers can hold 48
NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
>
> 2. Should we just raid 5 by groups of NVMe drives instead ( and buy less
> CPU/RAM )? There is a reluctance to waste even a single drive on raid
> because redundancy is basically cephs job.
> 3. The plan was to build this with octopus ( hopefully there are no issues
> we should know about ). Though I just saw one posted today, but this is a
> few months off.
>
> 4. Any feedback on max OSDs?
>
> 5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking
> this would need at least 40Gb links to every node ( the hope is to use these
> to speed up image processing at the application layer locally in the DC ).
> I haven't spoken to the Dell engineers yet but my concern with NVMe is that
> the raid controller would end up being the bottleneck ( next in line after
> network connectivity ).
--
Dipl.-Inf. André Gemünd, Leiter IT / Head of IT
Fraunhofer-Institute for Algorithms and Scientific Computing
andre.gemuend(a)scai.fraunhofer.de
Tel: +49 2241 14-2193
/C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend
8:45 p.m.
Apologies for not consolidating these replys. My UMA is not my friend today.
With 10 NVMe drives per node, I'm guessing that a
single EPYC 7451 is
going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but
will be network bound for large IO workloads unless you are sticking
2x100GbE in. You might want to consider jumping up to the 7601. That
would get you closer to where you want to be for 10 NVMe drives
(3.2c/6.4t per OSD).
A note about this worthy excerpt — I believe it dates to 2019/03/09 ; the models described
are the first-gen (Naples) procs and these numbers I think assume one OSD / drive.
Today we have the second-gen (Rome) procs that deliver more bang/core and there are also
more options up to 64c / 128t.
Tying into the discussion favoring smaller, easier to manage nodes: with both generations
there are XXXXP models that are only single-socket capable — possibly binned dies but
that’s only speculation on my part. By going with a single-socket architecture one can
choose one of these models and save considerably compared to a dual-socket-capable model
with comparable performance.
So today one might consider say the 7502P, stepping to the 7702P if needed.
When comparing benchmarks, let’s remember that “performance” exists along multiple axes
and one needs to be specific about your use-case. Eg. for RBD vols attached to VMs for
block storage, I’ve found that users tend to be more concerned with latency than with
bandwidth, so sure you can pump mad GB/s out of a system, but if the average write latency
climbs to 100ms, that might not be compatible with the use-case. Similarly with high
queue depths that we see in benchmarks, those aren’t compatible with some common RBD
workloads — but might make a killer RGW OSD box.
— aad
2:50 p.m.
Hi
We currently run a SSD cluster and HDD clusters and
are looking at possibly
creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
max recommended per osd host server was 16 OSDs ( I know it depends on the
CPUs and RAM, like 1 cpu core and 2GB memory ).
What do you want to achieve?
NVMes aren't much better than good SATA SSDs in Ceph for random workloads. They're
only better for linear workloads.
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
In my opinion, no. 1 NVMe OSD = 3-6 CPU cores in Ceph. And JBODs of multiple NVMe
drives... O_o why would you want that? You won't get NVMe speeds from such JBODs.
2. Should we just raid 5 by groups of NVMe drives
instead ( and buy less
CPU/RAM )? There is a reluctance to waste even a single drive on raid
because redundancy is basically cephs job.
Ceph over RAID 5 is a bad idea.
3. The plan was to build this with octopus ( hopefully
there are no issues
we should know about ). Though I just saw one posted today, but this is a
few months off.
4. Any feedback on max OSDs?
min(CPU cores / 3..6, network bandwidth / 1..2 gigabyte per second, free PCIe lanes / 4,
free memory / at least 4 GB)
Otherwise you have a bottleneck and you're wasting money. :-)
Also I think you should forget about NVMe RAID controllers. 1 NVMe = 4 PCIe lanes,
that's all.
3:41 p.m.
I've just finishing doing our own benchmarking, and I can say, you want
to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers around 300-400% CPU per device.
2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be a
little more with top-tier low-core high-frequency CPU, but not much).
So, super-duper-nvme wont make difference. (btw, I have a stupid idea to
try to run two ceph-osd from the same LV with a single PV underneath VG,
but it not tested).
3. You wll find that any given client performance is heavily limited by
sum of all RTT in the network, plus own latencies of ceph, so very fast
NVME give a diminishing return.
4. CPU bounded ceph-osd completely wipe any differences for underlying
devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just run
ceph-osd on brd (block ram disk). ceph-osd won't run any faster on
anything else (ramdisk is the fastest), so numbers you get from brd is
supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of
NVME in server below 12, or, 15 (but sometimes you'll get CPU saturation).
In my opinion less fancy boxes with smaller number of drives per server
(but larger number of servers) would make your (or your operation
team's) life much less stressful.
NEVER ever use raid with ceph.
On 23/09/2020 08:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters and
are looking at possibly
creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
max recommended per osd host server was 16 OSDs ( I know it depends on the
CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy less
CPU/RAM )? There is a reluctance to waste even a single drive on raid
because redundancy is basically cephs job.
3. The plan was to build this with octopus ( hopefully there are no issues
we should know about ). Though I just saw one posted today, but this is a
few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking
this would need at least 40Gb links to every node ( the hope is to use these
to speed up image processing at the application layer locally in the DC ).
I haven't spoken to the Dell engineers yet but my concern with NVMe is that
the raid controller would end up being the bottleneck ( next in line after
network connectivity ).
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4
gateways, 2 iscsi gateways
UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways
US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways,
2 iscsi gateways
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:50 p.m.
On 9/23/20 5:41 AM, George Shuklin wrote:
I've just finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for community
ceph R&D we've pushed a single OSD to consume around 1400% CPU at 80K
write IOPS! :) I agree though, we typical see a peak of about 500-600%
CPU per OSD on multi-node clusters with a correspondingly lower write
throughput. I do believe that in some cases the mix of IO we are doing
is causing us to at least be partially bound by disk write latency with
the single writer thread in the rocksdb WAL though.
2. Ceph is unable to deliver more than 12k IOPS per
ceph-osd (may be a
little more with top-tier low-core high-frequency CPU, but not much).
So, super-duper-nvme wont make difference. (btw, I have a stupid idea
to try to run two ceph-osd from the same LV with a single PV
underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's
caches which internally has proven to help quite a bit with latency
bound workloads as it reduces lock contention in onode cache shards and
the impact of cache trimming (no more single trimming trim thread
constantly grabbing the lock for long periods of time!). In a 64 NVMe
drive setup (P4510s), we were able to do a little north of 400K write
IOPS with 3x replication, so about 19K IOPs per OSD once you factor rep
in. Also, in Nautilus you can see real benefits wtih running multiple
OSDs on a single device but with Octopus and master we've pretty much
closed the gap on our test setup:
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro…
Generally speaking using the latency-performance or latency-network
tuned profiles helps (mostly due to avoid C state CPU transitions) as
does higher clock speeds. Not using replication helps but that's
obviously not a realistic solution for most people. :)
3. You wll find that any given client performance is
heavily limited
by sum of all RTT in the network, plus own latencies of ceph, so very
fast NVME give a diminishing return.
4. CPU bounded ceph-osd completely wipe any differences for underlying
devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just
run ceph-osd on brd (block ram disk). ceph-osd won't run any faster on
anything else (ramdisk is the fastest), so numbers you get from brd is
supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number of
NVME in server below 12, or, 15 (but sometimes you'll get CPU
saturation).
In my opinion less fancy boxes with smaller number of drives per
server (but larger number of servers) would make your (or your
operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the Inktank
days. It costs more and is lower density, but the design is simpler,
you are less likely to under provision CPU, less likely to run into
memory bandwidth bottlenecks, and you have less recovery to do when a
node fails. Especially now with how many NVMe drives you can fit in a
single 1U server!
NEVER ever use raid with ceph.
NEVER is a strong word. There are some specialized products other there
that do raid behind the scenes fairly quickly. In very specific cases
you might consider a solution with very fast RAID6 backed OSDs and 2X
replication, but generally speaking I agree that simpler is better
especially if you are doing it yourself.
Mark
On 23/09/2020 08:39, Brent Kennedy wrote:
We currently run a SSD cluster and HDD clusters
and are looking at
possibly
creating a cluster for NVMe storage. For spinners and SSDs, it
seemed the
max recommended per osd host server was 16 OSDs ( I know it depends
on the
CPUs and RAM, like 1 cpu core and 2GB memory ).
Questions:
1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
were bought with 48 cores and 100+ GB of RAM, would this make sense?
2. Should we just raid 5 by groups of NVMe drives instead ( and buy
less
CPU/RAM )? There is a reluctance to waste even a single drive on raid
because redundancy is basically cephs job.
3. The plan was to build this with octopus ( hopefully there are no
issues
we should know about ). Though I just saw one posted today, but this
is a
few months off.
4. Any feedback on max OSDs?
5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking
this would need at least 40Gb links to every node ( the hope is to
use these
to speed up image processing at the application layer locally in the
DC ).
I haven't spoken to the Dell engineers yet but my concern with NVMe
is that
the raid controller would end up being the bottleneck ( next in line
after
network connectivity ).
Regards,
-Brent
Existing Clusters:
Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )
US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4
gateways, 2 iscsi gateways
UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4
gateways
US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3
gateways,
2 iscsi gateways
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:05 p.m.
I'm curious if you've tried octopus+ yet?
Why don't you publish results of your test cluster? You cannot expect
all new users to buy 4 servers with 40 disks, and try if the performance
is ok.
Get a basic cluster and start publishing results, and document changes
to the test cluster.
6:22 p.m.
On 9/23/20 8:05 AM, Marc Roos wrote:
By publish do you mean write up a report based on the spreadsheet I
linked? I periodically do that (both internally and externally) but
there's not enough time in the day to do it for everything I look at if
I also want to actually fix anything ala
https://github.com/ceph/ceph/pull/28597. If you mean more reference
architecture validation type stuff we have a team at Red Hat that does
that with RHCS and OCS. It's one of the benefits you get when you go
down the RH support path.
Mark
I'm
curious if you've tried octopus+ yet?
Why don't you publish results of your
test cluster? You cannot expect
all new users to buy 4 servers with 40 disks, and try if the performance
is ok.
Get a basic cluster and start publishing results, and document changes
to the test cluster.
6:49 p.m.
I would put that data on the ceph.com website. Eg. A performance/test
page with every release and compared to the previous release. Some
default fio tests like you now have in the spreadsheet. And maybe some
io patterns that relate to real world use cases like databases. Like eg
how these guys from storagereview publish stuff[1]
I think it is bad that someone needs to ask here on the mailing list,
whether or not he can put mariadb on a rbd image. I also had to search
the mailing list to discover info about the impact of using encrypted
osd's. Would have been great to look at charts of a reference cluster
from before and after such a change. I assume your choice for
aes-xts-plain64 was done on the basis of some default tests. If
performed on the reference cluster, they could be published straight
away on the internet. (And nobody needed to ask here)
[1]
https://www.storagereview.com/review/hgst-4tb-deskstar-nas-hdd-review
-----Original Message-----
Subject: Re: [ceph-users] Re: NVMe's
On 9/23/20 8:05 AM, Marc Roos wrote:
I'm
curious if you've tried octopus+ yet?
Why don't you publish results of your
test cluster? You cannot expect
all new users to buy 4 servers with 40 disks, and try if the
performance is ok.
Get a basic cluster and start publishing results, and document changes to the test cluster.
By publish do you mean write up a report based on the spreadsheet I
linked? I periodically do that (both internally and externally) but
there's not enough time in the day to do it for everything I look at if
I also want to actually fix anything ala
https://github.com/ceph/ceph/pull/28597. If you mean more reference
architecture validation type stuff we have a team at Red Hat that does
that with RHCS and OCS. It's one of the benefits you get when you go
down the RH support path.
Mark
6:06 p.m.
I see that in your tests Octopus delivers more than twice iops with 1 OSD.
Can I ask you what's my problem then? :-)
I have a 4-node Ceph cluster with 14 NVMe drives and fast CPUs (Threadrippers), I only get
about 30k iops in total! With size=2. And I can't say that anything improved when I
upgraded from Nautilus to Octopus...
6:23 p.m.
I've just
finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for
community ceph R&D we've pushed a single OSD to consume around 1400%
CPU at 80K write IOPS! :) I agree though, we typical see a peak of
about 500-600% CPU per OSD on multi-node clusters with a
correspondingly lower write throughput. I do believe that in some
cases the mix of IO we are doing is causing us to at least be
partially bound by disk write latency with the single writer thread in
the rocksdb WAL though.
2. Ceph is unable to deliver more than 12k IOPS
per ceph-osd (may be
a little more with top-tier low-core high-frequency CPU, but not
much). So, super-duper-nvme wont make difference. (btw, I have a
stupid idea to try to run two ceph-osd from the same LV with a single
PV underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's
caches which internally has proven to help quite a bit with latency
bound workloads as it reduces lock contention in onode cache shards
and the impact of cache trimming (no more single trimming trim thread
constantly grabbing the lock for long periods of time!). In a 64 NVMe
drive setup (P4510s), we were able to do a little north of 400K write
IOPS with 3x replication, so about 19K IOPs per OSD once you factor
rep in. Also, in Nautilus you can see real benefits wtih running
multiple OSDs on a single device but with Octopus and master we've
pretty much closed the gap on our test setup:
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro…
Generally speaking using the latency-performance or latency-network
tuned profiles helps (mostly due to avoid C state CPU transitions) as
does higher clock speeds. Not using replication helps but that's
obviously not a realistic solution for most people. :)
I used size=1 and 'no ssd, no network' as upper bound. If allows to find
limits for ceph-osd performance. Any real-life things (replication,
network, real block devices) will make things worse, not better. Knowing
upper performance bound is really nice when start to choose server
configuration.
>
>> 3. You wll find that any given client performance is heavily limited
>> by sum of all RTT in the network, plus own latencies of ceph, so very
>> fast NVME give a diminishing return.
>> 4. CPU bounded ceph-osd completely wipe any differences for
>> underlying devices (except for desktop-class crawlers).
>>
>> You can run your own tests, even without fancy 48-nvme boxes - just
>> run ceph-osd on brd (block ram disk). ceph-osd won't run any faster
>> on anything else (ramdisk is the fastest), so numbers you get from
>> brd is supremum (upper bound) for theoretical performance.
>>
>> Given max 400-500% CPU per ceph-osd I'd say you need to keep number
>> of NVME in server below 12, or, 15 (but sometimes you'll get CPU
>> saturation).
>>
>> In my opinion less fancy boxes with smaller number of drives per
>> server (but larger number of servers) would make your (or your
>> operation team's) life much less stressful.
>
>
> That's pretty much the advice I've been giving people since the
> Inktank days. It costs more and is lower density, but the design is
> simpler, you are less likely to under provision CPU, less likely to
> run into memory bandwidth bottlenecks, and you have less recovery to
> do when a node fails. Especially now with how many NVMe drives you
> can fit in a single 1U server!
>
>
6:53 p.m.
On 9/23/20 8:23 AM, George Shuklin wrote:
I'd really like to see how they done this without offloading (their
configuration).
I went back and looked over some of the old results. I didn't find the
really high test scores (and now that I'm thinking about it they may
have been from when I was ripping out pglog OMAP updates!), but here's
one example I did find from earlier testing last winter that at least
got into roughly the right ballpark with stock master from last December
(~66K IOPS):
Avg 4K FIO randwrite IOPS: 65841.7
- 1 p4510 NVMe backed OSD
- 8GB osd memory target
- 4K min alloc size
- 4 clients, 1 128GB RBD volume per client, io_depth=128, time=300s
- 128 PGs (fixed)
- latency-network tuned profile
- bluestore_rocksdb_options =
"compression=kNoCompression,max_total_wal_size=1073741824,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=4,write_buffer_size=67108864,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,compaction_style=kCompactionStyleUniversal"
- bluestore_default_buffered_write = true
- bluestore_default_buffered_read = true
- rbd cache = false
Beyond that general stuff like background scrubbing and pg autoscaling
was disabled. I should note that these results are using universal
compaction in rocksdb which you probably don't want to do in production
because it can require 2x the total DB space to perform a compaction.
It might actually be feasible now that we are doing column family
sharding thanks to Adam's PR because you will only need 2x the space of
any individual column family for compaction rather than the whole DB,
but it's still unsupported for now.
Mark
It's octopus. I was doing single-osd benchmark, removing all movable
parts (brd instead of nvme, no network, size=1, etc). Moreover, I've
focused on rados benchmark, as RBD is just a derivative from rados
performance.
Anyway, big thank you for input.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I've
just finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers around 300-400% CPU per device.
In fact in isolation on the test setup that Intel donated for
community ceph R&D we've pushed a single OSD to consume around 1400%
CPU at 80K write IOPS! :) I agree though, we typical see a peak of
about 500-600% CPU per OSD on multi-node clusters with a
correspondingly lower write throughput. I do believe that in some
cases the mix of IO we are doing is causing us to at least be
partially bound by disk write latency with the single writer thread
in the rocksdb WAL though.
2. Ceph is unable to deliver more than 12k IOPS
per ceph-osd (may be
a little more with top-tier low-core high-frequency CPU, but not
much). So, super-duper-nvme wont make difference. (btw, I have a
stupid idea to try to run two ceph-osd from the same LV with a
single PV underneath VG, but it not tested).
I'm curious if you've tried octopus+ yet? We refactored bluestore's
caches which internally has proven to help quite a bit with latency
bound workloads as it reduces lock contention in onode cache shards
and the impact of cache trimming (no more single trimming trim thread
constantly grabbing the lock for long periods of time!). In a 64
NVMe drive setup (P4510s), we were able to do a little north of 400K
write IOPS with 3x replication, so about 19K IOPs per OSD once you
factor rep in. Also, in Nautilus you can see real benefits wtih
running multiple OSDs on a single device but with Octopus and master
we've pretty much closed the gap on our test setup:
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro…
Generally speaking using the latency-performance or latency-network
tuned profiles helps (mostly due to avoid C state CPU transitions) as
does higher clock speeds. Not using replication helps but that's
obviously not a realistic solution for most people. :)
I used size=1 and 'no ssd, no network' as upper bound. If allows to
find limits for ceph-osd performance. Any real-life things
(replication, network, real block devices) will make things worse, not
better. Knowing upper performance bound is really nice when start to
choose server configuration.
3. You wll find that any given client performance
is heavily limited
by sum of all RTT in the network, plus own latencies of ceph, so
very fast NVME give a diminishing return.
4. CPU bounded ceph-osd completely wipe any differences for
underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just
run ceph-osd on brd (block ram disk). ceph-osd won't run any faster
on anything else (ramdisk is the fastest), so numbers you get from
brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number
of NVME in server below 12, or, 15 (but sometimes you'll get CPU
saturation).
In my opinion less fancy boxes with smaller number of drives per
server (but larger number of servers) would make your (or your
operation team's) life much less stressful.
That's pretty much the advice I've been giving people since the
Inktank days. It costs more and is lower density, but the design is
simpler, you are less likely to under provision CPU, less likely to
run into memory bandwidth bottlenecks, and you have less recovery to
do when a node fails. Especially now with how many NVMe drives you
can fit in a single 1U server!
8:58 p.m.
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all
pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images.
I got 17k iops.
OK, in fact that's not the worst result for Ceph, but problem is that I only get 30k
write iops when benchmarking 4 RBD images spread over all OSDs _in_the_same_cluster_. And
there are 14 of them.
>> I've just finishing doing our own benchmarking, and I can say, you
>> want to do something very unbalanced and CPU bounded.
>>
>> 1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
>> ceph-osd at top-performance (see the recent thread on 'ceph on brd')
>> with more realistic numbers around 300-400% CPU per device.
>>> In fact in isolation on the test setup that Intel donated for
>>> community ceph R&D we've pushed a single OSD to consume around 1400%
>>> CPU at 80K write IOPS! :) I agree though, we typical see a peak of
>>> about 500-600% CPU per OSD on multi-node clusters with a
>>> correspondingly lower write throughput. I do believe that in some
>>> cases the mix of IO we are doing is causing us to at least be
>>> partially bound by disk write latency with the single writer thread
>>> in the rocksdb WAL though.
>>
>> I'd really like to see how they done this without offloading (their
>> configuration).
>
> I went back and looked over some of the old results. I didn't find the
> really high test scores (and now that I'm thinking about it they may
> have been from when I was ripping out pglog OMAP updates!), but here's
> one example I did find from earlier testing last winter that at least
> got into roughly the right ballpark with stock master from last December
> (~66K IOPS):
>
> Avg 4K FIO randwrite IOPS: 65841.7
>
> - 1 p4510 NVMe backed OSD
>
> - 8GB osd memory target
>
> - 4K min alloc size
>
> - 4 clients, 1 128GB RBD volume per client, io_depth=128, time=300s
>
> - 128 PGs (fixed)
>
> - latency-network tuned profile
>
> - bluestore_rocksdb_options =
>
"compression=kNoCompression,max_total_wal_size=1073741824,max_write_buffer_number=16,min_write_buffe
>
_number_to_merge=3,recycle_log_file_num=4,write_buffer_size=67108864,writable_file_max_buffer_size=0
>
compaction_readahead_size=2097152,max_background_compactions=2,compaction_style=kCompactionStyleUniv
> rsal"
>
> - bluestore_default_buffered_write = true
>
> - bluestore_default_buffered_read = true
>
> - rbd cache = false
>
> Beyond that general stuff like background scrubbing and pg autoscaling
> was disabled. I should note that these results are using universal
> compaction in rocksdb which you probably don't want to do in production
> because it can require 2x the total DB space to perform a compaction.
> It might actually be feasible now that we are doing column family
> sharding thanks to Adam's PR because you will only need 2x the space of
> any individual column family for compaction rather than the whole DB,
> but it's still unsupported for now.
>
> Mark
>
>>>
>>
>> 2. Ceph is unable to deliver more than 12k IOPS per ceph-osd (may be
>> a little more with top-tier low-core high-frequency CPU, but not
>> much). So, super-duper-nvme wont make difference. (btw, I have a
>> stupid idea to try to run two ceph-osd from the same LV with a
>> single PV underneath VG, but it not tested).
>>> I'm curious if you've tried octopus+ yet? We refactored
bluestore's
>>> caches which internally has proven to help quite a bit with latency
>>> bound workloads as it reduces lock contention in onode cache shards
>>> and the impact of cache trimming (no more single trimming trim thread
>>> constantly grabbing the lock for long periods of time!). In a 64
>>> NVMe drive setup (P4510s), we were able to do a little north of 400K
>>> write IOPS with 3x replication, so about 19K IOPs per OSD once you
>>> factor rep in. Also, in Nautilus you can see real benefits wtih
>>> running multiple OSDs on a single device but with Octopus and master
>>> we've pretty much closed the gap on our test setup:
>>
>> It's octopus. I was doing single-osd benchmark, removing all movable
>> parts (brd instead of nvme, no network, size=1, etc). Moreover, I've
>> focused on rados benchmark, as RBD is just a derivative from rados
>> performance.
>>
>> Anyway, big thank you for input.
>>
>>>
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro…
>>>
>>> Generally speaking using the latency-performance or latency-network
>>> tuned profiles helps (mostly due to avoid C state CPU transitions) as
>>> does higher clock speeds. Not using replication helps but that's
>>> obviously not a realistic solution for most people. :)
>>
>> I used size=1 and 'no ssd, no network' as upper bound. If allows to
>> find limits for ceph-osd performance. Any real-life things
>> (replication, network, real block devices) will make things worse, not
>> better. Knowing upper performance bound is really nice when start to
>> choose server configuration.
>>
>>>
>>
>> 3. You wll find that any given client performance is heavily limited
>> by sum of all RTT in the network, plus own latencies of ceph, so
>> very fast NVME give a diminishing return.
>> 4. CPU bounded ceph-osd completely wipe any differences for
>> underlying devices (except for desktop-class crawlers).
>>
>> You can run your own tests, even without fancy 48-nvme boxes - just
>> run ceph-osd on brd (block ram disk). ceph-osd won't run any faster
>> on anything else (ramdisk is the fastest), so numbers you get from
>> brd is supremum (upper bound) for theoretical performance.
>>
>> Given max 400-500% CPU per ceph-osd I'd say you need to keep number
>> of NVME in server below 12, or, 15 (but sometimes you'll get CPU
>> saturation).
>>
>> In my opinion less fancy boxes with smaller number of drives per
>> server (but larger number of servers) would make your (or your
>> operation team's) life much less stressful.
>>> That's pretty much the advice I've been giving people since the
>>> Inktank days. It costs more and is lower density, but the design is
>>> simpler, you are less likely to under provision CPU, less likely to
>>> run into memory bandwidth bottlenecks, and you have less recovery to
>>> do when a node fails. Especially now with how many NVMe drives you
>>> can fit in a single 1U server!
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:18 p.m.
On 9/23/20 10:58 AM, vitalif(a)yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD
)
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all
pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images.
I got 17k iops.
OK, in fact that's not the worst result for Ceph, but problem is that I only get 30k
write iops when benchmarking 4 RBD images spread over all OSDs _in_the_same_cluster_. And
there are 14 of them.
The key parts are the single OSD, the tuning, the fast HW, the massive
aggregate io depth, and the fact that it's a fresh OSD with only this
dataset sitting on it. Like I said, you can't get that when you go to a
multi-node setup with replication. The same hardware is only achieving
about 19K IOPS per OSD before replication in an 8 node, 64 NVMe, 3x rep
configuration. Feel free to look at the data from the run yourself
though (2.5MB tgz file):
https://drive.google.com/file/d/1SuFoJtxAauKTBjgrV9XB00_M3Yr9LwW1/view?usp=…
In the results directory you'll see the 4 separate copies of fio doing
4k random writes each to it's own 128GB dataset on it's own RBD volume
via the librbd engine with iodepth=128:
[nhm@o10 id-f7e4b44f]$ grep "randwrite," output.*.localhost
output.0.localhost:cbt-librbd/o10.front.sepia.ceph.com-0-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
output.1.localhost:cbt-librbd/o10.front.sepia.ceph.com-1-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
output.2.localhost:cbt-librbd/o10.front.sepia.ceph.com-2-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
output.3.localhost:cbt-librbd/o10.front.sepia.ceph.com-3-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
[nhm@o10 id-f7e4b44f]$ grep rbdname output.*.localhost
output.0.localhost: "rbdname" : "o10.front.sepia.ceph.com-0",
output.1.localhost: "rbdname" : "o10.front.sepia.ceph.com-1",
output.2.localhost: "rbdname" : "o10.front.sepia.ceph.com-2",
output.3.localhost: "rbdname" : "o10.front.sepia.ceph.com-3",
[nhm@o10 id-f7e4b44f]$ grep size output.*.localhost
output.0.localhost: "size" : "131072M",
output.1.localhost: "size" : "131072M",
output.2.localhost: "size" : "131072M",
output.3.localhost: "size" : "131072M",
[nhm@o10 id-f7e4b44f]$ grep IOPS output.*.localhost
output.0.localhost: write: IOPS=15.5k, BW=60.5MiB/s
(63.4MB/s)(17.7GiB/300003msec); 0 zone resets
output.1.localhost: write: IOPS=15.7k, BW=61.2MiB/s
(64.2MB/s)(17.9GiB/300005msec); 0 zone resets
output.2.localhost: write: IOPS=15.6k, BW=60.9MiB/s
(63.9MB/s)(17.8GiB/300005msec); 0 zone resets
output.3.localhost: write: IOPS=19.1k, BW=74.5MiB/s
(78.2MB/s)(21.8GiB/300006msec); 0 zone resets
timestamp is virtually identical:
[nhm@o10 id-f7e4b44f]$ grep timestamp_ms output.*.localhost
output.0.localhost: "timestamp_ms" : 1570222502962,
output.1.localhost: "timestamp_ms" : 1570222502965,
output.2.localhost: "timestamp_ms" : 1570222502965,
output.3.localhost: "timestamp_ms" : 1570222502959,
as is job runtime:
output.0.localhost: "job_runtime" : 300002,
output.1.localhost: "job_runtime" : 300004,
output.2.localhost: "job_runtime" : 300004,
output.3.localhost: "job_runtime" : 300005,
We can perform some sanity checks by looking at collectl data from the run:
[nhm@o10 id-f7e4b44f]$ collectl -sD -oT -p
collectl.localhost/o10-20191004-204957.raw.gz | grep -E "nvme0n1p1|#" |
tail -n 21
# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time Name KBytes Merged IOs Size Wait KBytes Merged IOs
Size Wait RWSize QLen Wait SvcTim Util
20:54:45 nvme0n1p1 0 0 0 0 0 227780 55001 1944
117 1 117 7 1 0 39
20:54:46 nvme0n1p1 0 0 0 0 0 485056 117319 3945
123 4 122 38 4 0 45
20:54:47 nvme0n1p1 0 0 0 0 0 409144 98947 3339
123 7 122 52 7 0 42
20:54:48 nvme0n1p1 0 0 0 0 0 384940 93056 3179
121 2 121 14 2 0 51
20:54:49 nvme0n1p1 0 0 0 0 0 499152 120724 4064
123 4 122 28 4 0 55
20:54:50 nvme0n1p1 0 0 0 0 0 388560 93931 3209
121 4 121 26 4 0 52
20:54:51 nvme0n1p1 0 0 0 0 0 316304 76415 2661
119 5 118 26 5 0 51
20:54:52 nvme0n1p1 0 0 0 0 0 418876 107305 3443
122 5 121 30 5 0 52
20:54:53 nvme0n1p1 0 0 0 0 0 367420 82728 3098
119 4 118 26 4 0 50
20:54:54 nvme0n1p1 0 0 0 0 0 293444 70912 2449
120 5 119 33 5 0 41
20:54:55 nvme0n1p1 0 0 0 0 0 300680 72687 2483
121 6 121 39 6 0 38
20:54:56 nvme0n1p1 0 0 0 0 0 274404 66311 2290
120 2 119 10 2 0 40
20:54:57 nvme0n1p1 0 0 0 0 0 244220 58996 2059
119 5 118 28 5 0 39
20:54:58 nvme0n1p1 0 0 0 0 0 242116 58491 2038
119 6 118 31 6 0 37
20:54:59 nvme0n1p1 0 0 0 0 0 436208 105512 3540
123 5 123 40 5 0 44
20:55:00 nvme0n1p1 0 0 0 0 0 427420 103459 3474
123 7 123 54 7 0 42
20:55:01 nvme0n1p1 0 0 0 0 0 152428 36677 1352
113 0 112 1 0 0 36
20:55:02 nvme0n1p1 0 0 0 0 0 448740 108542 3643
123 5 123 45 5 0 44
So in this case the block layer in the kernel was able to play some
tricks and merge a bunch of IOs helping us get that nice juicy benchmark
number (though these P4510 NVMe drives may have been able to handle more
randomly scattered non-merged 4K writes relatively well too). in this
case we actually have a number of IOs waiting in the device queue
meaning we are pushing the NVMe at least moderately hard. Next we look
at OSD process data over time:
[nhm@o10 id-f7e4b44f]$ collectl -sZ -oT -p
collectl.localhost/o10-20191004-204957.raw.gz | awk 'NR==5 ||
/ceph-osd/' | head -n 10
#Time PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct
AccuTime RKB WKB MajF MinF Command
20:50:08 7964 root 20 1 71 S 7G 6G 84 10.13 49.34 594
09:03.66 0 0 0 75K /usr/local/bin/ceph-osd
20:50:18 7964 root 20 1 71 S 7G 6G 84 15.26 84.28 995
10:43.21 0 0 0 42K /usr/local/bin/ceph-osd
20:50:28 7964 root 20 1 71 S 7G 6G 84 15.87 89.10
1049 12:28.18 0 0 0 59K /usr/local/bin/ceph-osd
20:50:38 7964 root 20 1 71 S 7G 6G 84 15.46 94.56
1100 14:18.20 0 0 0 66K /usr/local/bin/ceph-osd
20:50:48 7964 root 20 1 71 S 7G 6G 84 15.19 94.82
1100 16:08.21 0 0 0 76K /usr/local/bin/ceph-osd
20:50:58 7964 root 20 1 71 S 7G 6G 84 14.64 92.22
1068 17:55.07 0 0 0 52K /usr/local/bin/ceph-osd
20:51:08 7964 root 20 1 71 S 7G 6G 84 14.91 95.36
1102 19:45.34 0 0 0 55K /usr/local/bin/ceph-osd
20:51:18 7964 root 20 1 71 S 7G 6G 84 15.48 100.92
1164 21:41.74 0 0 0 64K /usr/local/bin/ceph-osd
20:51:28 7964 root 20 1 71 S 7G 6G 84 14.91 90.63
1055 23:27.28 0 0 0 53K /usr/local/bin/ceph-osd
The OSD process is consuming a little under 8GB which is reasonably on
target. PCT used by the OSD stabilizes between about 10-12 cores. The
important point is that the OSD/bluestore machinery is perfectly happy
consuming a ton of cores and driving ~66K IOPs worth of work at the
underlying device assuming the block layer and the device itself can
handle it. It used to be that the kv sync and onode cache triming
threads were much bigger bottlenecks but we've improved both over the
last couple of years. Having said that we're getting pretty close to
the current limits of bluestore here. Also once you start involving the
network and replication we introduce enough extra latency and contention
in our stack that the OSD can no longer maintain these kinds of numbers
even if the device itself and bluestore can handle it. That's where
seastore, crimson, and future msgr work come in imho. Finally, we can
use cbt's osd log parser to look at rocksdb compaction statistics. This
also probably explains at least some of the large IO workload we see in
collectl since we are spending nearly 1/3rd of the overall OSD uptime in
compaction (and given that the test only ran for 300s of that uptime,
rocksdb may be spending the majority of the actual test portion of the
OSD uptime performing universal compaction):
[nhm@o10 nvme_nvme_bs]$ ~/src/ceph/cbt/tools/ceph_rocksdb_log_parser.py
./osd.0.log | head -n 13
Compaction Statistics ./osd.0.log
Total OSD Log Duration (seconds) 605.512
Number of Compaction Events 112
Avg Compaction Time (seconds) 1.68255873214
Total Compaction Time (seconds) 188.446578
Avg Output Size: (MB) 380.801597553
Total Output Size: (MB) 42649.7789259
Total Input Records 243648735
Total Output Records 174897784
Avg Output Throughput (MB/s) 250.561577229
Avg Input Records/second 1242038.3797
Avg Output Records/second 1059967.66123
Avg Output/Input Ratio 0.862494228971
Mark
10:33 p.m.
On 9/23/20 12:18 PM, Mark Nelson wrote:
On 9/23/20 10:58 AM, vitalif(a)yourcmc.ru wrote:
Minor correction, nvme0n1p1 was a DB/WAL partition while nvme1n1p2 is
the block storage device where the 4k writes are happening and
unmerged. That makes way more sense since those merged writes are all
just the WAL and big SST writes. Sorry, it's been a while since I
looked at these numbers. IE here we are using a separate NVMe drive in
this case for the DB/WAL. In reality I don't recall it actually helping
that much vs just putting everything on a single P4510, but in any event
I wanted to update for clarity.
Actual block partition writes:
[nhm@o10 id-f7e4b44f]$ collectl -sD -oT -p
collectl.localhost/o10-20191004-204957.raw.gz | grep -E "nvme1n1p2|#" |
tail -n 21
# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time Name KBytes Merged IOs Size Wait KBytes Merged IOs
Size Wait RWSize QLen Wait SvcTim Util
20:54:45 nvme1n1p2 0 0 0 0 0 261804 0 65K
4 0 3 0 0 0 89
20:54:46 nvme1n1p2 0 0 0 0 0 206916 0 51K
4 0 4 0 0 0 67
20:54:47 nvme1n1p2 0 0 0 0 0 218888 0 54K
4 0 4 0 0 0 75
20:54:48 nvme1n1p2 0 0 0 0 0 261032 0 65K
4 0 4 0 0 0 79
20:54:49 nvme1n1p2 0 0 0 0 0 237132 0 59K
4 0 4 0 0 0 70
20:54:50 nvme1n1p2 0 0 0 0 0 269404 0 67K
4 0 4 0 0 0 80
20:54:51 nvme1n1p2 0 0 0 0 0 280064 0 70K
4 0 4 0 0 0 82
20:54:52 nvme1n1p2 0 0 0 0 0 251956 0 62K
4 0 4 0 0 0 75
20:54:53 nvme1n1p2 0 0 0 0 0 267232 0 66K
4 0 4 0 0 0 82
20:54:54 nvme1n1p2 0 0 0 0 0 263884 0 65K
4 0 4 0 0 0 83
20:54:55 nvme1n1p2 0 0 0 0 0 245224 0 61K
4 0 4 0 0 0 85
20:54:56 nvme1n1p2 0 0 0 0 0 262472 0 65K
4 0 4 0 0 0 86
20:54:57 nvme1n1p2 0 0 0 0 0 270272 0 67K
4 0 4 0 0 0 89
20:54:58 nvme1n1p2 0 0 0 0 0 264256 0 66K
4 0 4 0 0 0 90
20:54:59 nvme1n1p2 0 0 0 0 0 226052 0 56K
4 0 4 0 0 0 75
20:55:00 nvme1n1p2 0 0 0 0 0 224352 0 56K
4 0 4 0 0 0 76
20:55:01 nvme1n1p2 0 0 0 0 0 287760 0 71K
4 0 4 0 0 0 96
20:55:02 nvme1n1p2 0 0 0 0 0 220128 0 55K
4 0 4 0 0 0 73
Mark
I have no idea how you get 66k write iops with
one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD
with 8 PGs (all pinned to the same OSD with pg-upmap). Then I ran 4x
fio randwrite q128 over 4 RBD images. I got 17k iops.
OK, in fact that's not the worst result for Ceph, but problem is that
I only get 30k write iops when benchmarking 4 RBD images spread over
all OSDs _in_the_same_cluster_. And there are 14 of them.
The key parts are the single OSD, the tuning, the fast HW, the massive
aggregate io depth, and the fact that it's a fresh OSD with only this
dataset sitting on it. Like I said, you can't get that when you go to
a multi-node setup with replication. The same hardware is only
achieving about 19K IOPS per OSD before replication in an 8 node, 64
NVMe, 3x rep configuration. Feel free to look at the data from the
run yourself though (2.5MB tgz file):
https://drive.google.com/file/d/1SuFoJtxAauKTBjgrV9XB00_M3Yr9LwW1/view?usp=…
In the results directory you'll see the 4 separate copies of fio doing
4k random writes each to it's own 128GB dataset on it's own RBD volume
via the librbd engine with iodepth=128:
[nhm@o10 id-f7e4b44f]$ grep "randwrite," output.*.localhost
output.0.localhost:cbt-librbd/o10.front.sepia.ceph.com-0-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
output.1.localhost:cbt-librbd/o10.front.sepia.ceph.com-1-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
output.2.localhost:cbt-librbd/o10.front.sepia.ceph.com-2-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
output.3.localhost:cbt-librbd/o10.front.sepia.ceph.com-3-0: (g=0):
rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=rbd, iodepth=128
[nhm@o10 id-f7e4b44f]$ grep rbdname output.*.localhost
output.0.localhost: "rbdname" : "o10.front.sepia.ceph.com-0",
output.1.localhost: "rbdname" : "o10.front.sepia.ceph.com-1",
output.2.localhost: "rbdname" : "o10.front.sepia.ceph.com-2",
output.3.localhost: "rbdname" : "o10.front.sepia.ceph.com-3",
[nhm@o10 id-f7e4b44f]$ grep size output.*.localhost
output.0.localhost: "size" : "131072M",
output.1.localhost: "size" : "131072M",
output.2.localhost: "size" : "131072M",
output.3.localhost: "size" : "131072M",
[nhm@o10 id-f7e4b44f]$ grep IOPS output.*.localhost
output.0.localhost: write: IOPS=15.5k, BW=60.5MiB/s
(63.4MB/s)(17.7GiB/300003msec); 0 zone resets
output.1.localhost: write: IOPS=15.7k, BW=61.2MiB/s
(64.2MB/s)(17.9GiB/300005msec); 0 zone resets
output.2.localhost: write: IOPS=15.6k, BW=60.9MiB/s
(63.9MB/s)(17.8GiB/300005msec); 0 zone resets
output.3.localhost: write: IOPS=19.1k, BW=74.5MiB/s
(78.2MB/s)(21.8GiB/300006msec); 0 zone resets
timestamp is virtually identical:
[nhm@o10 id-f7e4b44f]$ grep timestamp_ms output.*.localhost
output.0.localhost: "timestamp_ms" : 1570222502962,
output.1.localhost: "timestamp_ms" : 1570222502965,
output.2.localhost: "timestamp_ms" : 1570222502965,
output.3.localhost: "timestamp_ms" : 1570222502959,
as is job runtime:
output.0.localhost: "job_runtime" : 300002,
output.1.localhost: "job_runtime" : 300004,
output.2.localhost: "job_runtime" : 300004,
output.3.localhost: "job_runtime" : 300005,
We can perform some sanity checks by looking at collectl data from the
run:
[nhm@o10 id-f7e4b44f]$ collectl -sD -oT -p
collectl.localhost/o10-20191004-204957.raw.gz | grep -E "nvme0n1p1|#"
| tail -n 21
# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time Name KBytes Merged IOs Size Wait KBytes Merged IOs
Size Wait RWSize QLen Wait SvcTim Util
20:54:45 nvme0n1p1 0 0 0 0 0 227780 55001
1944 117 1 117 7 1 0 39
20:54:46 nvme0n1p1 0 0 0 0 0 485056 117319
3945 123 4 122 38 4 0 45
20:54:47 nvme0n1p1 0 0 0 0 0 409144 98947
3339 123 7 122 52 7 0 42
20:54:48 nvme0n1p1 0 0 0 0 0 384940 93056
3179 121 2 121 14 2 0 51
20:54:49 nvme0n1p1 0 0 0 0 0 499152 120724
4064 123 4 122 28 4 0 55
20:54:50 nvme0n1p1 0 0 0 0 0 388560 93931
3209 121 4 121 26 4 0 52
20:54:51 nvme0n1p1 0 0 0 0 0 316304 76415
2661 119 5 118 26 5 0 51
20:54:52 nvme0n1p1 0 0 0 0 0 418876 107305
3443 122 5 121 30 5 0 52
20:54:53 nvme0n1p1 0 0 0 0 0 367420 82728
3098 119 4 118 26 4 0 50
20:54:54 nvme0n1p1 0 0 0 0 0 293444 70912
2449 120 5 119 33 5 0 41
20:54:55 nvme0n1p1 0 0 0 0 0 300680 72687
2483 121 6 121 39 6 0 38
20:54:56 nvme0n1p1 0 0 0 0 0 274404 66311
2290 120 2 119 10 2 0 40
20:54:57 nvme0n1p1 0 0 0 0 0 244220 58996
2059 119 5 118 28 5 0 39
20:54:58 nvme0n1p1 0 0 0 0 0 242116 58491
2038 119 6 118 31 6 0 37
20:54:59 nvme0n1p1 0 0 0 0 0 436208 105512
3540 123 5 123 40 5 0 44
20:55:00 nvme0n1p1 0 0 0 0 0 427420 103459
3474 123 7 123 54 7 0 42
20:55:01 nvme0n1p1 0 0 0 0 0 152428 36677
1352 113 0 112 1 0 0 36
20:55:02 nvme0n1p1 0 0 0 0 0 448740 108542
3643 123 5 123 45 5 0 44
So in this case the block layer in the kernel was able to play some
tricks and merge a bunch of IOs helping us get that nice juicy
benchmark number (though these P4510 NVMe drives may have been able to
handle more randomly scattered non-merged 4K writes relatively well
too). in this case we actually have a number of IOs waiting in the
device queue meaning we are pushing the NVMe at least moderately
hard. Next we look at OSD process data over time:
[nhm@o10 id-f7e4b44f]$ collectl -sZ -oT -p
collectl.localhost/o10-20191004-204957.raw.gz | awk 'NR==5 ||
/ceph-osd/' | head -n 10
#Time PID User PR PPID THRD S VSZ RSS CP SysT UsrT
Pct AccuTime RKB WKB MajF MinF Command
20:50:08 7964 root 20 1 71 S 7G 6G 84 10.13 49.34
594 09:03.66 0 0 0 75K /usr/local/bin/ceph-osd
20:50:18 7964 root 20 1 71 S 7G 6G 84 15.26 84.28
995 10:43.21 0 0 0 42K /usr/local/bin/ceph-osd
20:50:28 7964 root 20 1 71 S 7G 6G 84 15.87 89.10
1049 12:28.18 0 0 0 59K /usr/local/bin/ceph-osd
20:50:38 7964 root 20 1 71 S 7G 6G 84 15.46 94.56
1100 14:18.20 0 0 0 66K /usr/local/bin/ceph-osd
20:50:48 7964 root 20 1 71 S 7G 6G 84 15.19 94.82
1100 16:08.21 0 0 0 76K /usr/local/bin/ceph-osd
20:50:58 7964 root 20 1 71 S 7G 6G 84 14.64 92.22
1068 17:55.07 0 0 0 52K /usr/local/bin/ceph-osd
20:51:08 7964 root 20 1 71 S 7G 6G 84 14.91 95.36
1102 19:45.34 0 0 0 55K /usr/local/bin/ceph-osd
20:51:18 7964 root 20 1 71 S 7G 6G 84 15.48 100.92
1164 21:41.74 0 0 0 64K /usr/local/bin/ceph-osd
20:51:28 7964 root 20 1 71 S 7G 6G 84 14.91 90.63
1055 23:27.28 0 0 0 53K /usr/local/bin/ceph-osd
The OSD process is consuming a little under 8GB which is reasonably on
target. PCT used by the OSD stabilizes between about 10-12 cores.
The important point is that the OSD/bluestore machinery is perfectly
happy consuming a ton of cores and driving ~66K IOPs worth of work at
the underlying device assuming the block layer and the device itself
can handle it. It used to be that the kv sync and onode cache triming
threads were much bigger bottlenecks but we've improved both over the
last couple of years. Having said that we're getting pretty close to
the current limits of bluestore here. Also once you start involving
the network and replication we introduce enough extra latency and
contention in our stack that the OSD can no longer maintain these
kinds of numbers even if the device itself and bluestore can handle
it. That's where seastore, crimson, and future msgr work come in
imho. Finally, we can use cbt's osd log parser to look at rocksdb
compaction statistics. This also probably explains at least some of
the large IO workload we see in collectl since we are spending nearly
1/3rd of the overall OSD uptime in compaction (and given that the test
only ran for 300s of that uptime, rocksdb may be spending the majority
of the actual test portion of the OSD uptime performing universal
compaction):
[nhm@o10 nvme_nvme_bs]$
~/src/ceph/cbt/tools/ceph_rocksdb_log_parser.py ./osd.0.log | head -n 13
Compaction Statistics ./osd.0.log
Total OSD Log Duration (seconds) 605.512
Number of Compaction Events 112
Avg Compaction Time (seconds) 1.68255873214
Total Compaction Time (seconds) 188.446578
Avg Output Size: (MB) 380.801597553
Total Output Size: (MB) 42649.7789259
Total Input Records 243648735
Total Output Records 174897784
Avg Output Throughput (MB/s) 250.561577229
Avg Input Records/second 1242038.3797
Avg Output Records/second 1059967.66123
Avg Output/Input Ratio 0.862494228971
Mark
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
24 Sep
24 Sep
9:46 p.m.
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it
myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658%
CPU and pushing only 4138 write iops...
10:33 p.m.
On 9/24/20 11:46 AM, vitalif(a)yourcmc.ru wrote:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I see it
myself on my machine. :-))
It's better to see evidence with your own eyes of course!
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating 658%
CPU and pushing only 4138 write iops...
If you have the time to investigate, you might try gdbpmp or Adam's
wallclock profiler to see where the OSD is spending it's time. That
might help diagnose where the OSD is bottlenecked. I would certainly be
interested in the results. We have quite fast systems in the lab but
very limited diversity of hardware.
11:43 p.m.
I did not see 10 cores, but 7 cores per osd over a long period on pm1725a
disks with around 60k IO/s according to sysstat of each disk.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif(a)yourcmc.ru>ru>:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it until I
see it myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It was eating
658% CPU and pushing only 4138 write iops...
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Sep
25 Sep
12:01 a.m.
Mind if I ask what size of IOs those where, what kind of IOs
(reads/writes/sequential/random?) and what kind of cores?
Mark
On 9/24/20 1:43 PM, Martin Verges wrote:
I did not see 10 cores, but 7 cores per osd over a
long period on
pm1725a disks with around 60k IO/s according to sysstat of each disk.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif(a)yourcmc.ru
<mailto:vitalif@yourcmc.ru>>:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't believe it
until I see it myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It
was eating 658% CPU and pushing only 4138 write iops...
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
12:49 a.m.
Hello,
It was some time ago but as far as I remember and found in the chat log, it
was during backfill/recovery and high client workload and on Intel Xeon
Silver 4110, 2.10GHz, 8C/16T Cpu.
I found a screenshot in my chat history stating 775% and 722% cpu usage in
htop for 2 OSDs (the server has 2 PCIe PM1725a NVMe OSDs and 12 HDD OSDs).
Unfortunately I have no console log output that would show more details
like IO pattern.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 21:01 Uhr schrieb Mark Nelson <mnelson(a)redhat.com
:
> Mind if I ask what size of IOs those where, what kind of IOs
> (reads/writes/sequential/random?) and what kind of cores?
>
>
> Mark
>
>
> On 9/24/20 1:43 PM, Martin Verges wrote:
> > I did not see 10 cores, but 7 cores per osd over a long period on
> > pm1725a disks with around 60k IO/s according to sysstat of each disk.
> >
> > --
> > Martin Verges
> > Managing director
> >
> > Mobile: +49 174 9335695
> > E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
> > Chat: https://t.me/MartinVerges
> >
> > croit GmbH, Freseniusstr. 31h, 81247 Munich
> > CEO: Martin Verges - VAT-ID: DE310638492
> > Com. register: Amtsgericht Munich HRB 231263
> >
> > Web: https://croit.io
> > YouTube: https://goo.gl/PGE1Bx
> >
> >
> > Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif(a)yourcmc.ru
> > <mailto:vitalif@yourcmc.ru> :
> >
> > OK, I'll retry my tests several times more.
> >
> > But I've never seen OSD utilize 10 cores, so... I won't believe it
> > until I see it myself on my machine. :-))
> >
> > I tried a fresh OSD on a block ramdisk ("brd"), for example. It
> > was eating 658% CPU and pushing only 4138 write iops...
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > <mailto:ceph-users-leave@ceph.io>
> >
>
>
12:54 a.m.
Thanks for the info! Interesting numbers. Probably not 60K client
IOPs/OSD then, but the tp_osd_tp threads were probably working pretty
hard under the combined client/recovery workload.
Mark
On 9/24/20 2:49 PM, Martin Verges wrote:
Hello,
It was some time ago but as far as I remember and found in the chat
log, it was during backfill/recovery and high client workload and on
Intel Xeon Silver 4110, 2.10GHz, 8C/16T Cpu.
I found a screenshot in my chat history stating 775% and 722% cpu
usage in htop for 2 OSDs (the server has 2 PCIe PM1725a NVMe OSDs and
12 HDD OSDs).
Unfortunately I have no console log output that would show more
details like IO pattern.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 21:01 Uhr schrieb Mark Nelson
<mnelson(a)redhat.com <mailto:mnelson@redhat.com>>:
Mind if I ask what size of IOs those where, what kind of IOs
(reads/writes/sequential/random?) and what kind of cores?
Mark
On 9/24/20 1:43 PM, Martin Verges wrote:
I did not see 10 cores, but 7 cores per osd over
a long period on
pm1725a disks with around 60k IO/s according to sysstat of each
disk.
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io <mailto:martin.verges@croit.io>
<mailto:martin.verges@croit.io <mailto:martin.verges@croit.io>>
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Do., 24. Sept. 2020 um 18:47 Uhr schrieb <vitalif(a)yourcmc.ru
<mailto:vitalif@yourcmc.ru>
<mailto:vitalif@yourcmc.ru
<mailto:vitalif@yourcmc.ru>>>:
OK, I'll retry my tests several times more.
But I've never seen OSD utilize 10 cores, so... I won't
believe
it
until I see it myself on my machine. :-))
I tried a fresh OSD on a block ramdisk ("brd"), for example. It
was eating 658% CPU and pushing only 4138 write iops...
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
12:04 a.m.
Yeah but you should divide sysstat of each disk by 5. Which is Ceph's WA.
60k/5 = 12k external iops, pretty realistic.
> I did not see 10 cores, but 7 cores per osd over a long period on pm1725a disks with
around 60k
> IO/s according to sysstat of each disk.
24 Sep
24 Sep
2:58 a.m.
On 23/09/2020 17:58, vitalif(a)yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD
)
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs (all
pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128 over 4 RBD images.
I got 17k iops.
8 PGs is a low number, there will be a lot of PG lock contention across
your 4x 128 queue depth.
OK, in fact that's not the worst result for Ceph,
but problem is that I only get 30k write iops when benchmarking 4 RBD images spread over
all OSDs _in_the_same_cluster_. And there are 14 of them.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I've
just finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers around 300-400% CPU per device.
I went back and looked over some of the old results. I didn't
find the
really high test scores (and now that I'm thinking about it they may
have been from when I was ripping out pglog OMAP updates!), but here's
one example I did find from earlier testing last winter that at least
got into roughly the right ballpark with stock master from last December
(~66K IOPS):
Avg 4K FIO randwrite IOPS: 65841.7
- 1 p4510 NVMe backed OSD
- 8GB osd memory target
- 4K min alloc size
- 4 clients, 1 128GB RBD volume per client, io_depth=128, time=300s
- 128 PGs (fixed)
- latency-network tuned profile
- bluestore_rocksdb_options =
"compression=kNoCompression,max_total_wal_size=1073741824,max_write_buffer_number=16,min_write_buffe
_number_to_merge=3,recycle_log_file_num=4,write_buffer_size=67108864,writable_file_max_buffer_size=0
compaction_readahead_size=2097152,max_background_compactions=2,compaction_style=kCompactionStyleUniv
rsal"
- bluestore_default_buffered_write = true
- bluestore_default_buffered_read = true
- rbd cache = false
Beyond that general stuff like background scrubbing and pg autoscaling
was disabled. I should note that these results are using universal
compaction in rocksdb which you probably don't want to do in production
because it can require 2x the total DB space to perform a compaction.
It might actually be feasible now that we are doing column family
sharding thanks to Adam's PR because you will only need 2x the space of
any individual column family for compaction rather than the whole DB,
but it's still unsupported for now.
Mark
In fact in isolation on the test setup that Intel
donated for
community ceph R&D we've pushed a single OSD to consume around 1400%
CPU at 80K write IOPS! :) I agree though, we typical see a peak of
about 500-600% CPU per OSD on multi-node clusters with a
correspondingly lower write throughput. I do believe that in some
cases the mix of IO we are doing is causing us to at least be
partially bound by disk write latency with the single writer thread
in the rocksdb WAL though.
I'd really like to see how they done this without
offloading (their
configuration). 2. Ceph is unable to deliver more than 12k IOPS
per ceph-osd (may be
a little more with top-tier low-core high-frequency CPU, but not
much). So, super-duper-nvme wont make difference. (btw, I have a
stupid idea to try to run two ceph-osd from the same LV with a
single PV underneath VG, but it not tested).
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io I'm curious if you've tried octopus+ yet?
We refactored bluestore's
caches which internally has proven to help quite a bit with latency
bound workloads as it reduces lock contention in onode cache shards
and the impact of cache trimming (no more single trimming trim thread
constantly grabbing the lock for long periods of time!). In a 64
NVMe drive setup (P4510s), we were able to do a little north of 400K
write IOPS with 3x replication, so about 19K IOPs per OSD once you
factor rep in. Also, in Nautilus you can see real benefits wtih
running multiple OSDs on a single device but with Octopus and master
we've pretty much closed the gap on our test setup:
It's octopus. I was
doing single-osd benchmark, removing all movable
parts (brd instead of nvme, no network, size=1, etc). Moreover, I've
focused on rados benchmark, as RBD is just a derivative from rados
performance.
Anyway, big thank you for input.
https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro…
Generally speaking using the latency-performance or latency-network
tuned profiles helps (mostly due to avoid C state CPU transitions) as
does higher clock speeds. Not using replication helps but that's
obviously not a realistic solution for most people. :)
I used size=1 and 'no
ssd, no network' as upper bound. If allows to
find limits for ceph-osd performance. Any real-life things
(replication, network, real block devices) will make things worse, not
better. Knowing upper performance bound is really nice when start to
choose server configuration.
3. You wll find that any given client performance is heavily limited
by sum of all RTT in the network, plus own latencies of ceph, so
very fast NVME give a diminishing return.
4. CPU bounded ceph-osd completely wipe any differences for
underlying devices (except for desktop-class crawlers).
You can run your own tests, even without fancy 48-nvme boxes - just
run ceph-osd on brd (block ram disk). ceph-osd won't run any faster
on anything else (ramdisk is the fastest), so numbers you get from
brd is supremum (upper bound) for theoretical performance.
Given max 400-500% CPU per ceph-osd I'd say you need to keep number
of NVME in server below 12, or, 15 (but sometimes you'll get CPU
saturation).
In my opinion less fancy boxes with smaller number of drives per
server (but larger number of servers) would make your (or your
operation team's) life much less stressful.
That's pretty much the advice I've been
giving people since the
Inktank days. It costs more and is lower density, but the design is
simpler, you are less likely to under provision CPU, less likely to
run into memory bandwidth bottlenecks, and you have less recovery to
do when a node fails. Especially now with how many NVMe drives you
can fit in a single 1U server!
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Sep
23 Sep
8:11 p.m.
That's pretty much the advice I've been giving
people since the Inktank days. It costs more and is lower density, but the design is
simpler, you are less likely to under provision CPU, less likely to run into memory
bandwidth bottlenecks, and you have less recovery to do when a node fails.
Agreed. I’ve seen very clever presentations extolling the benefits of pinning within,
say, a 4-socket server — OSDs to cores to HBAs to NICs to NUMA domains. A lot of the
diagrams start looking like 4x 1-socket servers in the same chassis, but with more work.
With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense
probably is fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-…
Especially now with how many NVMe drives you can fit
in a single 1U server!
I’ve seen 10 for a conventional server, though depending on CPU choice some of those
options don’t treat dual PSUs as redundant. EDSFF, the “ruler” form factor, shows a lot
of promise in this space. Especially once the drives are available from more than one
manufacturer. With TLC flash and the right Epyc P CPU it seems like a killer OSD node for
RBD use. And for some Object / RGW use cases QLC drives start looking like a viable
alternative to HDDs.
24 Sep
24 Sep
12:21 a.m.
On Wed, Sep 23, 2020 at 8:12 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote:
With today’s networking, _maybe_ a super-dense NVMe
box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI
lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-…
I was able to reach 35 Gb/s network traffic on each server (5 servers,
with 6 NVMEs per server, one OSD per NVME) during a read benchmark
from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s
may be a bit too tight.
--
Alexander E. Patrakov
CV: http://pc.cd/PLz7
12:53 a.m.
On 9/23/20 2:21 PM, Alexander E. Patrakov wrote:
On Wed, Sep 23, 2020 at 8:12 PM Anthony D'Atri
<anthony.datri(a)gmail.com> wrote:
FWIW, in the nodes I mentioned earlier, we have 8 P4510 NVMe drives and
4x25GbE ports. We can do about 60-70GB/s (even some times closer to
80GB/s) per server depending on the exact workload and network/client
setup. I suspect it should be possible to saturate 100GbE in ideal
setups and maybe even semi-utilize 200GbE (especially for reads) in the
future. Sadly we don't have any in-house to test right now.
Mark
With today’s networking, _maybe_ a super-dense
NVMe box needs 100Gb/s where a less-dense probably is fine with 25Gb/s. And of course PCI
lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-…
I was able to reach 35 Gb/s network traffic on each server (5 servers,
with 6 NVMEs per server, one OSD per NVME) during a read benchmark
from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s
may be a bit too tight.
--
Alexander E. Patrakov
CV: http://pc.cd/PLz7
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:39 a.m.
With today’s
networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a less-dense probably is
fine with 25Gb/s. And of course PCI lanes.
https://cephalocon2019.sched.com/event/M7uJ/affordable-nvme-performance-on-…
I was able to reach 35 Gb/s network traffic on each server (5 servers,
with 6 NVMEs per server, one OSD per NVME) during a read benchmark
from cephfs, and I wouldn't treat that as a super-dense box. So 25Gb/s
may be a bit too tight. The workload doesn’t demand NVMe performance, so SSD
seems to be the most cost effective way to handle this.
To be pedantic, NVMe devices *are* SSDs, but you most likely mean SATA SSDs.
The thing is, with recent drives, chassis, and CPU models, it can be very possible to
provision an NVMe server at a cost comparable to a conventional SATA SSD server, in which
case, why not?
— aad
1320
days inactive
1321
days old
33 comments
12 participants
participants (12)
-
Alexander E. Patrakov -
André Gemünd -
Anthony D'Atri -
Brent Kennedy -
George Shuklin -
Maged Mokhtar -
Marc Roos -
Mark Nelson -
Martin Verges -
Stefan Kooman -
tri＠postix.net -
vitalif＠yourcmc.ru