I think this depends on the type of backing disk. We use the following CPUs: Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz My experience is, that a HDD OSD hardly gets to 100% of 1 hyper thread load even under heavy recovery/rebalance operations on 8+2 and 6+2 EC pools with compression set to aggressive. The CPU is mostly doing wait-IO, that is, the disk is the real bottle neck, not the processor power. With SSDs I have seen 2HT at 100% and 2 more at 50% each. I guess NVMe might be more demanding. A server with 12 HDD and 1 SSD should be fine with a modern CPU with 8 cores. 16 threads sounds like an 8 core CPU. The 2nd generation Intel® Xeon® Silver 4209T with 8 cores should easily handle that (single socket system). We have the 16-core Intel silver in a dual socket system currently connected to 5HDD and 7SSD and I did a rebalance operation yesterday. The CPU user load did not exceed 2%, it can handle OSD processes easily. The server is dimensioned to run up to 12HDD and 14SSD OSDs (Dell R740xd2). As far as I can tell, the CPU configuration is overpowered for that. Just for info, we use ganglia to record node utilisation. I use 1-year records and pick peak loads I observed for dimensioning the CPUs. These records include some very heavy recovery periods. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 04:57:53 To: Nathan Fish Cc: ceph-users@ceph.io Subject: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? Thanks Nathan! Tony

-----Original Message----- From: Nathan Fish <lordcirth@gmail.com> Sent: Thursday, November 12, 2020 7:43 PM To: Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] which of cpu frequency and number of threads servers osd better?

From what I've seen, OSD daemons tend to bottleneck on the first 2 threads, while getting some use out of another 2. So 32 threads at 3.0 would be a lot better. Note that you may get better performance splitting off some of that SSD for block.db partitions or at least block.wal for the HDDs.

On Thu, Nov 12, 2020 at 9:57 PM Tony Liu <tonyliu0592@hotmail.com> wrote:

...

Hi,

For example, 16 threads with 3.2GHz and 32 threads with 3.0GHz, which makes 11 OSDs (10x12TB HDD and 1x960GB SSD) with better performance?

Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

If each OSD requires 4T

Nobody said that. What was said is HDD=1T, SSD=3T. It depends on the drive type! The %-utilisation information is just from top observed during heavy load. It does not show how the kernel schedules things on physical Ts. So, 2x50% utilisation could run on the same HT. I don't know how the OSDs are organised into threads, I'm just stating observations from real life (mimic cluster). So, for an SSD OSD I have seen a maximum of 4 threads in R state, two with 100% and two with 50% CPU, a load that fits on 3HT. So, real life says 1HT per HDD and 3HT per SSD plus a bit for kernel and networking and you are set - based on worst-case performance monitoring I have seen in 2 years. Note that this is worst-case load. The average load is much lower. A 16 core machine is totally overpowered. Assuming 1C=2HT, I count (2*3+8*1)/2=7 or (1*3+10*1)/2=6.5. So an 8 core CPU should do in either case. A 10 core CPU might be better, but 16C is a waste of money. I should mention that these estimates apply to Intel CPUs (x86_64 architectures). Other architectures might not provide the same cycle efficiency. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 08:32:55 To: Frank Schilder; Nathan Fish Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? You all mentioned first 2T and another 2T. Could you give more details how OSD works with multi-thread, or share the link if it's already documented somewhere? Is it always 4T, or start with 1T and grow up to 4T? Is it max 4T? Does each T run different job or just multiple instances of the same job? Does disk type affect how T works, like 1T is good enough for HDD while 4T is required for SSD? If I change my plan to 2 SSD OSDs and 8 HDD OSDs (with 1 SSD for WAL and DB). If each OSD requires 4T, then 16C/32T 3.0GHz could be a better choice, because it provides sufficient Ts? If SSD OSD requires 4T and HDD OSD only requires 1T, then 8C/16T 3.2GHz would be better, because it provides sufficient Ts as well as stronger computing? Thanks! Tony

-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Thursday, November 12, 2020 10:59 PM To: Tony Liu <tonyliu0592@hotmail.com>; Nathan Fish <lordcirth@gmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

I think this depends on the type of backing disk. We use the following CPUs:

Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz

My experience is, that a HDD OSD hardly gets to 100% of 1 hyper thread load even under heavy recovery/rebalance operations on 8+2 and 6+2 EC pools with compression set to aggressive. The CPU is mostly doing wait- IO, that is, the disk is the real bottle neck, not the processor power. With SSDs I have seen 2HT at 100% and 2 more at 50% each. I guess NVMe might be more demanding.

A server with 12 HDD and 1 SSD should be fine with a modern CPU with 8 cores. 16 threads sounds like an 8 core CPU. The 2nd generation Intel® Xeon® Silver 4209T with 8 cores should easily handle that (single socket system). We have the 16-core Intel silver in a dual socket system currently connected to 5HDD and 7SSD and I did a rebalance operation yesterday. The CPU user load did not exceed 2%, it can handle OSD processes easily. The server is dimensioned to run up to 12HDD and 14SSD OSDs (Dell R740xd2). As far as I can tell, the CPU configuration is overpowered for that.

Just for info, we use ganglia to record node utilisation. I use 1-year records and pick peak loads I observed for dimensioning the CPUs. These records include some very heavy recovery periods.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 04:57:53 To: Nathan Fish Cc: ceph-users@ceph.io Subject: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

...

-----Original Message----- From: Nathan Fish <lordcirth@gmail.com> Sent: Thursday, November 12, 2020 7:43 PM To: Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] which of cpu frequency and number of threads servers osd better?

From what I've seen, OSD daemons tend to bottleneck on the first 2 threads, while getting some use out of another 2. So 32 threads at 3.0 would be a lot better. Note that you may get better performance splitting off some of that SSD for block.db partitions or at least block.wal for the HDDs.

On Thu, Nov 12, 2020 at 9:57 PM Tony Liu <tonyliu0592@hotmail.com> wrote:

...

Hi,

For example, 16 threads with 3.2GHz and 32 threads with 3.0GHz, which makes 11 OSDs (10x12TB HDD and 1x960GB SSD) with better

Thanks Nathan! Tony performance?

...

...

Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

We have 12TB HDD OSDs with 32GiB of (Optane) NVMe for block.db, used for cephfs_data pools, and NVMe-only OSDs used for cephfs_data pools. The NVMe DB about doubled our random IO performance - a great investment - doubling max CPU load as a result. We had to turn up "osd op num threads per shard hdd" from 1 to 2. (2 is the default for SSDs). This didn't noticeably improve performance, but without it, OSDs under max load would sometimes fail to respond to heartbeats. So with the load that we have - millions of mostly small files on CephFS - I wouldn't go below 2 real cores per OSD. But this may be a fringe workload. On Fri, Nov 13, 2020 at 3:36 AM Frank Schilder <frans@dtu.dk> wrote:

...

If each OSD requires 4T

Nobody said that. What was said is HDD=1T, SSD=3T. It depends on the drive type!

The %-utilisation information is just from top observed during heavy load. It does not show how the kernel schedules things on physical Ts. So, 2x50% utilisation could run on the same HT. I don't know how the OSDs are organised into threads, I'm just stating observations from real life (mimic cluster). So, for an SSD OSD I have seen a maximum of 4 threads in R state, two with 100% and two with 50% CPU, a load that fits on 3HT.

So, real life says 1HT per HDD and 3HT per SSD plus a bit for kernel and networking and you are set - based on worst-case performance monitoring I have seen in 2 years. Note that this is worst-case load. The average load is much lower.

A 16 core machine is totally overpowered. Assuming 1C=2HT, I count (2*3+8*1)/2=7 or (1*3+10*1)/2=6.5. So an 8 core CPU should do in either case. A 10 core CPU might be better, but 16C is a waste of money.

I should mention that these estimates apply to Intel CPUs (x86_64 architectures). Other architectures might not provide the same cycle efficiency.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 08:32:55 To: Frank Schilder; Nathan Fish Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

You all mentioned first 2T and another 2T. Could you give more details how OSD works with multi-thread, or share the link if it's already documented somewhere?

Is it always 4T, or start with 1T and grow up to 4T? Is it max 4T? Does each T run different job or just multiple instances of the same job? Does disk type affect how T works, like 1T is good enough for HDD while 4T is required for SSD?

If I change my plan to 2 SSD OSDs and 8 HDD OSDs (with 1 SSD for WAL and DB). If each OSD requires 4T, then 16C/32T 3.0GHz could be a better choice, because it provides sufficient Ts? If SSD OSD requires 4T and HDD OSD only requires 1T, then 8C/16T 3.2GHz would be better, because it provides sufficient Ts as well as stronger computing?

Thanks! Tony

...

-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Thursday, November 12, 2020 10:59 PM To: Tony Liu <tonyliu0592@hotmail.com>; Nathan Fish <lordcirth@gmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

I think this depends on the type of backing disk. We use the following CPUs:

Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz

My experience is, that a HDD OSD hardly gets to 100% of 1 hyper thread load even under heavy recovery/rebalance operations on 8+2 and 6+2 EC pools with compression set to aggressive. The CPU is mostly doing wait- IO, that is, the disk is the real bottle neck, not the processor power. With SSDs I have seen 2HT at 100% and 2 more at 50% each. I guess NVMe might be more demanding.

A server with 12 HDD and 1 SSD should be fine with a modern CPU with 8 cores. 16 threads sounds like an 8 core CPU. The 2nd generation Intel® Xeon® Silver 4209T with 8 cores should easily handle that (single socket system). We have the 16-core Intel silver in a dual socket system currently connected to 5HDD and 7SSD and I did a rebalance operation yesterday. The CPU user load did not exceed 2%, it can handle OSD processes easily. The server is dimensioned to run up to 12HDD and 14SSD OSDs (Dell R740xd2). As far as I can tell, the CPU configuration is overpowered for that.

Just for info, we use ganglia to record node utilisation. I use 1-year records and pick peak loads I observed for dimensioning the CPUs. These records include some very heavy recovery periods.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 04:57:53 To: Nathan Fish Cc: ceph-users@ceph.io Subject: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

...

-----Original Message----- From: Nathan Fish <lordcirth@gmail.com> Sent: Thursday, November 12, 2020 7:43 PM To: Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] which of cpu frequency and number of threads servers osd better?

From what I've seen, OSD daemons tend to bottleneck on the first 2 threads, while getting some use out of another 2. So 32 threads at 3.0 would be a lot better. Note that you may get better performance splitting off some of that SSD for block.db partitions or at least block.wal for the HDDs.

On Thu, Nov 12, 2020 at 9:57 PM Tony Liu <tonyliu0592@hotmail.com> wrote:

...

Hi,

For example, 16 threads with 3.2GHz and 32 threads with 3.0GHz, which makes 11 OSDs (10x12TB HDD and 1x960GB SSD) with better

Thanks Nathan! Tony performance?

...

...

Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I found this here: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/co... . Nothing in the ceph docs. It would be interesting to know what a shard is and what it does. Can anyone shed a bit of light on this? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 14 November 2020 10:48:57 To: Nathan Fish Cc: Tony Liu; ceph-users@ceph.io Subject: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? Yeah, I forgot to mention this. Our HDD OSDs are simplest set-up, WAB, DB, BLOCK all collocated on the HDD. My plan for the future is to use dm-cache for LVM OSDs instead of WAL/DB device. Then I might also see some more CPU utilisation with small-file I/O. From the question and the suggested per-server disk configuration I assumed that the target config would be everything collocated on HDD. By the way, is there any documentation on "osd op num threads per shard" somewhere? Can you post a link? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nathan Fish <lordcirth@gmail.com> Sent: 13 November 2020 21:07:20 To: Frank Schilder Cc: Tony Liu; ceph-users@ceph.io Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? We have 12TB HDD OSDs with 32GiB of (Optane) NVMe for block.db, used for cephfs_data pools, and NVMe-only OSDs used for cephfs_data pools. The NVMe DB about doubled our random IO performance - a great investment - doubling max CPU load as a result. We had to turn up "osd op num threads per shard hdd" from 1 to 2. (2 is the default for SSDs). This didn't noticeably improve performance, but without it, OSDs under max load would sometimes fail to respond to heartbeats. So with the load that we have - millions of mostly small files on CephFS - I wouldn't go below 2 real cores per OSD. But this may be a fringe workload. On Fri, Nov 13, 2020 at 3:36 AM Frank Schilder <frans@dtu.dk> wrote:

...

If each OSD requires 4T

Nobody said that. What was said is HDD=1T, SSD=3T. It depends on the drive type!

The %-utilisation information is just from top observed during heavy load. It does not show how the kernel schedules things on physical Ts. So, 2x50% utilisation could run on the same HT. I don't know how the OSDs are organised into threads, I'm just stating observations from real life (mimic cluster). So, for an SSD OSD I have seen a maximum of 4 threads in R state, two with 100% and two with 50% CPU, a load that fits on 3HT.

So, real life says 1HT per HDD and 3HT per SSD plus a bit for kernel and networking and you are set - based on worst-case performance monitoring I have seen in 2 years. Note that this is worst-case load. The average load is much lower.

A 16 core machine is totally overpowered. Assuming 1C=2HT, I count (2*3+8*1)/2=7 or (1*3+10*1)/2=6.5. So an 8 core CPU should do in either case. A 10 core CPU might be better, but 16C is a waste of money.

I should mention that these estimates apply to Intel CPUs (x86_64 architectures). Other architectures might not provide the same cycle efficiency.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 08:32:55 To: Frank Schilder; Nathan Fish Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

You all mentioned first 2T and another 2T. Could you give more details how OSD works with multi-thread, or share the link if it's already documented somewhere?

Is it always 4T, or start with 1T and grow up to 4T? Is it max 4T? Does each T run different job or just multiple instances of the same job? Does disk type affect how T works, like 1T is good enough for HDD while 4T is required for SSD?

If I change my plan to 2 SSD OSDs and 8 HDD OSDs (with 1 SSD for WAL and DB). If each OSD requires 4T, then 16C/32T 3.0GHz could be a better choice, because it provides sufficient Ts? If SSD OSD requires 4T and HDD OSD only requires 1T, then 8C/16T 3.2GHz would be better, because it provides sufficient Ts as well as stronger computing?

Thanks! Tony

...

-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Thursday, November 12, 2020 10:59 PM To: Tony Liu <tonyliu0592@hotmail.com>; Nathan Fish <lordcirth@gmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

I think this depends on the type of backing disk. We use the following CPUs:

Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz

My experience is, that a HDD OSD hardly gets to 100% of 1 hyper thread load even under heavy recovery/rebalance operations on 8+2 and 6+2 EC pools with compression set to aggressive. The CPU is mostly doing wait- IO, that is, the disk is the real bottle neck, not the processor power. With SSDs I have seen 2HT at 100% and 2 more at 50% each. I guess NVMe might be more demanding.

A server with 12 HDD and 1 SSD should be fine with a modern CPU with 8 cores. 16 threads sounds like an 8 core CPU. The 2nd generation Intel® Xeon® Silver 4209T with 8 cores should easily handle that (single socket system). We have the 16-core Intel silver in a dual socket system currently connected to 5HDD and 7SSD and I did a rebalance operation yesterday. The CPU user load did not exceed 2%, it can handle OSD processes easily. The server is dimensioned to run up to 12HDD and 14SSD OSDs (Dell R740xd2). As far as I can tell, the CPU configuration is overpowered for that.

Just for info, we use ganglia to record node utilisation. I use 1-year records and pick peak loads I observed for dimensioning the CPUs. These records include some very heavy recovery periods.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 13 November 2020 04:57:53 To: Nathan Fish Cc: ceph-users@ceph.io Subject: [ceph-users] Re: which of cpu frequency and number of threads servers osd better?

...

-----Original Message----- From: Nathan Fish <lordcirth@gmail.com> Sent: Thursday, November 12, 2020 7:43 PM To: Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] which of cpu frequency and number of threads servers osd better?

From what I've seen, OSD daemons tend to bottleneck on the first 2 threads, while getting some use out of another 2. So 32 threads at 3.0 would be a lot better. Note that you may get better performance splitting off some of that SSD for block.db partitions or at least block.wal for the HDDs.

On Thu, Nov 12, 2020 at 9:57 PM Tony Liu <tonyliu0592@hotmail.com> wrote:

...

Hi,

For example, 16 threads with 3.2GHz and 32 threads with 3.0GHz, which makes 11 OSDs (10x12TB HDD and 1x960GB SSD) with better

Thanks Nathan! Tony performance?

...

...

Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io