Hi all, My Ceph setup: - 12 OSD nodes, 4 OSD nodes per rack. Replication of 3, 1 replica per rack. - 20 spinning SAS disks per node. - Some nodes have 256GB RAM, some nodes 128GB. - CPU varies between Intel E5-2650 and Intel Gold 5317. - Each node has 10Gbit/s network. Using rados bench I am getting decent results (depending on block size): - 1000 MB/s throughput, 1000 IOps with 1MB block size - 30 MB/s throughput, 7500 IOps with 4K block size Unfortunately not getting the same performance with Rados Gateway (S3). - 1x HAProxy with 3 backend RGW's. I am using Minio Warp for benchmarking (PUT). I am 1 Warp server and 5 Warp clients. Benchmarking towards the HAProxy. Results: - Using 10MB object size, I am hitting the 10Gbit/s link of the HAProxy server. Thats good. - Using 500K object size, I am getting a throughput of 70 up to 150 MB/s with 140 up to 300 obj/s. It depends on the concurrency setting of Warp. It look likes the objects/s is the bottleneck, not the throughput. Max memory usage is about 80-90GB per node. CPU's are quite idling. Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s. Ceph version is 15.2. Thanks!
Hi all,
My Ceph setup: - 12 OSD nodes, 4 OSD nodes per rack. Replication of 3, 1 replica per rack. - 20 spinning SAS disks per node.
Don't use legacy HDDs if you care about performance.
- Some nodes have 256GB RAM, some nodes 128GB.
128GB is on the low side for 20 OSDs.
- CPU varies between Intel E5-2650 and Intel Gold 5317.
E5-2650 is underpowered for 20 OSDs. 5317 isn't the ideal fit, it'd make a decent MDS system but assuming a dual socket system, you have ~2 threads per OSD, which is maybe acceptable for HDDs, but I assume you have mon/mgr/rgw on some of them too.
- Each node has 10Gbit/s network.
Using rados bench I am getting decent results (depending on block size): - 1000 MB/s throughput, 1000 IOps with 1MB block size - 30 MB/s throughput, 7500 IOps with 4K block size
rados bench is a useful for smoke testing but not always a reflection of E2E experience.
Unfortunately not getting the same performance with Rados Gateway (S3).
- 1x HAProxy with 3 backend RGW's.
Run an RGW on every node.
I am using Minio Warp for benchmarking (PUT). I am 1 Warp server and 5 Warp clients. Benchmarking towards the HAProxy.
Results: - Using 10MB object size, I am hitting the 10Gbit/s link of the HAProxy server. Thats good. - Using 500K object size, I am getting a throughput of 70 up to 150 MB/s with 140 up to 300 obj/s.
Tiny objects are the devil of any object storage deployment. The HDDs are killing you here, especially for the index pool. You might get a bit better by upping pg_num from the party line. You might also disable Nagle on the RGW nodes.
It depends on the concurrency setting of Warp.
It look likes the objects/s is the bottleneck, not the throughput.
Max memory usage is about 80-90GB per node. CPU's are quite idling.
Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s.
HDDs are a false economy.
Ceph version is 15.2.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024-06-10 15:20, Anthony D'Atri wrote:
Hi all,
My Ceph setup: - 12 OSD nodes, 4 OSD nodes per rack. Replication of 3, 1 replica per rack. - 20 spinning SAS disks per node.
Don't use legacy HDDs if you care about performance.
You are right here, but we use Ceph mainly for RBD. It performs 'good enough' for our RBD load.
- Some nodes have 256GB RAM, some nodes 128GB.
128GB is on the low side for 20 OSDs.
Agreed, but with 20 OSD's x osd_memory_target 4GB (80GB) it is enough. We haven't had any server that OOM'ed yet.
- CPU varies between Intel E5-2650 and Intel Gold 5317.
E5-2650 is underpowered for 20 OSDs. 5317 isn't the ideal fit, it'd make a decent MDS system but assuming a dual socket system, you have ~2 threads per OSD, which is maybe acceptable for HDDs, but I assume you have mon/mgr/rgw on some of them too.
The (CPU) load on the OSD nodes is quite low. Our MON/MGR/RGW aren't hosted on the OSD nodes and are running on modern hardware.
- Each node has 10Gbit/s network.
Using rados bench I am getting decent results (depending on block size): - 1000 MB/s throughput, 1000 IOps with 1MB block size - 30 MB/s throughput, 7500 IOps with 4K block size
rados bench is a useful for smoke testing but not always a reflection of E2E experience.
Unfortunately not getting the same performance with Rados Gateway (S3).
- 1x HAProxy with 3 backend RGW's.
Run an RGW on every node.
On every OSD node?
I am using Minio Warp for benchmarking (PUT). I am 1 Warp server and 5 Warp clients. Benchmarking towards the HAProxy.
Results: - Using 10MB object size, I am hitting the 10Gbit/s link of the HAProxy server. Thats good. - Using 500K object size, I am getting a throughput of 70 up to 150 MB/s with 140 up to 300 obj/s.
Tiny objects are the devil of any object storage deployment. The HDDs are killing you here, especially for the index pool. You might get a bit better by upping pg_num from the party line.
I would expect high write await times, but all OSD/disks have write await times of 1 ms up to 3 ms.
You might also disable Nagle on the RGW nodes.
I need to lookup what that exactly is and does.
It depends on the concurrency setting of Warp.
It look likes the objects/s is the bottleneck, not the throughput.
Max memory usage is about 80-90GB per node. CPU's are quite idling.
Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s.
HDDs are a false economy.
Got it :)
Ceph version is 15.2.
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
- 20 spinning SAS disks per node. Don't use legacy HDDs if you care about performance.
You are right here, but we use Ceph mainly for RBD. It performs 'good enough' for our RBD load.
You use RBD for archival?
- Some nodes have 256GB RAM, some nodes 128GB. 128GB is on the low side for 20 OSDs.
Agreed, but with 20 OSD's x osd_memory_target 4GB (80GB) it is enough. We haven't had any server that OOM'ed yet.
Remember that's a *target* not a *limit*. Say one or more of your failure domains goes offline or you have some other large topology change. Your OSDs might want up to 2x osd_memory_target, then you OOM and it cascades. I've been there, had to do an emergency upgrade of 24xOSD nodes from 128GB to 192GB.
- CPU varies between Intel E5-2650 and Intel Gold 5317. E5-2650 is underpowered for 20 OSDs. 5317 isn't the ideal fit, it'd make a decent MDS system but assuming a dual socket system, you have ~2 threads per OSD, which is maybe acceptable for HDDs, but I assume you have mon/mgr/rgw on some of them too.
The (CPU) load on the OSD nodes is quite low. Our MON/MGR/RGW aren't hosted on the OSD nodes and are running on modern hardware.
You didn't list additional nodes so I assumed. You might still do well to have a larger number of RGWs, wherever they run. RGWs often scale better horizontally than vertically.
rados bench is a useful for smoke testing but not always a reflection of E2E experience.
Unfortunately not getting the same performance with Rados Gateway (S3). - 1x HAProxy with 3 backend RGW's. Run an RGW on every node.
On every OSD node?
Yep, why not?
I am using Minio Warp for benchmarking (PUT). I am 1 Warp server and 5 Warp clients. Benchmarking towards the HAProxy. Results: - Using 10MB object size, I am hitting the 10Gbit/s link of the HAProxy server. Thats good. - Using 500K object size, I am getting a throughput of 70 up to 150 MB/s with 140 up to 300 obj/s. Tiny objects are the devil of any object storage deployment. The HDDs are killing you here, especially for the index pool. You might get a bit better by upping pg_num from the party line.
I would expect high write await times, but all OSD/disks have write await times of 1 ms up to 3 ms.
There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs?
You might also disable Nagle on the RGW nodes.
I need to lookup what that exactly is and does.
It depends on the concurrency setting of Warp. It look likes the objects/s is the bottleneck, not the throughput. Max memory usage is about 80-90GB per node. CPU's are quite idling. Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s. HDDs are a false economy.
Got it :)
Ceph version is 15.2. Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024-06-10 17:42, Anthony D'Atri wrote:
- 20 spinning SAS disks per node. Don't use legacy HDDs if you care about performance.
You are right here, but we use Ceph mainly for RBD. It performs 'good enough' for our RBD load.
You use RBD for archival?
No, storage for (light-weight) virtual machines.
- Some nodes have 256GB RAM, some nodes 128GB. 128GB is on the low side for 20 OSDs.
Agreed, but with 20 OSD's x osd_memory_target 4GB (80GB) it is enough. We haven't had any server that OOM'ed yet.
Remember that's a *target* not a *limit*. Say one or more of your failure domains goes offline or you have some other large topology change. Your OSDs might want up to 2x osd_memory_target, then you OOM and it cascades. I've been there, had to do an emergency upgrade of 24xOSD nodes from 128GB to 192GB.
+1
- CPU varies between Intel E5-2650 and Intel Gold 5317. E5-2650 is underpowered for 20 OSDs. 5317 isn't the ideal fit, it'd make a decent MDS system but assuming a dual socket system, you have ~2 threads per OSD, which is maybe acceptable for HDDs, but I assume you have mon/mgr/rgw on some of them too.
The (CPU) load on the OSD nodes is quite low. Our MON/MGR/RGW aren't hosted on the OSD nodes and are running on modern hardware.
You didn't list additional nodes so I assumed. You might still do well to have a larger number of RGWs, wherever they run. RGWs often scale better horizontally than vertically.
Good to know. I'll check if adding more RGW nodes is possible.
rados bench is a useful for smoke testing but not always a reflection of E2E experience.
Unfortunately not getting the same performance with Rados Gateway (S3). - 1x HAProxy with 3 backend RGW's. Run an RGW on every node.
On every OSD node?
Yep, why not?
I am using Minio Warp for benchmarking (PUT). I am 1 Warp server and 5 Warp clients. Benchmarking towards the HAProxy. Results: - Using 10MB object size, I am hitting the 10Gbit/s link of the HAProxy server. Thats good. - Using 500K object size, I am getting a throughput of 70 up to 150 MB/s with 140 up to 300 obj/s. Tiny objects are the devil of any object storage deployment. The HDDs are killing you here, especially for the index pool. You might get a bit better by upping pg_num from the party line.
I would expect high write await times, but all OSD/disks have write await times of 1 ms up to 3 ms.
There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs?
Index as the data pool has 256 PG's.
You might also disable Nagle on the RGW nodes.
I need to lookup what that exactly is and does.
It depends on the concurrency setting of Warp. It look likes the objects/s is the bottleneck, not the throughput. Max memory usage is about 80-90GB per node. CPU's are quite idling. Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s. HDDs are a false economy.
Got it :)
Ceph version is 15.2. Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You are right here, but we use Ceph mainly for RBD. It performs 'good enough' for our RBD load. You use RBD for archival?
No, storage for (light-weight) virtual machines.
I'm surprised that it's enough, I've seen HDDs fail miserably in that role.
The (CPU) load on the OSD nodes is quite low. Our MON/MGR/RGW aren't hosted on the OSD nodes and are running on modern hardware.
You didn't list additional nodes so I assumed. You might still do well to have a larger number of RGWs, wherever they run. RGWs often scale better horizontally than vertically.
Good to know. I'll check if adding more RGW nodes is possible.
To be clear, you don't need more nodes. You can add RGWs to the ones you already have. You have 12 OSD nodes - why not put an RGW on each?
There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs?
Index as the data pool has 256 PG's.
To be clear, that means whatever.rgw.buckets.index ?
You might also disable Nagle on the RGW nodes. I need to lookup what that exactly is and does.
It depends on the concurrency setting of Warp. It look likes the objects/s is the bottleneck, not the throughput. Max memory usage is about 80-90GB per node. CPU's are quite idling. Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s. HDDs are a false economy. Got it :) Ceph version is 15.2. Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024-06-10 21:37, Anthony D'Atri wrote:
You are right here, but we use Ceph mainly for RBD. It performs 'good enough' for our RBD load. You use RBD for archival?
No, storage for (light-weight) virtual machines.
I'm surprised that it's enough, I've seen HDDs fail miserably in that role.
The (CPU) load on the OSD nodes is quite low. Our MON/MGR/RGW aren't hosted on the OSD nodes and are running on modern hardware.
You didn't list additional nodes so I assumed. You might still do well to have a larger number of RGWs, wherever they run. RGWs often scale better horizontally than vertically.
Good to know. I'll check if adding more RGW nodes is possible.
To be clear, you don't need more nodes. You can add RGWs to the ones you already have. You have 12 OSD nodes - why not put an RGW on each?
Might be an option, just don't like the idea to host multiple components on nodes. But I'll consider it.
There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs?
Index as the data pool has 256 PG's.
To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256.
You might also disable Nagle on the RGW nodes. I need to lookup what that exactly is and does.
It depends on the concurrency setting of Warp. It look likes the objects/s is the bottleneck, not the throughput. Max memory usage is about 80-90GB per node. CPU's are quite idling. Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s. HDDs are a false economy. Got it :) Ceph version is 15.2. Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
To be clear, you don't need more nodes. You can add RGWs to the ones you already have. You have 12 OSD nodes - why not put an RGW on each?
Might be an option, just don't like the idea to host multiple components on nodes. But I'll consider it.
I really don't like mixing mon/mgr with other components because of coupled failure domains, and past experience with mon misbehavior, but many people do that. ymmv. With a bunch of RGWs none of them need grow to consume significant resources, and it can be difficult to get an RGW daemon to itself really use all of a dedicated node.
There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs? Index as the data pool has 256 PG's. To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256.
Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler
On 2024-06-11 01:01, Anthony D'Atri wrote:
To be clear, you don't need more nodes. You can add RGWs to the ones you already have. You have 12 OSD nodes - why not put an RGW on each?
Might be an option, just don't like the idea to host multiple components on nodes. But I'll consider it.
I really don't like mixing mon/mgr with other components because of coupled failure domains, and past experience with mon misbehavior, but many people do that. ymmv. With a bunch of RGWs none of them need grow to consume significant resources, and it can be difficult to get an RGW daemon to itself really use all of a dedicated node.
I am not sure adding more RGW's will increase the performance. Just tested with 1 and with 2 RGW's: Client 1 -> RGW Node A = 150-250 objects/s Client 1 -> RGW Node A = 60-120 objects/s and simultaneously Client 2 -> RGW Node B = 60-120 objects/s. Together makes 150-250 objects/s. So, it does not matter performance wise if I am using 1 or 2 RGW nodes. Client 1 -> HAProxy -> 3 RGW's = 150-250 objects/s.
There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs? Index as the data pool has 256 PG's. To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256.
Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally.
No, my RBD pool is larger. My average PG per OSD is round 60-70.
Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well.
If you have RBD or other pools colocated, those numbers would change.
^ above assume disabling the autoscaler
I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s. The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. Other suggestions, or does anyone else has suggestions?
I am not sure adding more RGW's will increase the performance.
That was a tangent.
To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally.
No, my RBD pool is larger. My average PG per OSD is round 60-70.
Ah. Aim for 100-200 with spinners.
Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler
I bumped my .data pool from 256 to 1024 and .index from 32 to 128.
Your index pool still only benefits from half of your OSDs with a value of 128.
Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s.
The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3.
Trust iostat about as far as you can throw it.
Other suggestions, or does anyone else has suggestions?
I have doing some further testing. My RGW pool is placed on spinning disks. I created a 2nd RGW data pool, placed on flash disks. Benchmarking on HDD pool: Client 1 -> 1 RGW Node: 150 obj/s Client 1-5 -> 1 RGW Node: 150 ob/s (30 obj/s each client) Client 1 -> HAProxy -> 3 RGW Nodes: 150 obj/s Client 1-5 -> HAProxy -> 3 RGW Nodes: 150 obj/s (30 obj/s each client) I did the same tests towards the RGW pool on flash disks: same results So, it doesn't matter if my pool is hosted on HDD or SSD. It doesn't matter if I am using 1 RGW or 3 RGW nodes. It doesn't matter if I am using 1 client or 5 clients. I am constantly limited at around 140-160 objects/s. I see some TCP Retransmissions on the RGW Node, but maybe thats 'normal'. Any ideas/suggestions? On 2024-06-11 22:08, Anthony D'Atri wrote:
I am not sure adding more RGW's will increase the performance.
That was a tangent.
To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally.
No, my RBD pool is larger. My average PG per OSD is round 60-70.
Ah. Aim for 100-200 with spinners.
Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler
I bumped my .data pool from 256 to 1024 and .index from 32 to 128.
Your index pool still only benefits from half of your OSDs with a value of 128.
Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s.
The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3.
Trust iostat about as far as you can throw it.
Other suggestions, or does anyone else has suggestions?
How large are the objects you tested with?
On Jun 13, 2024, at 14:46, sinan@turka.nl wrote:
I have doing some further testing.
My RGW pool is placed on spinning disks. I created a 2nd RGW data pool, placed on flash disks.
Benchmarking on HDD pool: Client 1 -> 1 RGW Node: 150 obj/s Client 1-5 -> 1 RGW Node: 150 ob/s (30 obj/s each client) Client 1 -> HAProxy -> 3 RGW Nodes: 150 obj/s Client 1-5 -> HAProxy -> 3 RGW Nodes: 150 obj/s (30 obj/s each client)
I did the same tests towards the RGW pool on flash disks: same results
So, it doesn't matter if my pool is hosted on HDD or SSD. It doesn't matter if I am using 1 RGW or 3 RGW nodes. It doesn't matter if I am using 1 client or 5 clients.
I am constantly limited at around 140-160 objects/s.
I see some TCP Retransmissions on the RGW Node, but maybe thats 'normal'.
Any ideas/suggestions?
On 2024-06-11 22:08, Anthony D'Atri wrote:
I am not sure adding more RGW's will increase the performance. That was a tangent. To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. No, my RBD pool is larger. My average PG per OSD is round 60-70. Ah. Aim for 100-200 with spinners. Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Your index pool still only benefits from half of your OSDs with a value of 128. Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s. The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. Trust iostat about as far as you can throw it. Other suggestions, or does anyone else has suggestions?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
500K object size
Op 13 jun 2024 om 21:11 heeft Anthony D'Atri <aad@dreamsnake.net> het volgende geschreven:
How large are the objects you tested with?
On Jun 13, 2024, at 14:46, sinan@turka.nl wrote:
I have doing some further testing.
My RGW pool is placed on spinning disks. I created a 2nd RGW data pool, placed on flash disks.
Benchmarking on HDD pool: Client 1 -> 1 RGW Node: 150 obj/s Client 1-5 -> 1 RGW Node: 150 ob/s (30 obj/s each client) Client 1 -> HAProxy -> 3 RGW Nodes: 150 obj/s Client 1-5 -> HAProxy -> 3 RGW Nodes: 150 obj/s (30 obj/s each client)
I did the same tests towards the RGW pool on flash disks: same results
So, it doesn't matter if my pool is hosted on HDD or SSD. It doesn't matter if I am using 1 RGW or 3 RGW nodes. It doesn't matter if I am using 1 client or 5 clients.
I am constantly limited at around 140-160 objects/s.
I see some TCP Retransmissions on the RGW Node, but maybe thats 'normal'.
Any ideas/suggestions?
On 2024-06-11 22:08, Anthony D'Atri wrote:
I am not sure adding more RGW's will increase the performance. That was a tangent. To be clear, that means whatever.rgw.buckets.index ?
No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. No, my RBD pool is larger. My average PG per OSD is round 60-70. Ah. Aim for 100-200 with spinners. Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Your index pool still only benefits from half of your OSDs with a value of 128. Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s. The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. Trust iostat about as far as you can throw it. Other suggestions, or does anyone else has suggestions?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There you go. Tiny objects are the hardest thing for any object storage service: you can have space amplification and metadata operations become a very high portion of the overall workload. With 500KB objects, you may waste a significant fraction of underlying space -- especially if you have large-IU QLC OSDs, or OSDs made with an older Ceph release where the min_alloc_size was 64KB vs the current 4KB. This is exacerbated by EC if you're using it, as many do for buckets pools. https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToPQjN8bDI/edit?gid=358760253#gid=358760253 Bluestore Space Amplification Cheat Sheet docs.google.com Things to do: Disable Nagle https://docs.ceph.com/en/quincy/radosgw/frontends/ Putting your index pool on as many SSDs as you can would also help, I don't recall if it's on HDD now. Index doesn't use all that much data, but benefits from a generous pg_num and multiple OSDs so that it isn't bottlenecked.
On Jun 13, 2024, at 15:13, Sinan Polat <sinan@turka.nl> wrote:
500K object size
Op 13 jun 2024 om 21:11 heeft Anthony D'Atri <aad@dreamsnake.net> het volgende geschreven:
How large are the objects you tested with?
On Jun 13, 2024, at 14:46, sinan@turka.nl wrote:
I have doing some further testing.
My RGW pool is placed on spinning disks. I created a 2nd RGW data pool, placed on flash disks.
Benchmarking on HDD pool: Client 1 -> 1 RGW Node: 150 obj/s Client 1-5 -> 1 RGW Node: 150 ob/s (30 obj/s each client) Client 1 -> HAProxy -> 3 RGW Nodes: 150 obj/s Client 1-5 -> HAProxy -> 3 RGW Nodes: 150 obj/s (30 obj/s each client)
I did the same tests towards the RGW pool on flash disks: same results
So, it doesn't matter if my pool is hosted on HDD or SSD. It doesn't matter if I am using 1 RGW or 3 RGW nodes. It doesn't matter if I am using 1 client or 5 clients.
I am constantly limited at around 140-160 objects/s.
I see some TCP Retransmissions on the RGW Node, but maybe thats 'normal'.
Any ideas/suggestions?
On 2024-06-11 22:08, Anthony D'Atri wrote:
I am not sure adding more RGW's will increase the performance. That was a tangent. To be clear, that means whatever.rgw.buckets.index ?
> No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. No, my RBD pool is larger. My average PG per OSD is round 60-70. Ah. Aim for 100-200 with spinners. Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Your index pool still only benefits from half of your OSDs with a value of 128. Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s. The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. Trust iostat about as far as you can throw it. Other suggestions, or does anyone else has suggestions?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Disabling Nagle didn't have any effect. I created a new RGW pool (data, index), both on flash disks. No effect. I set the size=2, no effect. Btw, cluster is running on Octopus (15.2). When using 3 MB/s objects, I am still getting 150 objects/s. Just a higher throughput (150x3MB = 450MB/s). But the objects/s doesn't increase. Its like, a Ceph configuration is limiting it or something. On 2024-06-13 21:37, Anthony D'Atri wrote:
There you go.
Tiny objects are the hardest thing for any object storage service: you can have space amplification and metadata operations become a very high portion of the overall workload.
With 500KB objects, you may waste a significant fraction of underlying space -- especially if you have large-IU QLC OSDs, or OSDs made with an older Ceph release where the min_alloc_size was 64KB vs the current 4KB. This is exacerbated by EC if you're using it, as many do for buckets pools.
Bluestore Space Amplification Cheat Sheet [1] docs.google.com [1]
Things to do: Disable Nagle https://docs.ceph.com/en/quincy/radosgw/frontends/
Putting your index pool on as many SSDs as you can would also help, I don't recall if it's on HDD now. Index doesn't use all that much data, but benefits from a generous pg_num and multiple OSDs so that it isn't bottlenecked.
On Jun 13, 2024, at 15:13, Sinan Polat <sinan@turka.nl> wrote:
500K object size
Op 13 jun 2024 om 21:11 heeft Anthony D'Atri <aad@dreamsnake.net> het volgende geschreven:
How large are the objects you tested with?
On Jun 13, 2024, at 14:46, sinan@turka.nl wrote:
I have doing some further testing.
My RGW pool is placed on spinning disks. I created a 2nd RGW data pool, placed on flash disks.
Benchmarking on HDD pool: Client 1 -> 1 RGW Node: 150 obj/s Client 1-5 -> 1 RGW Node: 150 ob/s (30 obj/s each client) Client 1 -> HAProxy -> 3 RGW Nodes: 150 obj/s Client 1-5 -> HAProxy -> 3 RGW Nodes: 150 obj/s (30 obj/s each client)
I did the same tests towards the RGW pool on flash disks: same results
So, it doesn't matter if my pool is hosted on HDD or SSD. It doesn't matter if I am using 1 RGW or 3 RGW nodes. It doesn't matter if I am using 1 client or 5 clients.
I am constantly limited at around 140-160 objects/s.
I see some TCP Retransmissions on the RGW Node, but maybe thats 'normal'.
Any ideas/suggestions?
On 2024-06-11 22:08, Anthony D'Atri wrote: I am not sure adding more RGW's will increase the performance. That was a tangent. To be clear, that means whatever.rgw.buckets.index ? No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. No, my RBD pool is larger. My average PG per OSD is round 60-70. Ah. Aim for 100-200 with spinners.
Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Your index pool still only benefits from half of your OSDs with a value of 128.
Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s. The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. Trust iostat about as far as you can throw it.
Other suggestions, or does anyone else has suggestions?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Links: ------ [1] https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
participants (4)
-
Anthony D'Atri
-
Anthony D'Atri
-
Sinan Polat
-
sinan＠turka.nl