RGW returning HTTP 500 during resharding
Hi, I am doing an attempt at loading a large number (billions) of tiny objects from other existing system into Ceph (18.2.4 reef) RadosGW/S3. I wrote a little program for that using the AWS S3 Java SDK (using CRT/S3AsyncClient), and am using 250 simultanous connections to get the data in. However I am experiencing problems when it starts to dynamically reshard. The first problem I was experiencing is that the SDK then complains connection throughput is too low, and errors out: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Http connectio n channel shut down due to failure to meet throughput minimum == While I configured the SDK to retry several times on errors, that does not seem to be happening. Or perhaps it does retry, but using one of the other connections, which obviously will fail as well for the same reason. Anyway, I shoved this issue under the carpet by configuring a very high timeout (creating a S3CrtConnectionHealthConfiguration and setting minimumThroughputTimeout() ) That seemed to work around it for a while. However I found out not much later that if the resharding takes longer than some minutes, RadosGW starts returning HTTP status code 500 to PUT requests: == Sep 27 18:14:33 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:15:03 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:21:59 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-16f7e. ret=(2 300) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-15b3. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-147ea. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7865710 op status=-2300 http_status=500 latency=450.223876953s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7ff4710 op status=-2300 http_status=500 latency=450.231872559s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfca7c3710 op status=-2300 http_status=500 latency=450.239868164s ====== Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7865710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.851 +0000] "PUT /datastore/some-obj-16f7e HTTP/1.1" 500 905 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.223876953s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7ff4710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.843 +0000] "PUT /datastore/some-obj-15b3 HTTP/1.1" 500 293 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.231872559s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfca7c3710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.835 +0000] "PUT /datastore/some-obj-147ea HTTP/1.1" 500 318 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.239868164s S == AWS SDK considers 500 quite fatal: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates internal server error == No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives) And is it correct behavior that it returns HTTP response code 500, instead of something that could indicate it is a retry'able condition? If I would add my own code that does retry for a very long time, is there any way I can detect the 500 is due to the resharding, instead of some other condition that do is fatal? Also, is there any more efficient way to get a large amount of objects into Ceph than individual PUTs? Yours sincerely, Floris Bos
No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives)
Which drive SKU(s)? How full are they? Is their firmware up to date? How many RGWs? Have you tuned your server network stack? Disabled Nagle? How many bucket OSDs? How many index OSDs? How many PGs in the bucket and index pools? How many buckets? Do you have like 200M objects per? Do you have the default max objects/shard setting? Tiny objects are the devil of many object systems. I can think of cases where the above questions could affect this case. I think you resharding in advance might help.
And is it correct behavior that it returns HTTP response code 500, instead of something that could indicate it is a retry'able condition?
If I would add my own code that does retry for a very long time, is there any way I can detect the 500 is due to the resharding, instead of some other condition that do is fatal? Also, is there any more efficient way to get a large amount of objects into Ceph than individual PUTs? Yours sincerely,
Floris Bos _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
"Anthony D'Atri" <aad@dreamsnake.net> schreef op 28 september 2024 16:24:
No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives)
Which drive SKU(s)? How full are they? Is their firmware up to date? How many RGWs? Have you tuned your server network stack? Disabled Nagle? How many bucket OSDs? How many index OSDs? How many PGs in the bucket and index pools? How many buckets? Do you have like 200M objects per? Do you have the default max objects/shard setting?
Tiny objects are the devil of many object systems. I can think of cases where the above questions could affect this case. I think you resharding in advance might help.
- Drives advertise themselves as “Dell Ent NVMe v2 AGN MU U.2 6.4TB” (think that is Samsung under the Dell sticker), newest 2.5.0 firmware. They are pretty empty. Although there is some 10% capacity being used by other stuff (RBD images) - Single bucket. My import application already errored out after only 72 M objects/476 GiB of data, and need a lot more. Objects are between 0 bytes and 1 MB, 7 KB average. - Currently using only 1 RGW during my test run to simplify looking at logs, although I have 4. - I cannot touch TCP socket options settings in my Java application. When you build a S3AsyncClient with the Java AWS SDK using the .crtBuilder(), the SDK outsources the communication to the AWS aws-c-s3/aws-c-http/aws-io CRT libraries written in C, and I never get to see the raw socket in Java. Looking at the source I don’t think Amazon is disabling the nagle algorithm in their code. At least I don’t see TCP_NODELAY or similar options being used at the place they seem to set the socket options: https://github.com/awslabs/aws-c-io/blob/c345d77274db83c0c2e30331814093e7c84... - Did not tune any network settings, and it is pretty quiet on the network side, nowhere near saturating bandwidth because objects are so small. - Did not really tune anything else either yet. Pretty much a default cephadm setup for now. - See it (automagically) allocated 1024 PGs for .data and 32 for .index. - Think the main delay is just Ceph wanting to make sure everything is sync’ed to storage before reporting success. So that is why I am making a lot of concurrent connections to perform multiple PUT requests simultaneously. But even with 250 connections, it only does around 5000 objects per second according to the “object ingress/egress” Grafana graph. Can probably raise it some more… Had the default max. objects per shard settings for the dynamic sharding. But have now manually resharded to 10069 shards, and will have a go to see if it works better now. Yours sincerely, Floris Bos
On Sep 28, 2024, at 5:21 PM, Floris Bos <bos@je-eigen-domein.nl> wrote:
"Anthony D'Atri" <aad@dreamsnake.net> schreef op 28 september 2024 16:24:
No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives)
Which drive SKU(s)? How full are they? Is their firmware up to date? How many RGWs? Have you tuned your server network stack? Disabled Nagle? How many bucket OSDs? How many index OSDs? How many PGs in the bucket and index pools? How many buckets? Do you have like 200M objects per? Do you have the default max objects/shard setting?
Tiny objects are the devil of many object systems. I can think of cases where the above questions could affect this case. I think you resharding in advance might help.
- Drives advertise themselves as “Dell Ent NVMe v2 AGN MU U.2 6.4TB” (think that is Samsung under the Dell sticker)
AGN or AG is Dell for “Agnostic”, i.e. whatever the cheapest is they have on the shelf. Samsung indeed is one of the potentials. `smartctl -a` should show you what it actually is.
newest 2.5.0 firmware.
Verified by a DSU run?
They are pretty empty. Although there is some 10% capacity being used by other stuff (RBD images)
- Single bucket. My import application already errored out after only 72 M objects/476 GiB of data, and need a lot more. Objects are between 0 bytes and 1 MB, 7 KB average.
Only 72M? That’s a rather sizable bucket. Were there existing objects as well? Do you have the ability to spread across multiple buckets? That would decrease your need to reshard. As I interpret the docs, 199M is the default max number of objects above which auto-resharding won’t happen. Since you *know* that you will be pulling in extreme numbers of objects, consider pre-sharding the bucket while it’s empty. That will be dramatically faster in every way.
- Currently using only 1 RGW during my test run to simplify looking at logs, although I have 4.
That’s a lot of ingest for even 4. Would not be surprised if you’re saturating your connection limit or Linux networking.
- I cannot touch TCP socket options settings in my Java application.
Your RGW daemons are running on a Java application, not a Linux system????
When you build a S3AsyncClient with the Java AWS SDK using the .crtBuilder(), the SDK outsources the communication to the AWS aws-c-s3/aws-c-http/aws-io CRT libraries written in C, and I never get to see the raw socket in Java. Looking at the source I don’t think Amazon is disabling the nagle algorithm in their code.
On the server(s). You’re unhappy with the *server* performance, no? RGW can configure the front end options to disable Nagle; search the archives for an articles where doing so significantly improved small object latency
At least I don’t see TCP_NODELAY or similar options being used at the place they seem to set the socket options: https://github.com/awslabs/aws-c-io/blob/c345d77274db83c0c2e30331814093e7c84...
- Did not tune any network settings, and it is pretty quiet on the network side, nowhere near saturating bandwidth because objects are so small.
There’s more to life than bandwidth. somaxconn, nf_conntrack filling up, filling buffers, etc.
- Did not really tune anything else either yet. Pretty much a default cephadm setup for now.
- See it (automagically) allocated 1024 PGs for .data and 32 for .index.
Those numbers are nearly useless without context: the rest of the info I requested. There was a reason for everything on the list. Folks contribute to the list out of the goodness of their hearts, and aren’t paid for back-and-forth tooth-pulling. If your index pool and bucket pool share say 3x HDDs or 3x coarse-IU QLC, then don’t expect much. Sounds like you have the pg autoscaler enabled, which probably doesn’t help. Your index pool almost certainly needs more PGs. Probably log as well, or set the rgw log levels to 0.
- Think the main delay is just Ceph wanting to make sure everything is sync’ed to storage before reporting success. So that is why I am making a lot of concurrent connections to perform multiple PUT requests simultaneously. But even with 250 connections, it only does around 5000 objects per second according to the “object ingress/egress” Grafana graph. Can probably raise it some more…
With one RGW you aren’t going to get far, unless you have a 500 core CPU, and probably not even than.
Had the default max. objects per shard settings for the dynamic sharding. But have now manually resharded to 10069 shards, and will have a go to see if it works better now.
Yours sincerely,
Floris Bos _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
"Anthony D'Atri" <anthony.datri@gmail.com> schreef op 29 september 2024 02:01:
On Sep 28, 2024, at 5:21 PM, Floris Bos <bos@je-eigen-domein.nl> wrote:
"Anthony D'Atri" <aad@dreamsnake.net> schreef op 28 september 2024 16:24: No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives)
Which drive SKU(s)? How full are they? Is their firmware up to date? How many RGWs? Have you tuned your server network stack? Disabled Nagle? How many bucket OSDs? How many index OSDs? How many PGs in the bucket and index pools? How many buckets? Do you have like 200M objects per? Do you have the default max objects/shard setting?
Tiny objects are the devil of many object systems. I can think of cases where the above questions could affect this case. I think you resharding in advance might help.
- Drives advertise themselves as “Dell Ent NVMe v2 AGN MU U.2 6.4TB” (think that is Samsung under the Dell sticker)
AGN or AG is Dell for “Agnostic”, i.e. whatever the cheapest is they have on the shelf. Samsung indeed is one of the potentials. `smartctl -a` should show you what it actually is.
Smartctl only gave “Dell”. But see iDRAC is more telling and says 90% Samsung, 10% SkHynix (*sigh*)
newest 2.5.0 firmware.
Verified by a DSU run?
Was updated through iDRAC. Cannot install DSU on the OS itself.
They are pretty empty. Although there is some 10% capacity being used by other stuff (RBD images)
- Single bucket. My import application already errored out after only 72 M objects/476 GiB of data, and need a lot more. Objects are between 0 bytes and 1 MB, 7 KB average.
Only 72M? That’s a rather sizable bucket. Were there existing objects as well? Do you have the ability to spread across multiple buckets? That would decrease your need to reshard. As I interpret the docs, 199M is the default max number of objects above which auto-resharding won’t happen.
Since you *know* that you will be pulling in extreme numbers of objects, consider pre-sharding the bucket while it’s empty. That will be dramatically faster in every way.
No existing data. And yes, I set it manually to 10069 shards now. So now it should not happen again, since that is above the 1999 rgw_max_dynamic_shards It still feels a bit wrong to me to have to set this manually though. I am not against having to tune applications for performance gains, but think it is unfortunate that one seems to have to do so just to prevent the “500 internal server errors” that the resharding can effectively cause.
- I cannot touch TCP socket options settings in my Java application.
Your RGW daemons are running on a Java application, not a Linux system????
Sorry, thought you were asking if my (Java based) import-existing-stuff-to-S3 program disabled nagle, in its communication with the rgw. Ceph has default settings, which is nagle disabled (ms_tcp_nodelay true).
Those numbers are nearly useless without context: the rest of the info I requested. There was a reason for everything on the list. Folks contribute to the list out of the goodness of their hearts, and aren’t paid for back-and-forth tooth-pulling. If your index pool and bucket pool share say 3x HDDs or 3x coarse-IU QLC, then don’t expect much.
Sounds like you have the pg autoscaler enabled, which probably doesn’t help. Your index pool almost certainly needs more PGs. Probably log as well, or set the rgw log levels to 0.
Reason I am a bit vague about amount of OSDs and such, is that the numbers are not written on stone yet. Currently using 40 in total, but may or may not be able to “steal” more gear from another project, if the need arises. So I am rather fishing for advice in the form of “if you have >X OSDs, tune this and that” than specific to my exact current situation. ;-) Not doing anything special placement wise at the moment (not configured anything that would make the index and data go to different OSDs) Also note that my use-case is a bit non-standard in other ways. While I have a lot of objects, the application itself is internal within an organization and does not have many concurrent users. The performance demand is less than one would need if you have say a public facing web application with lots of visitors and peaks. So my main concern right now is having it run without internal server errors (the resharding thing), and fast enough so that my initial import completes within a couple weeks… I do have control over the import application, but cannot change the application that will be using the bucket afterwise, so am stuck with their design choice for tiny objects and a single bucket. Will play a bit with PGs and loglevel.
- Think the main delay is just Ceph wanting to make sure everything is sync’ed to storage before reporting success. So that is why I am making a lot of concurrent connections to perform multiple PUT requests simultaneously. But even with 250 connections, it only does around 5000 objects per second according to the “object ingress/egress” Grafana graph. Can probably raise it some more…
With one RGW you aren’t going to get far, unless you have a 500 core CPU, and probably not even than.
I wonder if that also applies to tiny objects. As I have the impression I am slowed down by RGW/Ceph waiting for individual objects to be synced to storage, instead of having to compute much for a tiny object. I am experimenting a bit with the settings at the moment. If I raise the amount of concurrent connections from 250 to 1024 with a single RGW it does raise the number of ingress objects per second from 5k to 9k. Which means that the RGW was not quite saturated before. And the server the RGW is on, is only using 14% CPU (1 min avg, taken over all CPU cores) If I create a DNS entry that includes all 4 RGW IP, and tell it to connect to that with 1024 connections, I see in netstat that it indeeds spreads the connections over the 4 RGW, but I get the same 9k objects/sec. No improvement at all with that, at least not when it is just my import program running with that number of connections... (I also noticed that the AWS SDK is not really written with small objects in mind. By default it only sets up a mere 25 connections. You have to lie to the SDK that you have a 400 Gbit connection by calling .targetThroughputInGbps(400), to persuade it to setup 1024 connections. Even though those connections only use up roughly 400 Mbit of client bandwidth in practice due to the tiny objects, and a similar amount for Ceph’s internal traffic. Nowhere near the 2x25 Gbit SFP28 the servers have). Yours sincerely, Floris Bos
AGN or AG is Dell for “Agnostic”, i.e. whatever the cheapest is they have on the shelf. Samsung indeed is one of the potentials. `smartctl -a` should show you what it actually is.
Smartctl only gave “Dell”.
That’s weird. Send me a full `smartctl -a` privately please. I’m very close to forking smartmontools publicly or at least drivedb.h
But see iDRAC is more telling and says 90% Samsung, 10% SkHynix (*sigh*)
iDRAC uses the same interfaces, so that’s weird. Last year I found Dell shipping Hynix drives that my contacts said had been cancelled. Wouldn’t give me a SMART reference, I ended up up getting upstream to accept a regex assumption.
newest 2.5.0 firmware.
Verified by a DSU run?
Was updated through iDRAC. Cannot install DSU on the OS itself.
Why not? I’ve done so thousands of times.
They are pretty empty. Although there is some 10% capacity being used by other stuff (RBD images)
- Single bucket. My import application already errored out after only 72 M objects/476 GiB of data, and need a lot more. Objects are between 0 bytes and 1 MB, 7 KB average.
Only 72M? That’s a rather sizable bucket. Were there existing objects as well? Do you have the ability to spread across multiple buckets? That would decrease your need to reshard. As I interpret the docs, 199M is the default max number of objects above which auto-resharding won’t happen.
Since you *know* that you will be pulling in extreme numbers of objects, consider pre-sharding the bucket while it’s empty. That will be dramatically faster in every way.
No existing data. And yes, I set it manually to 10069 shards now. So now it should not happen again, since that is above the 1999 rgw_max_dynamic_shards
Cool.
It still feels a bit wrong to me to have to set this manually though.
Might be a failsafe against fat fingers. Your bucket is an outlier in my world.
I am not against having to tune applications for performance gains, but think it is unfortunate that one seems to have to do so just to prevent the “500 internal server errors” that the resharding can effectively cause.
I’m just speculating with my limited info.
- I cannot touch TCP socket options settings in my Java application.
Your RGW daemons are running on a Java application, not a Linux system????
Sorry, thought you were asking if my (Java based) import-existing-stuff-to-S3 program disabled nagle, in its communication with the rgw. Ceph has default settings, which is nagle disabled (ms_tcp_nodelay true).
Those numbers are nearly useless without context: the rest of the info I requested. There was a reason for everything on the list. Folks contribute to the list out of the goodness of their hearts, and aren’t paid for back-and-forth tooth-pulling. If your index pool and bucket pool share say 3x HDDs or 3x coarse-IU QLC, then don’t expect much.
Sounds like you have the pg autoscaler enabled, which probably doesn’t help. Your index pool almost certainly needs more PGs. Probably log as well, or set the rgw log levels to 0.
Reason I am a bit vague about amount of OSDs and such, is that the numbers are not written on stone yet.
But can inform what you’re seeing today, esp if your poc is a gating one.
Currently using 40 in total
Which drives? Replicated pools? You aren’t equipped for the workload you’re throwing.
, but may or may not be able to “steal” more gear from another project, if the need arises. So I am rather fishing for advice in the form of “if you have >X OSDs, tune this and that” than specific to my exact current situation. ;-)
That’s the thing, it isn’t one size fits all. That advice is a function of your hardware choices.
Not doing anything special placement wise at the moment (not configured anything that would make the index and data go to different OSDs)
All pools on all OSDs? 3x replication?
Also note that my use-case is a bit non-standard in other ways. While I have a lot of objects, the application itself is internal within an organization and does not have many concurrent users.
Sounds ripe for tweaking for larger objects, and is more common than you think. Yours must be like <1KB? Tiny objects don’t work well with EC, so you’ll want 3x replication. Which costs more.
The performance demand is less than one would need if you have say a public facing web application with lots of visitors and peaks. So my main concern right now is having it run without internal server errors (the resharding thing), and fast enough so that my initial import completes within a couple weeks…
Trying to help. Can’t do that without info. Network and other tuning likely would help dramatically. Improving your PoC is the first step toward improving prod.
I do have control over the import application, but cannot change the application that will be using the bucket afterwise, so am stuck with their design choice for tiny objects and a single bucket.
Will play a bit with PGs and loglevel
PGs are a function of your replication and media. Disable the autoscaler and set pg_num manually so that the PGS column at right of `Ceph osd df` shows like 300, assuming you have conventional TLC. Index pool is crucial for you, I’d set pg_num to like 256. You probably have like 32 now. The buckets pool is important too. Put an RGW (orvtwo!) every node and use nginx, haproxy, or spendy hw to load balance, exposing a VIP endpoint. I was trying to advise on this, but can’t without the requested node details. This is the last time I’ll ask.
- Think the main delay is just Ceph wanting to make sure everything is sync’ed to storage before reporting success. So that is why I am making a lot of concurrent connections to perform multiple PUT requests simultaneously. But even with 250 connections, it only does around 5000 objects per second according to the “object ingress/egress” Grafana graph. Can probably raise it some more…
With one RGW you aren’t going to get far, unless you have a 500 core CPU, and probably not even than.
I wonder if that also applies to tiny objects.
Especially to. Metadata and intake costs way more than shoveling bits. The header stuff, metadata, index updates, persistence ops are the lion’s share of the overall incremental cost, vs background noise when you’re storing pirated TV episodes dubbed into Portuguese.
As I have the impression I am slowed down by RGW/Ceph waiting for individual objects to be synced to storage, instead of having to compute much for a tiny object.
In part. Far more to it than throughput.
I am experimenting a bit with the settings at the moment. If I raise the amount of concurrent connections from 250 to 1024 with a single RGW it does raise the number of ingress objects per second from 5k to 9k. Which means that the RGW was not quite saturated before.
Fair enough. Probably doesn’t do your average latency any favors, but you don’t care.
And the server the RGW is on, is only using 14% CPU (1 min avg, taken over all CPU cores)
Yep, cores usually aren’t the limitation. Run `powertop` and see if you’re dipping into C6, which can be more crippling than people often realize. You will need to LB proxy RGWs in prod anyway, start thinking that way. RADOS itself often isn’t the bottleneck here.
If I create a DNS entry that includes all 4 RGW IP, and tell it to connect to that with 1024 connections, I see in netstat that it indeeds spreads the connections over the 4 RGW,
Round robin LB. Not for pros but easy to do.
but I get the same 9k objects/sec.
Aggregate?
No improvement at all with that, at least not when it is just my import program running with that number of connections...
(I also noticed that the AWS SDK is not really written with small objects in mind. By default it only sets up a mere 25 connections. You have to lie to the SDK that you have a 400 Gbit connection by calling .targetThroughputInGbps(400), to persuade it to setup 1024 connections.
I’m not at all familiar with that SDK. Are you using the STANDARD storage class?
Even though those connections only use up roughly 400 Mbit of client bandwidth in practice due to the tiny objects, and a similar amount for Ceph’s internal traffic. Nowhere near the 2x25 Gbit SFP28 the servers have).
LACP bonded together, no replication network? Which xmit hash policy?
Yours sincerely,
Floris Bos _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
Anthony D'Atri
-
Floris Bos