Hi, I am doing an attempt at loading a large number (billions) of tiny objects from other existing system into Ceph (18.2.4 reef) RadosGW/S3. I wrote a little program for that using the AWS S3 Java SDK (using CRT/S3AsyncClient), and am using 250 simultanous connections to get the data in. However I am experiencing problems when it starts to dynamically reshard. The first problem I was experiencing is that the SDK then complains connection throughput is too low, and errors out: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Http connectio n channel shut down due to failure to meet throughput minimum == While I configured the SDK to retry several times on errors, that does not seem to be happening. Or perhaps it does retry, but using one of the other connections, which obviously will fail as well for the same reason. Anyway, I shoved this issue under the carpet by configuring a very high timeout (creating a S3CrtConnectionHealthConfiguration and setting minimumThroughputTimeout() ) That seemed to work around it for a while. However I found out not much later that if the resharding takes longer than some minutes, RadosGW starts returning HTTP status code 500 to PUT requests: == Sep 27 18:14:33 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:15:03 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:21:59 myserver radosgw[6395]: INFO: RGWReshardLock::lock found lock on datastore:585f246e-937f-43e3-8966-8f1329279c81.54352.8 to be held by another RGW process; skipping for now [...] Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 9716400394278448257 450.223876953s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-16f7e. ret=(2 300) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 14148855427882423133 450.231872559s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-15b3. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj block_while_resharding ERROR: bucket is still resharding, please retry Sep 27 18:22:04 myserver radosgw[6395]: req 2645819454948921814 450.239868164s s3:put_obj ERROR: bucket shard callback failed. obj=some-obj-147ea. ret=(23 00) Unknown error 2300 Sep 27 18:22:04 myserver radosgw[6395]: WARNING: set_req_state_err err_no=2300 resorting to 500 Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7865710 op status=-2300 http_status=500 latency=450.223876953s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfc7ff4710 op status=-2300 http_status=500 latency=450.231872559s ====== Sep 27 18:22:04 myserver radosgw[6395]: ====== req done req=0x7fcfca7c3710 op status=-2300 http_status=500 latency=450.239868164s ====== Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7865710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.851 +0000] "PUT /datastore/some-obj-16f7e HTTP/1.1" 500 905 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.223876953s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfc7ff4710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.843 +0000] "PUT /datastore/some-obj-15b3 HTTP/1.1" 500 293 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.231872559s Sep 27 18:22:04 myserver radosgw[6395]: beast: 0x7fcfca7c3710: 10.x.x.x - bucket_user [27/Sep/2024:16:14:33.835 +0000] "PUT /datastore/some-obj-147ea HTTP/1.1" 500 318 - "aws-sdk-java/2.28.5 Linux/5.4.0-192-generic OpenJDK_64-Bit_Server_VM/25.222-b10 Java/1.8.0_222 vendor/AdoptOpenJDK io/async http/s3crt cfg/retry-mode/standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=450.239868164s S == AWS SDK considers 500 quite fatal: == Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates internal server error == No retries. Is it expected that resharding can take so long? (in a setup with all NVMe drives) And is it correct behavior that it returns HTTP response code 500, instead of something that could indicate it is a retry'able condition? If I would add my own code that does retry for a very long time, is there any way I can detect the 500 is due to the resharding, instead of some other condition that do is fatal? Also, is there any more efficient way to get a large amount of objects into Ceph than individual PUTs? Yours sincerely, Floris Bos