Issue about "ERROR: S3 Temporary Error: Request failed for: /. Please try again later"
I have mapped port 32505 to 23860, however when connect via s3cmd it fails with "ERROR: S3 Temporary Error: Request failed for: /. Please try again later." . has anyone ecounted same issue? [root@vm-04 ~]# s3cmd ls WARNING: Retrying failed request: / ('') WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 6 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 9 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 12 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 15 sec... ERROR: S3 Temporary Error: Request failed for: /. Please try again later. [root@vm-04 ~]# [root@vm-04 ~]# cat .s3cfg [default] access_key = E9BJAC6QKLTOKVJR4TZC secret_key = KBvm0YqeAFtVM0I1gUhX5wrnwpHZZ5SLhiUa56ss host_base = xxxxxx:23860 host_bucket = xxxxxx:23860/bucket-king use_https = False signature_v2 = False [root@vm-04 ~]# bash-4.4$ ceph -s cluster: id: dd6a082e-868d-44e0-b93b-0f4956e5dd1a health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 4m) mgr: b(active, since 4m), standbys: a mds: 1/1 daemons up, 1 standby osd: 3 osds: 3 up (since 4m), 3 in (since 26m) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 233 pgs objects: 548 objects, 808 MiB usage: 2.4 GiB used, 148 GiB / 150 GiB avail pgs: 233 active+clean bash-4.4$ [root@vm-01 ~]# kubectl get svc -nrook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rook-ceph-mgr ClusterIP 10.105.165.62 <none> 9283/TCP 26m rook-ceph-mgr-dashboard ClusterIP 10.102.18.178 <none> 8443/TCP 26m rook-ceph-mgr-dashboard-external-https NodePort 10.100.66.254 <none> 8443:31091/TCP 24m rook-ceph-rgw-my-store ClusterIP 10.104.146.147 <none> 80/TCP 16m rook-ceph-rgw-my-store-external NodePort 10.106.39.241 <none> 80:32505/TCP 14m [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# kubectl get pods -nrook-ceph | grep rgw rook-ceph-rgw-my-store-a-67d579d99b-5xd64 2/2 Running 0 5m22s [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# kubectl -nrook-ceph logs rook-ceph-rgw-my-store-a-67d579d99b-5xd64 | tail -20 Defaulted container "rgw" out of: rgw, log-collector, chown-container-data-dir (init) debug 2024-04-11T13:42:33.635+0000 7fee46b43700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.630 +0000] "DELETE /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 204 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.005000210s debug 2024-04-11T13:42:33.635+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.647+0000 7fee4333c700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.012000503s ====== debug 2024-04-11T13:42:33.647+0000 7fee4333c700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.635 +0000] "PUT /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 HTTP/1.1" 200 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.012000503s debug 2024-04-11T13:42:33.648+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.666+0000 7fee3b32c700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.018000754s ====== debug 2024-04-11T13:42:33.666+0000 7fee3b32c700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.648 +0000] "PUT /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 200 21 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.018000754s debug 2024-04-11T13:42:33.667+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.669+0000 7fee4e352700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.002000083s ====== debug 2024-04-11T13:42:33.669+0000 7fee4e352700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.667 +0000] "GET /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 200 21 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.002000083s debug 2024-04-11T13:42:33.710+0000 7fee43b3d700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.719+0000 7fee2e312700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=204 latency=0.009000377s ====== debug 2024-04-11T13:42:33.719+0000 7fee2e312700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.710 +0000] "DELETE /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 204 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.009000377s debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 ====== starting new request req=0x7fee078c0730 ===== debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 ====== req done req=0x7fee078c0730 op status=0 http_status=200 latency=0.000000000s ====== debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 beast: 0x7fee078c0730: 192.168.100.8 - - [11/Apr/2024:13:42:37.863 +0000] "GET /swift/healthcheck HTTP/1.1" 200 0 - "kube-probe/1.28" - latency=0.000000000s debug 2024-04-11T13:42:41.078+0000 7fef3a54b700 0 rgw UsageLogger: WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 ====== starting new request req=0x7fee07a43730 ===== debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 ====== req done req=0x7fee07a43730 op status=0 http_status=200 latency=0.000000000s ====== debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 beast: 0x7fee07a43730: 192.168.100.8 - - [11/Apr/2024:13:42:47.862 +0000] "GET /swift/healthcheck HTTP/1.1" 200 0 - "kube-probe/1.28" - latency=0.000000000s [root@vm-01 ~]# [root@vm-01 ~]#
Add more information, once telnet the port, it will connected, but close by foreign host. [root@vm-04 ~]# telnet xxxxxx 23860 Trying xxxxxx... Connected to xxxxxx. Escape character is '^]'. Connection closed by foreign host. [root@vm-04 ~]# [root@vm-04 ~]# [root@vm-04 ~]# [root@vm-04 ~]# [root@vm-04 ~]# s3cmd ls s3://bucket-king -d host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'x-amz-date': '20240411T140729Z', 'Authorization': u'AWS4-HMAC-SHA256 Credential=E9BJAC6QKLTOKVJR4TZC/20240411/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=f830a69ed774e99312bab9137bcf0eeb75119d82dc7a86ee60e0607b67e48b87'} DEBUG: Processing request, please wait... DEBUG: get_hostname(bucket-king): xxxxxx :23860 DEBUG: ConnMan.get(): creating new connection: http://xxxxxx :23860 DEBUG: non-proxied HTTPConnection(xxxxxx 0, 23860) DEBUG: format_uri(): /bucket-king/?location DEBUG: Sending request method_string='GET', uri=u'/bucket-king/?location', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'x-amz-date': '20240411T140729Z', 'Authorization': u'AWS4-HMAC-SHA256 Credential=E9BJAC6QKLTOKVJR4TZC/20240411/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=f830a69ed774e99312bab9137bcf0eeb75119d82dc7a86ee60e0607b67e48b87'}, body=(0 bytes) DEBUG: Response: {} WARNING: Retrying failed request: /?location ('') WARNING: Waiting 15 sec...
Den tors 11 apr. 2024 kl 15:55 skrev <elite_stu@163.com>:
I have mapped port 32505 to 23860, however when connect via s3cmd it fails with "ERROR: S3 Temporary Error: Request failed for: /. Please try again later." . has anyone ecounted same issue?
[root@vm-04 ~]# s3cmd ls WARNING: Retrying failed request: / ('') WARNING: Waiting 3 sec...
[root@vm-04 ~]# cat .s3cfg [default] access_key = E9BJAC6QKLTOKVJR4TZC secret_key = KBvm0YqeAFtVM0I1gUhX5wrnwpHZZ5SLhiUa56ss host_base = xxxxxx:23860 host_bucket = xxxxxx:23860/bucket-king
^^^^^^^^^ this is not correct, you should not have the bucket name there. Leave it at host_bucket = xxxxxx:23860 -- May the most significant bit of your life be positive.
Thanks for your reply! but xxxxxx is the actual ip of my nodes, I mapped it on internet, so i use xxxxxx to replace the public ip.
I have tried to remove bucket name on host_bucket, and seems the same problem.. [root@vm-04 ~]# cat .s3cfg [default] access_key = E9BJAC6QKLTOKVJR4TZC secret_key = KBvm0YqeAFtVM0I1gUhX5wrnwpHZZ5SLhiUa56ss host_base = xxxxxxxxx:23860 host_bucket = xxxxxxxxx:23860 use_https = False signature_v2 = False [root@vm-04 ~]# [root@vm-04 ~]# s3cmd ls -d DEBUG: s3cmd version 2.3.0 DEBUG: ConfigParser: Reading file '/root/.s3cfg' DEBUG: ConfigParser: access_key->E9...17_chars...C DEBUG: ConfigParser: secret_key->KB...37_chars...s DEBUG: ConfigParser: host_base->xxxxxxxxx:23860 DEBUG: ConfigParser: host_bucket->xxxxxxxxx:23860 DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: signature_v2->False DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Unicodising 'ls' using UTF-8 DEBUG: Command: ls DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(None): xxxxxxxxx:23860 DEBUG: canonical_headers = host:xxxxxxxxx:23860 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240411T151844Z DEBUG: Canonical Request: GET / host:xxxxxxxxx:23860 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240411T151844Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=E9BJAC6QKLTOKVJR4TZC/20240411/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=04d20f39b5de9ad0991ac6ab875bebf473061d267d8898d9e50a3e89810a4d62', 'x-amz-date': '20240411T151844Z'} DEBUG: Processing request, please wait... DEBUG: get_hostname(None): xxxxxxxxx:23860 DEBUG: ConnMan.get(): creating new connection: http://xxxxxxxxx:23860 DEBUG: non-proxied HTTPConnection(xxxxxxxxx, 23860) DEBUG: format_uri(): / DEBUG: Sending request method_string='GET', uri=u'/', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=E9BJAC6QKLTOKVJR4TZC/20240411/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=04d20f39b5de9ad0991ac6ab875bebf473061d267d8898d9e50a3e89810a4d62', 'x-amz-date': '20240411T151844Z'}, body=(0 bytes) DEBUG: Response: {} WARNING: Retrying failed request: / ('') WARNING: Waiting 3 sec... DEBUG: Using signature v4 DEBUG: get_hostname(None): xxxxxxxxx:23860 DEBUG: canonical_headers = host:xxxxxxxxx:23860 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240411T151847Z
participants (3)
-
elite_stu＠163.com
-
Janne Johansson
-
king .