this is ES error log： { "error": { "root_cause": [ { "type": "repository_verification_exception", "reason": "[test] path is not accessible on master node" } ], "type": "repository_verification_exception", "reason": "[test] path is not accessible on master node", "caused_by": { "type": "i_o_exception", "reason": "Unable to upload object [tests-CX3jGTbyRgOeOZJYci8MnQ/master.dat] using a single upload", "caused_by": { "type": "sdk_client_exception", "reason": "sdk_client_exception: Unable to execute HTTP request: oldelk-snapshot.rgw.abc.cn", "caused_by": { "type": "i_o_exception", "reason": "oldelk-snapshot.rgw.abc.cn" } } } }, "status": 500 } Tom Black <tom(a)pobox.store> 于2020年9月2日周三 下午4:55写道：

Hi Tom Thanks for the reply. Here is my nginx configuration. Did I miss something or is there some special option to set? What's more, our Flink can work well by connecting to the frontend. [image: image.png] Tom Black <tom(a)pobox.store> 于2020年9月3日周四 上午8:13写道：

I think you might need to set some headers. Here is what we use (connecting to Swift, but should be generally applicable). We are running nginx and swift (swift proxy server) on the same host. but again maybe some useful ideas for you to try (below). Note that we explicitly stop nginx writing a temporary copy of any objects being uploaded (that is the last 3 lines) --- config --- server {   listen       *:8443 ssl;   server_name  swift-proxy;   ssl on;   ssl_certificate           /var/*refacted*;   ssl_certificate_key       /var/*redacted*;   ssl_session_cache         shared:SSL:10m;   ssl_session_timeout       5m;   ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;   ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;   ssl_prefer_server_ciphers on;   client_max_body_size 5368709124;   index  index.html index.htm index.php;   access_log /var/log/nginx/swift-proxy-access.log combined;   error_log /var/log/nginx/swift-proxy-error.log;   location / {     proxy_pass            http://127.0.0.1:8080;     proxy_read_timeout    90;     proxy_connect_timeout 90;     proxy_redirect        off;     proxy_set_header      Host $host;     proxy_set_header      X-Real-IP $remote_addr;     proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;     proxy_set_header      Proxy "";     proxy_http_version 1.1;     proxy_max_temp_file_size 0;     proxy_request_buffering off;   } } On 3/09/20 2:19 pm, Zhenshi Zhou wrote: > Hi Tom > > Thanks for the reply. Here is my nginx configuration. > Did I miss something or is there some special option to set? > What's more, our Flink can work well by connecting to the frontend. > > image.png > > Tom Black &lt;tom(a)pobox.store&gt; 于2020年9月3日周四 上午8:13写道： > > It seems like your nginx has the wrong configuration for reverse > proxy > of S3. > > Thanks. > > Zhenshi Zhou wrote: > > this is ES error log： > > { > >    "error": { > >      "root_cause": [ > >        { > >          "type": "repository_verification_exception", > >          "reason": "[test] path  is not accessible on master node" > >        } > >      ], > >      "type": "repository_verification_exception", > >      "reason": "[test] path  is not accessible on master node", > >      "caused_by": { > >        "type": "i_o_exception", > >        "reason": "Unable to upload object > > [tests-CX3jGTbyRgOeOZJYci8MnQ/master.dat] using a single upload", > >        "caused_by": { > >          "type": "sdk_client_exception", > >          "reason": "sdk_client_exception: Unable to execute HTTP > > request: oldelk-snapshot.rgw.abc.cn > <http://oldelk-snapshot.rgw.abc.cn> > <http://oldelk-snapshot.rgw.abc.cn>", > >          "caused_by": { > >            "type": "i_o_exception", > >            "reason": "oldelk-snapshot.rgw.abc.cn > <http://oldelk-snapshot.rgw.abc.cn> > > <http://oldelk-snapshot.rgw.abc.cn>" > >          } > >        } > >      } > >    }, > >    "status": 500 > > } > > > > Tom Black &lt;tom(a)pobox.store&gt; 于2020年9月2日周三 下午4:55写道： > > > >     Zhenshi Zhou wrote: > >      > My fellows wanna use ceph rgw to store ES backup and > Nexus blobs. > >      > But the services cannot connect to the rgw with s3 > protocol when I > >      > provided them with the frontend nginx address(virtual > ip). Only when > >      > they use the backend rgw's address(real ip) the ES and > Nexus works > >      > well with rgw. > > > >     you should provide both the client and server's error logs. > > > >     Thanks. > >     _______________________________________________ > >     ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > >     <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> > >     To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> > >     <mailto:ceph-users-leave@ceph.io > <mailto:ceph-users-leave@ceph.io>> > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io