Hi, we have customer that used multisite https successfully in a pre-production state. They switched to a stretched cluster later, but the replication worked (and still works). The configs were slightly different, they never tried both http and https simultanously (civetweb port=7480+443s) as far as I know, only https: rgw frontends = "civetweb port=443s ssl_certificate=/path/to/certificate" They also have haproxy servers in front of the gateways, so they were specified in the zonegroup config instead of the rgw servers, but that shouldn't make a difference: "endpoints": [ "https://<HAPROXY1>:80443", "https://<HAPROXY2>:81443", This was the haproxy config: ---snip--- [...] frontend app_frontend1 bind *:80443 ssl crt /path/to/cert mode tcp option clitcpka default_backend app_backend frontend app_frontend2 bind *:81443 ssl crt /path/to/cert mode tcp option clitcpka default_backend app_backend backend app_backend [...] server rgw1 <IP>:443 weight 1 maxconn 100 check ssl verify required ca-file /... server rgw2 <IP>:443 weight 1 maxconn 100 check ssl verify required ca-file /... ---snip--- This is all in Luminous. Regards, Eugen Zitat von Richard Kearsley <richard@kearsley.me>:

Hi Just chasing up on this.. is anyone using multisite with HTTPS zone endpoints? I could not find any examples... should it work?

Thanks Richard

On 31 March 2020 22:35:30 BST, Richard Kearsley <richard@kearsley.me> wrote:

...

Hi there

I have a fairly simple ceph multisite configuration with 2 ceph clusters in 2 different datacenters in the same city The rgws have this config for ssl:

rgw_frontends = civetweb port=7480+443s ssl_certificate=/opt/ssl/ceph-bundle.pem

The certificate is a real issued certificate, not self signed

I configured the multisite with the guide from https://docs.ceph.com/docs/nautilus/radosgw/multisite/ More or less ok so far, some learning curve but that's ok

I can access and upload to buckets at both endpoints with s3 client using https - https://ceph01cs1.domain.com and https://ceph01cs2.domain.com - all good

Now the problem seems to be when my zones in the zonegroup use https endpoints, e.g.

{ "id": "4c6774fb-01eb-41fe-a74a-c2693f8e69fc", "name": "eu", "api_name": "eu", "is_master": "true", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "zones": [ { "id": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "name": "ceph01cs1", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "fec1fec8-a3c1-454d-8ed2-2c1da45f9c33", "name": "ceph01cs2", "endpoints": [ "https://ceph01cs2.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "08921dd5-1523-41b6-908f-2f58aa38c969" }

Meta syncs ok - buckets and users get created, but data doesn't, and period can be commited and appears on both clusters I can also curl between the two clusters over 443 However, data sync gets stuck on 'init':

realm 08921dd5-1523-41b6-908f-2f58aa38c969 (world) zonegroup 4c6774fb-01eb-41fe-a74a-c2693f8e69fc (eu) zone 0c203df2-6f31-4ad1-a899-91f85bf34c4e (ceph01cs2) metadata sync no sync (zone is master) data sync source: fec1fec8-a3c1-454d-8ed2-2c1da45f9c33 (ceph01cs1) init full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]

I find errors like: 2020-03-31 20:27:11.372 7f60c84e1700 0 RGW-SYNC:data:sync: ERROR: failed to init sync, retcode=-16 2020-03-31 20:27:29.548 7f60c84e1700 0 RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote data log shards 2020-03-31 20:29:48.499 7f60c94e3700 0 RGW-SYNC:meta: ERROR: failed to

fetch all metadata keys

If I change the endpoints in the zonegroup to plain http, e.g. http://ceph01cs1.domain.com:7480 and http://ceph01cs2.domain.com:7480 then sync starts!

So my question, and I couldn't find any examples of people using https to sync.. are https endpoints supported with multisite? and why would meta work over https but not data?

Many thanks Richard _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io