Hi, Could someone help me what is the issue with our dployment steps please? Initial RGW Cluster_1 ===================================================================================ADD_RGW_TO_CLUSTERCreate Default Realm - sudo radosgw-admin realm create --rgw-realm=default --defaultCreate Default Zone Group - sudo radosgw-admin zonegroup create --rgw-zonegroup=default --master --defaultCreate Default Zone - sudo radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=default --master --defaultUpdate Period - sudo radosgw-admin period update --rgw-realm=default --commitCreate RGW on 3 OSD_Node - sudo ceph orch apply rgw default default --placement=“3 bk-otpsmon-1001 bk-otpsmon-1002 bk-otpsmon-1003” ===================================================================================Initial RGW Cluster_2 - sudo ceph orch apply rgw default default --placement=“3 hy-otpsmon-2001 hy-otpsmon-2002 bk-otpsmon-2003" HOW_TO_CONFIGURE_CEPH_OCTOPUS_MULTISITE =================================================================================================== configure multisite radosgw-admin realm create \ --rgw-realm=agoda \ --default radosgw-admin zonegroup create \ --rgw-zonegroup=data \ --endpoints=http://bk-otpsmon-1001:80,http://bk-otpsmon-1002:80,http://bk-otpsmon-1003:8... \ --rgw-realm=agoda \ --master \ --default ***** BUG Run Command-line => Can't send shot command-line ***** radosgw-admin zone create --rgw-zonegroup=data --rgw-zone=bk --master --default --endpoints=http://bk-otpsmon-1001:80,http://bk-otpsmon-1002:80,http://bk-otpsmon-1003:8... ***** BUG Can't Delete Pool ***** ***** ceph config set mon mon_allow_pool_delete true ***** ceph tell mon.\* injectargs '--mon-allow-pool-delete=true' ***** (optional) systemctl |grep -i ceph ***** (optional) systemctl restart ceph-85b0c358-e2a2-11ea-8864-000c29fa922a@mon.bk-otpsmon-1001.service # radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default # radosgw-admin period update --commit # radosgw-admin zone delete --rgw-zone=default # radosgw-admin period update --commit # radosgw-admin zonegroup delete --rgw-zonegroup=default # radosgw-admin period update --commit # ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it # ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it # ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it radosgw-admin user create --uid="ceph-sync" --display-name="ceph-sync" --system "user": "ceph-sync", "access_key": "VWV6566957QGVDV6ITJM", "secret_key": "qAEGXslUHBeWv7O6VMCmdo0z2AgMyBZlcKqg38H7" radosgw-admin zone modify \ --rgw-zone=bk \ --access-key=VWV6566957QGVDV6ITJM \ --secret=qAEGXslUHBeWv7O6VMCmdo0z2AgMyBZlcKqg38H7 radosgw-admin period update --rgw-realm=agoda --commit Configure Rados Gateway Client at RGW Nodes [client.rgw.bk-otpsmon-1001] host = bk-otpsmon-1001 rgw frontends = "civetweb port=80" rgw_zone=bk [client.rgw.bk-otpsmon-1002] host = bk-otpsmon-1002 rgw frontends = "civetweb port=80" rgw_zone=bk [client.rgw.bk-otpsmon-1003] host = bk-otpsmon-1003 rgw frontends = "civetweb port=80" rgw_zone=bk Restart All RGW Container Node rgw_1: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836@rgw.default.default.bk-otpsmon-1001.atdfmv.service rgw_2: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836@rgw.default.default.bk-otpsmon-1002.wkieqj.service rgw_3: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836@rgw.default.default.bk-otpsmon-1003.jpyzdq.service Enable All RGW Container Node rgw_1: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836@rgw.default.default.bk-otpsmon-1001.atdfmv.service rgw_2: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836@rgw.default.default.bk-otpsmon-1002.wkieqj.service rgw_3: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836@rgw.default.default.bk-otpsmon-1003.jpyzdq.service ===================================== Secondary Zones radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret} radosgw-admin realm pull --url=http://bk-otpsmon-1001:80 --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 radosgw-admin realm pull --url=http://bk-otpsmon-1001:80 --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 --rgw-realm=agoda And here is the errors: Error From Cluster_2 After Realm Pull[root@hy-otpsmon-2001 ~]# radosgw-admin realm pull --url=http://bk-otpsmon-1001:80<http://bk-otpsmon-1001/> --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 --rgw-realm=agoda request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. [root@hy-otpsmon-2001 ~]# Error From Cluster_1 RGW Container LogSep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.963+0000 7f3b0747f700 1 ====== starting new request req=0x7f3b455118a0 ===== Sep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.967+0000 7f3b0747f700 1 op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 Sep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.967+0000 7f3b0747f700 1 ====== req done req=0x7f3b455118a0 op status=0 http_status=403 latency=0.002999559s ====== All help appreciated. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.