Multisite RGW setup not working when following the docs step by step - ceph-users

30 Aug 2023

Hello,

My goal is to setup multisite RGW with 2 separate CEPH clusters in separate datacenters,
where RGW data are being replicated. I created a lab for this purpose in both locations
(with latest reef ceph installed using cephadm) and tried to follow this guide:
https://docs.ceph.com/en/reef/radosgw/multisite/

Unfortunatelly, even after multiple attempts it always failed when creating a secondary
zone. I could succesfully pull the realm from master, but that was pretty much the last
trully succesful step. I can notice that immediately after pulling the realm to secondary,
radosgw-admin user list returns an empty list (which IMHO should contain replicated user
list from master). Continuing by setting default real and zonegroup and creating the
secondary zone in secondary cluster I end up having 2 zones in each cluster, both
seemingly in same zonegroup, but with replication failing - this is what I see in sync
status:

(master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin sync status
          realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
      zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
           zone 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
   current time 2023-08-30T12:58:12Z
zonegroup features enabled: resharding
                   disabled: compress-encrypted
  metadata sync no sync (zone is master)
2023-08-30T12:58:13.991+0000 7f583a52c780  0 ERROR: failed to fetch datalog info
      data sync source: 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
                        failed to retrieve sync info: (13) Permission denied

(secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin sync status
          realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
      zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
           zone 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
   current time 2023-08-30T12:58:54Z
zonegroup features enabled: resharding
                   disabled: compress-encrypted
  metadata sync failed to read sync status: (2) No such file or directory
2023-08-30T12:58:55.617+0000 7ff37c9db780  0 ERROR: failed to fetch datalog info
      data sync source: 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
                        failed to retrieve sync info: (13) Permission denied

In master there is one user created during the process (synchronization-user), on slave
there are no users and when I try to re-create this synchronization user it complains I
shouldn't even try and instead execute the command on master. I can see same realm and
zonegroup IDs on both sides, zone list is different though:

(master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin zone list
{
    "default_info": "97fb5842-713a-4995-8966-5afe1384f17f",
    "zones": [
        "cz-brn",
        "default"
    ]
}

(secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin zone list
{
    "default_info": "13a8c663-b241-4d8a-a424-8785fc539ec5",
    "zones": [
        "cz-hol",
        "default"
    ]
}

The permission denied error is puzzling me - could it be because real pull didn't sync
the users? I tried this multiple times with clean ceph install on both sides - and always
ended up the same. I even tried force creating the same user with same secrets on the
other side, but it didn't help. How can I debug what kind of secret is secondary
trying to use when communicating with master? Could it be that this multisite RGW setup is
not yet truly supported in reef? I noticed that the documentation itself seems written for
older ceph versions, as there are no mentions about orchestrator (for example in steps
where configuration files of RGW need to be edited, which is done differently when using
cephadm).

I think that documentation is simply wrong at this time. Either it's missing some
crucial steps, or it's outdated or otherwise unclear - simply by following all the
steps as outlined there, you are likely to end up the same.

Thanks for help!