Multisite RGW setup not working when following the docs step by step

30 Aug 2023

      Hello,

My goal is to setup multisite RGW with 2 separate CEPH clusters in separate datacenters, where RGW data are being replicated. I created a lab for this purpose in both locations (with latest reef ceph installed using cephadm) and tried to follow this guide: https://docs.ceph.com/en/reef/radosgw/multisite/

Unfortunatelly, even after multiple attempts it always failed when creating a secondary zone. I could succesfully pull the realm from master, but that was pretty much the last trully succesful step. I can notice that immediately after pulling the realm to secondary, radosgw-admin user list returns an empty list (which IMHO should contain replicated user list from master). Continuing by setting default real and zonegroup and creating the secondary zone in secondary cluster I end up having 2 zones in each cluster, both seemingly in same zonegroup, but with replication failing - this is what I see in sync status:

(master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin sync status
          realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
      zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
           zone 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
   current time 2023-08-30T12:58:12Z
zonegroup features enabled: resharding
                   disabled: compress-encrypted
  metadata sync no sync (zone is master)
2023-08-30T12:58:13.991+0000 7f583a52c780  0 ERROR: failed to fetch datalog info
      data sync source: 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
                        failed to retrieve sync info: (13) Permission denied

(secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin sync status
          realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
      zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
           zone 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
   current time 2023-08-30T12:58:54Z
zonegroup features enabled: resharding
                   disabled: compress-encrypted
  metadata sync failed to read sync status: (2) No such file or directory
2023-08-30T12:58:55.617+0000 7ff37c9db780  0 ERROR: failed to fetch datalog info
      data sync source: 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
                        failed to retrieve sync info: (13) Permission denied

In master there is one user created during the process (synchronization-user), on slave there are no users and when I try to re-create this synchronization user it complains I shouldn't even try and instead execute the command on master. I can see same realm and zonegroup IDs on both sides, zone list is different though:

(master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin zone list
{
    "default_info": "97fb5842-713a-4995-8966-5afe1384f17f",
    "zones": [
        "cz-brn",
        "default"
    ]
}

(secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin zone list
{
    "default_info": "13a8c663-b241-4d8a-a424-8785fc539ec5",
    "zones": [
        "cz-hol",
        "default"
    ]
}

The permission denied error is puzzling me - could it be because real pull didn't sync the users? I tried this multiple times with clean ceph install on both sides - and always ended up the same. I even tried force creating the same user with same secrets on the other side, but it didn't help. How can I debug what kind of secret is secondary trying to use when communicating with master? Could it be that this multisite RGW setup is not yet truly supported in reef? I noticed that the documentation itself seems written for older ceph versions, as there are no mentions about orchestrator (for example in steps where configuration files of RGW need to be edited, which is done differently when using cephadm).

I think that documentation is simply wrong at this time. Either it's missing some crucial steps, or it's outdated or otherwise unclear - simply by following all the steps as outlined there, you are likely to end up the same.

Thanks for help!

Petr Bena

Zac Dover

tags

participants (2)