Hey all, I've been having some problems with a relatively basic rgw multisite set up. I initially had it working a few weeks ago with a simple config but rebuilt it with ssl enabled for the endpoints and then started having some problems to work through. I eventually had that working but then tore it all down so I could verify the recreation steps but now I'm having problems with the multisite. There are a few different errors.... I set up the master zone without problems, but if I use the rgw module with a token and spec it fails to sync the sync user and the primary zone can't sync back from the secondary zone. A permission denied error for data syncing primary->secondary but secondary-> primary works because it has the credentials. So i used to be able to manually create the user on the secondary zone and get data syncing but the metadata wouldn't sync with the master showing an error message but when I check the mdlog on the primary it looks okay. 2025-09-11T15:15:18.771+0000 7fc29e96c640 1 req 14061042141473454829 0.001000005s failed to read mdlog history: (2) No such file or directory I ended up trying a full clean reset of rgw on both clusters so I deleted the daemons and all related pools. The primary starts up but I still have the problem of the sync user not being created on the secondary and now when I try to create a user on the secondary for syncing or just for the dashboard to work I get errors that I've tracked down to https://github.com/ceph/ceph/blob/62bcf65e8c0995783bb3e368909716346874ad62/s... but am not quite following whats all going on in the source to determine why. # radosgw-admin user create --uid=dashboard --display-name=dashboard --system couldn't init storage provider # radosgw-admin user create --uid=sysuser-asdf --display-name=sysuser-asdf --access-key=xxxxxxxxxxxxx --secret-key=yyyyyyyyyyyyyyy --system --rgw-realm=asdf Unable to initialize site config. Any thoughts? It seems like I may have messed something up in the ceph config database from multiple tear downs and restarts of the multisite setup but I'm not seeing what it could be. ceph 19.2.1 using cephadm on alma 9 Thanks! Kevin