Hi, I am using ceph 19.2.2 on Rocky 9 in a two-site Multisite configuration. I am using IAM roles for bucket access and all works fine, except the following. I have a role that people can sts assume on the master site. On the secondary site they can also sts assume the role, with the same credentials as on the master, as they are listed in the AssumeRolePolicyDocument for the role. When listing a bucket (that is in the role) on the master side 'using aws s3 --endpoint <master endpoint>, all is well. When doing a listing of the bucket on the secondary side, using --endpoint <slave endpoint> I get a code: InvalidAccessKey. When I use s3api to list a given key in that same bucket, I get error 403 on the secondary side. If I use the master side, I get a 200 and am able to list the object. The access keys and secret access key id are the same for the user who assumes the role on both sides, as they should be. Here is what i believe is the problem. If I list the role on both sides using: radosgw-admin role get --role-name <rolename> I get a different RoleID for the role on each side! So what I decided to do was pull all the roles on both sides using boto3/boto and iam. I discovered that there are TWO roles with same name on the secondary side, identical including the RoleID But this RoleId does NOT match that on the master side. I think this stems from an earlier release of ceph where roles were not included in metadata and I had to push them manually to the secondaryside and I must have made a mistake. All my other roles work fine on both sides and they are identical. I guess no one used the role on the secondary side until recently. Any idea how I can fix this? Will deleting the role delete both of the bad roles on the secondary side and then I just recreate it? Or should I just create a new role and have them use that and leave the 'duplicate' role there. -Chris