quincy user metadata constantly changing versions on multisite slave with radosgw roles

19 Apr 2023

      Hi,

I am using 17.2.6 on rocky linux for both the master and the slave site
I noticed that:
radosgw-admin sync status
often shows that the metadata sync is behind a minute or two on the slave. This didn't make sense, as the metadata isn't changing as far as I know.
radosgw-admin mdlog list 

(on slave) showed me that there were user changes in metadata very often. After doing a little research, here is a scenario I was able to develop:

1. user continually writes to a bucket he owns, pointing his aws cli (for this test) to the master side endpoint as specified in  ~/.aws/config

while this is running, do the following on the slave:
radosgw-admin metadata get user:<username>
This always shows the same result for the user, no changes. The data gets to the slave side bucket.

2. Restart the continual copy, but this time use a role that the user is a member of via profile in ~/.aws/credentials and .~/aws/config, again writing to the master endpoint as specified in ~/.aws/config

aws --profile <profile> s3 cp <file> s3://<bucket>/file
where profile is set up to use a role definiiton. The data gets to the bucket on both sides. I do not have access if I do not use the role  profille (to confirm I set it up right) However, while doing this second test, if I continually do:
radosgw-admin metadata get user:<username>
on the slave, I see a definite increase in versions. Here is a section of the json output:

"key": "user:<username>",
"ver": {   "tag": "somestring",   "ver:" 12145}
the 12145 value increases over and over again, and the mtime value in the json output increases too based on the current date. (not shown here).  The same value, when queried on the master side, remains 1, and the mtime value is the date the user was created or last changed by an admin. If I write a file only once, the vers value increases by 1 too, but not sure if the increase in vers is ncessarily 1:1 with the number of writes. This seems to be the source of my continual metadata lag.  Am I missing something? I suspect that this has been happening for awhile and not specific to 17.2.6 as I just upgraded and the ver value is over 12000 for the user that I discovered. (I used a python script to sync roles between master and slave prior to 17.2.6. Now roles are replicated in 17.2.6).
-Chris

Christopher Durham

Casey Bodley

tags

participants (2)