Recent multisite testing has uncovered a regression on Squid that happens when secondary zones are upgraded to Squid before the metadata master zone. User metadata replicates incorrectly in this configuration, such that their access keys are "inactive". As a result, these users are denied access to API requests on that secondary zone. This is tracked in https://tracker.ceph.com/issues/68985, and we'll prioritize the backport for 19.2.1. In the meantime, we strongly recommend upgrading the metadata master zone before others.