I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated. That resolved the issue. Any chance you also failed the MGR during the upgrade? I mean deliberately, not the failover that it has do perform on its own.
No, just the normal stop, rotate, import, and start again. Except for the OSDs, it all went without any apparent problems. It does look like I rotated my mds keys before the mgr ones though. Could that be a problem?
I assume your cluster is cephadm managed?
No, it's all direct on the OS. I've been upgrading this cluster since either dumpling or firefly. It's outlived a lot of hardware, a few distributions and a management tool or two, but it's been going without a hitch until this key rotation thing. Hopefully I'll be able to get it healthy again.