Looks like similar things are already being discussed on this list, but I'm not sure, so here goes. Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages: [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong? Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere? Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be: stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id} And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation? Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this: ceph-volume lvm list $OSD_ID --format json and find the lv_path field ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this: # ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij", # ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ... Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me. 0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrading-and-rotating-cephx-keys> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph-osd-lockbox-key-rotate> -- Jay ceph@tsuraan.net