I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated. That resolved the issue. Any chance you also failed the MGR during the upgrade? I mean deliberately, not the failover that it has do perform on its own. I assume your cluster is cephadm managed? Zitat von Jay <ceph-users@ceph.io>:
Looks like similar things are already being discussed on this list, but I'm not sure, so here goes.
Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages:
[WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes
I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong?
Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere?
Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be:
stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id}
And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation?
Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this:
ceph-volume lvm list $OSD_ID --format json and find the lv_path field
ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING
but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this:
# ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij",
# ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ...
Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me.
0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrading-and-rotating-cephx-keys> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph-osd-lockbox-key-rotate>
-- Jay ceph@tsuraan.net _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io