20.2.4 Upgrade questions
Looks like similar things are already being discussed on this list, but I'm not sure, so here goes. Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages: [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong? Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere? Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be: stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id} And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation? Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this: ceph-volume lvm list $OSD_ID --format json and find the lv_path field ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this: # ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij", # ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ... Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me. 0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrading-and-rotating-cephx-keys> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph-osd-lockbox-key-rotate> -- Jay ceph@tsuraan.net
I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated. That resolved the issue. Any chance you also failed the MGR during the upgrade? I mean deliberately, not the failover that it has do perform on its own. I assume your cluster is cephadm managed? Zitat von Jay <ceph-users@ceph.io>:
Looks like similar things are already being discussed on this list, but I'm not sure, so here goes.
Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages:
[WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes
I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong?
Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere?
Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be:
stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id}
And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation?
Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this:
ceph-volume lvm list $OSD_ID --format json and find the lv_path field
ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING
but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this:
# ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij",
# ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ...
Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me.
0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrading-and-rotating-cephx-keys> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph-osd-lockbox-key-rotate>
-- Jay ceph@tsuraan.net _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated. That resolved the issue. Any chance you also failed the MGR during the upgrade? I mean deliberately, not the failover that it has do perform on its own.
No, just the normal stop, rotate, import, and start again. Except for the OSDs, it all went without any apparent problems. It does look like I rotated my mds keys before the mgr ones though. Could that be a problem?
I assume your cluster is cephadm managed?
No, it's all direct on the OS. I've been upgrading this cluster since either dumpling or firefly. It's outlived a lot of hardware, a few distributions and a management tool or two, but it's been going without a hitch until this key rotation thing. Hopefully I'll be able to get it healthy again.
I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated.
Sorry, I think I misunderstood this on my first reading. Do you mean go through and rotate the mon and mgr keys on the existing cluster and see if that helps? I was thinking you meant roll back the cluster (like a bunch of VMs or something) and try again, but that doesn't seem so likely, now that I think about it. I should just try the key rotations again?
Hi all, We are running into a problem cleaning up abandoned multipart uploads on RGW and would appreciate a sanity check of our analysis, plus any advice on a better approach. Environment ----------- - Ceph Squid 19.2.6, cephadm-deployed, single zone. - Data pool: EC on HDD (12 x 24 TB per host, DB/WAL on NVMe). Index pool on NVMe. - 12 RGW daemons behind the cephadm ingress (HAProxy). - rgw_gc_max_objs = 128, rgw_gc_max_queue_size at its default (~128 MiB). What happened ------------- A customer's backup appliance (a NAS uploading backup files over S3) left 9 incomplete multipart uploads in one bucket: 3 objects x 3 attempts each. Each upload has ~26,000 parts of 128 MiB (~3.3 TiB per upload, ~31.6 TB in total), and the bucket has 0 visible objects. We suspect the uploads exceeded the 10,000-part limit, so CompleteMultipartUpload failed and the appliance retried from scratch, leaving the previous upload behind. We have not confirmed this yet (see question 5). Each 128 MiB part is 1 __multipart_ head plus 31 x 4 MiB __shadow_ objects, so each upload is ~830k RADOS objects and the bucket ~7.5M. Symptoms -------- 1. The customer's DeleteBucket timed out. As we understand it, RGW aborts pending MPUs synchronously inside DeleteBucket, which cannot finish within an HTTP request at this size. AbortMultipartUpload from the client returned 503. 2. "radosgw-admin bucket rm --purge-objects --bypass-gc" immediately logs: garbage collection: RGWGC::send_split_chain - send chain returned error: -28 abort: gc->send_chain() returned -28 delete_objs_inline: removing <data-pool>:<marker>__shadow_<key>.2~<upload-id>.1_1 delete_objs_inline: refcount put returned error -2 ... It then deletes the rest of the chain inline, one object at a time, from a single thread. Each refcount put takes between ~10 ms and ~2 s, so it averages roughly 1 object/s. --max-concurrent-ios has no effect on this phase, and --bypass-gc does not seem to apply to the MPU abort. 3. The GC as a whole was nearly empty at the time. The -28 is not global pressure. Our analysis (from reading the Squid source; please correct us) --------------------------------------------------------------- - RadosMultipartUpload::abort() collects the manifests of all parts into ONE chain and sends it to the GC with tag = upload id. - The GC shard is chosen from the tag (hash(tag) % rgw_gc_max_objs). RGWGC::send_split_chain() splits the chain into rgw_max_chunk_size (4 MiB) entries, but every entry keeps the same tag and therefore goes to the SAME gc.N shard. - Each shard is a single cls_queue object capped at rgw_gc_max_queue_size (~128 MiB, bounded by osd_max_object_size). A ~830k-object chain encodes to roughly 130-160 MB, so it cannot fit even in an empty shard. We observed 31 x 4 MiB entries with that upload's tag filling one shard. The enqueue then fails with ENOSPC, and the remainder of the chain is deleted inline via delete_objs_inline(). Consequences ------------ a) The multipart meta object is only removed at the very end of the abort. Any interruption (client timeout, Ctrl-C, RGW restart) means the next attempt restarts from part 1. It re-walks objects that are already gone (refcount put -> -2, which is still ~1 obj/s), and the shard is still full from the previous attempt, so this time the whole chain goes inline. b) While a shard is full, any unrelated deletion whose tag hashes to that shard also falls back to inline deletion. c) Raising rgw_gc_max_queue_size does not help: it is capped by osd_max_object_size, and existing queues are not resized (queue_init returns EEXIST). Workaround we are using ----------------------- 1. Drain the GC first (radosgw-admin gc process --include-all). 2. Put an AbortIncompleteMultipartUpload lifecycle rule on the bucket and run: radosgw-admin lc process --bucket=<bucket> --rgw-lc-max-wp-worker=9 This aborts all 9 uploads in parallel. Each upload fills its own GC shard (~75% of its chain), and the remaining ~25% is deleted inline, but with 9 threads instead of 1. Note that in this code path the inline deletion is logged under debug_rgw_lifecycle, not debug_rgw. 3. Run several "radosgw-admin gc process --include-all" in parallel, at most one per full shard, since each shard is processed under its own lock. 4. Finally, bucket rm --purge-objects --bypass-gc, which should be fast once no MPUs remain. Questions --------- 1. Is this a known issue? Is there a tracker for MPU aborts overflowing a single GC queue shard? 2. Would it make sense for the abort path to honour --bypass-gc in radosgw-admin, or to use AIO with concurrency for delete_objs_inline() instead of deleting one object at a time? 3. Could the abort make incremental progress (e.g. drop part index entries as they are released) so that a retry does not restart from part 1? 4. Is there any safe, supported way to resize existing GC queue objects, or a recommended setting for workloads with very large MPUs? 5. Does RGW enforce the 10,000-part limit at UploadPart time, or only at CompleteMultipartUpload (rgw_multipart_part_upload_limit)? If UploadPart accepts part numbers above 10,000, a client can build an upload that can never complete. Rejecting it early would avoid this whole situation. 6. Is there a better way to purge tens of TB of abandoned multipart uploads? Thanks in advance, Manuel Rios
I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated.
Turns out this one's almost definitely just me being an idiot. I completely missed step 5 (well, didn't actually get to it because I got overwhelmed by all the OSD upgrades). I just now issued ceph mon set auth_service_cipher aes256k which made the AUTH_INSECURE_SERVICE_TICKETS error go away, and I'm guessing that _now_ is when the couple hours starts and I'll be seeing the rotating service ticket errors vanish in a while. So yeah, Q1 is probably solved. Thanks for prodding me to go back and read the directions again :-)
Oh yeah, reading the docs can be quite important sometimes. ;-) Zitat von Jay <ceph-users@ceph.io>:
I can't go into the other details right now, but a customer reported the same thing: after the upgrade they still had rotating services. The only difference during the upgrade procedure compared to all others I have done so far was that they had failed the MGR multiple times during the upgrade. I recommended to run the upgrade again (with the same image, obviously) so the services would be properly rotated.
Turns out this one's almost definitely just me being an idiot. I completely missed step 5 (well, didn't actually get to it because I got overwhelmed by all the OSD upgrades). I just now issued
ceph mon set auth_service_cipher aes256k
which made the AUTH_INSECURE_SERVICE_TICKETS error go away, and I'm guessing that _now_ is when the couple hours starts and I'll be seeing the rotating service ticket errors vanish in a while.
So yeah, Q1 is probably solved. Thanks for prodding me to go back and read the directions again :-) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Jay
-
Manuel Rios - EDH