Hi, Yesterday we got an update via Proxmox. The initial update went well but hell broke loose when I tried to dissallow the old aes authentification in order top get rid of a warning. I'm looking for guidance before taking any further destructive action on a 3-node Proxmox cluster (6 OSDs, one replicated pool 3/2, ~2.5 TiB). No users on it, no time pressure. Timeline -------- 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: auth_allowed_ciphers aes256k auth_preferred_cipher aes New keys are created as aes but aes is not allowed to authenticate. Any OSD restarted after this point was locked out. 10.09 15:20 last good osdmap in the original mon store: epoch 573036, all 6 OSDs up/in. 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, 72 MB, 26k keys, intact). 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has 56 auth keys vs 128 in the original, 28 mon_config_key vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- clearly a reconstruction. 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was still working at this point (wait_auth_rotating done, and osd.2 had to re-add itself via "osd crush create-or-move"). 10.09 21:24 auth failures start on top of the crashes (handle_auth _bad_method). Resolved since: mon key was rotated without updating the bluestore osd_key label. Current state ------------- 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx sessions. osd.1, osd.2, osd.5 crash during peering: ./src/osd/PGLog.cc: 401: FAILED ceph_assert( log.head >= olog.tail && olog.head >= log.tail) PeeringState::Stray::react(MLogRec const&) -> merge_log The merge_log lines show an *empty* authoritative log being sent against a populated local one: merge_log log((573038'28581091,573038'28581091], crt=573038'28581091) from osd.3 into log((573040'28581121,573042'28581681], crt=573042'28581681) So the surviving OSDs are behind, and the crashing ones hold log entries from epochs 573038-573042 that the primary has no record of. That range starts two epochs after the last osdmap in the original store. original mon store max osdmap 573036 divergent pg log entries 573038-573042 current osdmap (after restarts) 573538 ceph -s (before I stopped the crashing OSDs): 278/708728 objects unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. Questions --------- 1. Is restoring the original mon store the right approach here? It is only 2 epochs behind where the divergence starts, which seems like the best possible starting point. 2. How should the 502-epoch gap be handled -- the OSDs have osdmaps far newer than the restored monitor would have? 3. Is the assert itself worth a tracker issue? An empty authoritative log against a populated local one looks like a case that should fall back to backfill rather than assert. What I have NOT done: no mark_unfound_lost, no min_size 1, no ceph-objectstore-tool remove. All three mon stores are backed up. The original store is untouched. One more constraint: the bluestore osd_key labels on the three running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. They run fine now but would not come back after a restart, and set-label-key refuses to write while the OSD is up. So any plan involving an OSD restart has to fix the label in the same stop/ start window. Happy to provide full logs with debug_osd=20, ceph pg query output, or monstore dumps. Thanks, --martin -- martin.konold@konsec.com Phone: +49 711 21 72 74-90 KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany