Please don't drop the list off your responses. I'm not entirely sure yet what to make of the epochs. I'm thinking about mon_min_osdmap_epochs (default 500) and similar parameters. But I'm heading out on vacation now, hopefully someone else will chime in. Good luck! Eugen Zitat von Martin Konold <martin.konold@konsec.com>:

Hi Eugen,

thanks for the quick reply -- one clarification: the monitor was already rebuilt with --mkfs yesterday evening, before I posted today. ceph-pve-07.bak is the leftover original store, not something I am proposing to move. So the question is whether restoring it is the right way back, not whether to abandon the current one.

...

Did you verify that all service keyrings were successfully rotated after the upgrade?

No, and that turned out to be the core of the auth problem. The rotation touched the mon DB but not the bluestore osd_key labels:

osd.2 mon DB AgBf7qJq... label/tmpfs AgA8cKJq... osd.5 mon DB AgBg7qJq... label/tmpfs AgA8cKJq...

osd.0, osd.3 and osd.9 are still in that state -- they run fine on existing sessions but hold pre-rotation keys in their labels, so they would not come back after a restart. set-label-key refuses to write while the OSD is up, so I cannot fix that pre-emptively.

Caps were inconsistent too: osd.0 and osd.3 have "caps mon = allow rwx" (the pre-profile form), osd.1 and osd.9 were missing "caps mgr" entirely.

...

You could have allowed the insecure cipher again

I did, that was the first thing after finding the monmap policy:

ceph mon set auth_allowed_ciphers aes,aes256k

That fixed authentication -- osd.2 and osd.5 now get through and reach peering. client.admin was fine throughout, so the command went through without trouble.

What remains is the PGLog assert, and it is not an auth problem. Checking the timestamps: osd.5 first hit the assert at 21:04:21, while its monclient log at 21:04:15 shows a healthy session (wait_auth_rotating done, uptodate secrets). The auth failures only start at 21:24. So the crashes came first; the cipher lockout landed on top and obscured them.

The assert shows an empty authoritative log against a populated local one:

merge_log log((573038'28581091,573038'28581091], crt=573038'28581091) from osd.3 into log((573040'28581121,573042'28581681], crt=573042'28581681)

Also worth noting: at 21:04 osd.2 had to re-add itself to the CRUSH map ("osd crush create-or-move id 2 weight 7.2774 host=pve-08"), which suggests the rebuilt store did not know it. My working assumption is that the incomplete CRUSH map during that window let writes be acked against acting sets that later could not reconcile.

So the open question stands: with the original store sitting at osdmap 573036 and the divergence starting at 573038, is a restore the right approach, and how should the ~500 epoch gap to the OSDs be handled?

root@pve-07:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 43.66434 root default -61 14.55478 host pve-07 0 nvme 7.27739 osd.0 up 1.00000 1.00000 3 nvme 7.27739 osd.3 up 1.00000 1.00000 -66 14.55478 host pve-08 2 nvme 7.27739 osd.2 down 1.00000 1.00000 5 nvme 7.27739 osd.5 down 1.00000 1.00000 -71 14.55478 host pve-09 1 nvme 7.27739 osd.1 down 1.00000 1.00000 9 nvme 7.27739 osd.9 up 1.00000 1.00000

Kind Regards, --martin

Am Freitag, September 11, 2026 10:56 CEST, schrieb Eugen Block <ceph-users@ceph.io>:

...

Hi,

that seems a little drastic to move the mon store. Did you verify that all service keyrings (mgr,mon,osd,mds) were successfully rotated after the upgrade?

You could have allowed the insecure cipher again to see if that resolves the issue and finish the upgrade (if steps were missing).

ceph mon set auth_allowed_ciphers aes,aes256k

I assume you rotated the admin keyring as well so changing auth_allowed_ciphers would be possible?

Zitat von Martin Konold <ceph-users@ceph.io>:

...

Hi,

Yesterday we got an update via Proxmox. The initial update went well but hell broke loose when I tried to dissallow the old aes authentification in order top get rid of a warning.

I'm looking for guidance before taking any further destructive action on a 3-node Proxmox cluster (6 OSDs, one replicated pool 3/2, ~2.5 TiB). No users on it, no time pressure.

Timeline -------- 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: auth_allowed_ciphers aes256k auth_preferred_cipher aes New keys are created as aes but aes is not allowed to authenticate. Any OSD restarted after this point was locked out. 10.09 15:20 last good osdmap in the original mon store: epoch 573036, all 6 OSDs up/in. 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, 72 MB, 26k keys, intact). 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has 56 auth keys vs 128 in the original, 28 mon_config_key vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- clearly a reconstruction. 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was still working at this point (wait_auth_rotating done, and osd.2 had to re-add itself via "osd crush create-or-move"). 10.09 21:24 auth failures start on top of the crashes (handle_auth _bad_method). Resolved since: mon key was rotated without updating the bluestore osd_key label.

Current state ------------- 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx sessions. osd.1, osd.2, osd.5 crash during peering:

./src/osd/PGLog.cc: 401: FAILED ceph_assert( log.head >= olog.tail && olog.head >= log.tail) PeeringState::Stray::react(MLogRec const&) -> merge_log

The merge_log lines show an *empty* authoritative log being sent against a populated local one:

merge_log log((573038'28581091,573038'28581091], crt=573038'28581091) from osd.3 into log((573040'28581121,573042'28581681], crt=573042'28581681)

So the surviving OSDs are behind, and the crashing ones hold log entries from epochs 573038-573042 that the primary has no record of. That range starts two epochs after the last osdmap in the original store.

original mon store max osdmap 573036 divergent pg log entries 573038-573042 current osdmap (after restarts) 573538

ceph -s (before I stopped the crashing OSDs): 278/708728 objects unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down.

Questions --------- 1. Is restoring the original mon store the right approach here? It is only 2 epochs behind where the divergence starts, which seems like the best possible starting point. 2. How should the 502-epoch gap be handled -- the OSDs have osdmaps far newer than the restored monitor would have? 3. Is the assert itself worth a tracker issue? An empty authoritative log against a populated local one looks like a case that should fall back to backfill rather than assert.

What I have NOT done: no mark_unfound_lost, no min_size 1, no ceph-objectstore-tool remove. All three mon stores are backed up. The original store is untouched.

One more constraint: the bluestore osd_key labels on the three running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. They run fine now but would not come back after a restart, and set-label-key refuses to write while the OSD is up. So any plan involving an OSD restart has to fix the label in the same stop/ start window.

Happy to provide full logs with debug_osd=20, ceph pg query output, or monstore dumps.

Thanks, --martin

-- martin.konold@konsec.com Phone: +49 711 21 72 74-90 KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- martin.konold@konsec.com Phone: +49 711 21 72 74-90 KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany