Hi Martin, reply inlined. On Sat, Sep 12, 2026 at 12:24 AM Eugen Block <ceph-users@ceph.io> wrote:
Please don't drop the list off your responses.
I'm not entirely sure yet what to make of the epochs. I'm thinking about mon_min_osdmap_epochs (default 500) and similar parameters. But I'm heading out on vacation now, hopefully someone else will chime in.
Questions --------- 1. Is restoring the original mon store the right approach here? It is only 2 epochs behind where the divergence starts, which seems like the best possible starting point.
No. Paxos would make the old store win and reissue osdmaps from 573037 that the OSDs already hold, which is the same problem again.
2. How should the 502-epoch gap be handled -- the OSDs have
osdmaps far newer than the restored monitor would have?
Please leave the current monitors as they are. The gap is only on the monitor side.
3. Is the assert itself worth a tracker issue? An empty
Yes, it's a known bug tracked by https://tracker.ceph.com/issues/36337. But backfill isn't the answer. osd.3's log is the stale one, so backfilling from it would overwrite osd.2's newer writes, and the assert is what stopped that. Your logs would be useful on #36337 What I need from you: - How was the store rebuilt. The caps on osd.0/3 and the missing mgr caps on osd.1/9 look like 'ceph-objectstore-tool --op update-mon-db' plus 'ceph-monstore-tool rebuild'. If so, which OSD stores did you collect from, and did the other two monitors get the rebuilt store or keep their own? And please ntoe, 'ceph-monstore-tool rebuild' is only for monstore recovery from a disaster, and we need to rebuild the mon store with *all* osdmaps we can collect, and keep all monstore identical. - Map 573040 from three places, so we can diff them: ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 \ --no-mon-config --op get-osdmap --epoch 573040 --file osd2.573040 the same from osd.3, and ceph osd getmap 573040 -o mon.573040 # and diff them osdmaptool --print osd2.573040 > osd2.txt osdmaptool --print osd3.573040 > osd3.txt diff osd2.txt osd3.txt diff osd3.txt mon.txt # if all three are identical, the rebuild theory is wrong. we need to find # another theory to explain all these observations. if osd.2's osdmap # is different from the other two, the theory is confirmed. - 'ceph pg <pgid> query' for the down, incomplete and unfound PGs and the PGs in the osd.1/2/5 crash logs. - 'ceph-monstore-tool <store> show-versions' for the pve-08 and pve-09 backups. - Whether you used force-create-pg, mark-complete or osd_find_best_info_ignore_history_les at any point. I think, the plan is, once we have the query output and the theory is confirmed, we need to recover the affected PG manually, without using the builtin recovery machinery: for each affected PG, the copy on osd.1/2/5 is the newest. We remove the stale copies on osd.0/3/9 after exporting them, fix the labels in the same stop window, then start osd.1/2/5. Peering picks their log and backfills the rest. Per OSD: systemctl stop ceph-osd@3 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 \ --no-mon-config --pgid <pgid> --op export --file /space/osd.3.<pgid> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 \ --no-mon-config --pgid <pgid> --op remove ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-3/block \ --key osd_key --value "$(ceph auth get-key osd.3)" ceph-bluestore-tool prime-osd-dir --dev /var/lib/ceph/osd/ceph-3/block \ --path /var/lib/ceph/osd/ceph-3 systemctl start ceph-osd@3 Which PGs and which copies comes from the query output, so please wait for that. One note on your reply to Eugen: "osd crush create-or-move" runs at every OSD start, it doesn't mean the store lacked osd.2. Afterwards, finish the cipher migration with the helper in pve-manager 9.2.17+, not by hand. It fixes the labels and keyring copies for you and refuses to restrict ciphers while anything still needs aes, see /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_cephx_migratio...
authoritative log against a populated local one looks like a
case that should fall back to backfill rather t <https://www.google.com/maps/search/at+should+fall+back+to+backfill+rather+t?entry=gmail&source=g>han assert.
What I have NOT done: no mark_unfound_lost, no min_size 1, no ceph-objectstore-tool remove. All three mon stores are backed up. The original store is untouched.
One more constraint: the bluestore osd_key labels on the three running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. They run fine now but would not come back after a restart, and set-label-key refuses to write while the OSD is up. So any plan involving an OSD restart has to fix the <https://www.google.com/maps/search/involving+an+OSD+restart+has+to+fix+the+?entry=gmail&source=g>label in the same stop/ start window.
Happy to provide full logs with debug_osd=20, ceph pg query output, or monstore dumps.
Thanks, --martin
-- martin.konold@konsec.com Phone: +49 711 21 72 74-90 <+49%20711%2021727490> KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- martin.konold@konsec.com Phone: +49 711 21 72 74-90 <+49%20711%2021727490> KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Kefu Chai