On Thu, 19 Jun 2025, at 00:04, Eugen Block wrote:
My main question now is "which is the 'latest' MON?"
Timestamp of the files within the mon db store. ;-) no need to dig through the db itself.
That's what I thought too. Sadly all the DBs get touched when the MONs start, so they're all dated "yesterday". Despite me saying "It's SQLite" - now I see this is all levelDB (don't know where I got confused). I wrote some janky Python code using plyvel to dump-out the keys and kv pairs as b-strings. Seems mon1 and mon2 have an identical key set. There are 2 keys that differ between mon1/2 and mon3 as well as 3 keys that differ on all 3 hosts: mon1/2: b'monitor\x00election_epoch' => b'\x16\x0c\x00\x00\x00\x00\x00\x00' # 3094 if it's a 64-bit little-endian integer mon3: b'monitor\x00election_epoch' => b'\x12\x0c\x00\x00\x00\x00\x00\x00' # 3090 mon1/2: b'monmap\x00last_committed' => b'\x04\x00\x00\x00\x00\x00\x00\x00' # 4 mon3: b'monmap\x00last_committed' => b'\x06\x00\x00\x00\x00\x00\x00\x00' # 6 mon1; b'monitor\x00election_writeable_test' => b'\xee\xeaI\\\x00\x00\x00\x00' mon2: b'monitor\x00election_writeable_test' => b'\xb2\x8a\xd6\x06\x00\x00\x00\x00' mon3: b'monitor\x00election_writeable_test' => b'i\x98<d\x00\x00\x00\x00' mon1: b'paxos\x00accepted_pn' => b'tS\x01\x00\x00\x00\x00\x00' # 86900 mon2: b'paxos\x00accepted_pn' => b'1V\x01\x00\x00\x00\x00\x00' # 87601 mon3: b'paxos\x00accepted_pn' => b'\xc0>\x01\x00\x00\x00\x00\x00' # 81600 mon1: b'paxos\x00last_pn' => b'1V\x01\x00\x00\x00\x00\x00' # 87601 mon2: b'paxos\x00last_pn' => b'\xf9=\x01\x00\x00\x00\x00\x00' # 81401 mon3: b'paxos\x00last_pn' => b'\xccU\x01\x00\x00\x00\x00\x00' # 87500 (Note all those integer decodes were done by hand and are subject to errors) Also there is one key in mon1/2 that's not in mon3 and 3 keys in mon3 that aren't in mon1/2: mon1/2: b'mon_sync\x00temp_newer_monmap' mon3: b'monmap\x005' mon3: b'monmap\x006' mon3: b'monmap\x00latest' So: Given that mon1/2 mention "mon_sync.tmp_newer_monmap" and mon3 has "monmap.latest", I feel like mon3 is the most complete and up-to-date. But really, this is all just "digital phrenology".
If you don't feel confident to manage the cluster with DeepSea (I know of people who were literally afraid of DeepSea stages :-D ), then don't. :-) Without cephadm you can try to deploy daemons relatively easy. Two years ago I wrote an article [1] how to migrate from SES 6 (Nautilus) to Upstream (Pacific) after testing this procedure for a potential customer. You can reach out to me if you should get to that point.
That looks totally like what I'm planning. Thanks!
I'm still in favor of the single mon approach, it has worked many times for us (actually it always worked), and it's relatively quick and easy to test. If that shouldn't work, there's still the procedure to collect the maps from the OSDs to create a new mon store. But let's see how far you get before exploring this option.
Yes, thanks again for all your great help. M0les.