Okay, this turned out to be down to cephadm rejecting the request because the new MON was not in the list of public networks. I had seen that error in the logs, but it looked as though it was a consequence of the store.db error, rather than a cause. After adding the new network, and repeating the request, the new MONs were created. On Tue, 11 Jul 2023 at 08:57, Adam Huffman <adam.huffman.lists@gmail.com> wrote:

Forgot to say we're on Pacific 16.2.13.

On Tue, 11 Jul 2023 at 08:55, Adam Huffman <adam.huffman.lists@gmail.com> wrote:

...

Hello

I'm trying to add MONs in advance of a planned downtime.

This has actually ended up removing an existing MON, which isn't helpful.

The error I'm seeing is:

Invalid argument: /var/lib/ceph/mon/ceph-<hostname>/store.db: does not exist (create_if_missing is false) error opening mon data directory at '/var/lib/ceph/mon/ceph-<hostname>': (22) Invalid argument

It appears that the fsid is being stripped, because the directory was there. It's now in /var/lib/ceph/<fsid>/removed

This appears to be similar to: https://tracker.ceph.com/issues/45167 which was closed for lack of a reproducer.

The command I ran was:

sudo ceph orch apply mon --placement="comma-separated hostname list"

after running that with "--dry-run".

Would be grateful for some advice here - I wasn't expecting to reduce the MON count.

Best Wishes, Adam