Yeah, looks like it. I just compared the output with a fresh Tentacle install, I only enabled log-to-file during bootstrap, so log_to_file is true in my cluster as well. There's no leveldb_log_to_ceph_log option anymore since the default has been rocksdb for a while now. Zitat von Iztok Gregori via ceph-users <ceph-users@ceph.io>:

Hi!

On 15/04/26 11:11, Eugen Block via ceph-users wrote:

...

Hi,

Zitat von Iztok Gregori via ceph-users <ceph-users@ceph.io>:

[cut]

...

...

...

So your cluster currently has quorum with 2 out of 3 MONs on Quincy already, correct? Restarting the mon on node3 shouldn't be dangerous, I think. You'd still have quorum with node4 and node5, even if node3 wouldn't be able to join. If it doesn't join as Octopus, you could still upgrade it and retry with Quincy.

Yes. I was just reluctant to do so because in the case it doesn't join our room to manoeuvre will be reduced. Plus the fact that node3 is the only daemon which is still logging something...

oh I didn't realize there was only one left logging something. I thought only the newly to be joined MON didn't log. In that case I would also try to find the root cause for that.

It could easily be, because this is an old cluster which was upgraded multiple times, that "default" log settings changed in the meantime. If I query the daemon for 'mon_status' and search for log_to I see the following:

# ceph daemon mon.node4 config show | grep log_to "clog_to_graylog": "false", "clog_to_graylog_host": "127.0.0.1", "clog_to_graylog_port": "12201", "clog_to_monitors": "default=true", "clog_to_syslog": "false", "clog_to_syslog_facility": "default=daemon audit=local0", "clog_to_syslog_level": "info", "leveldb_log_to_ceph_log": "true", "log_to_file": "true", "log_to_graylog": "false", "log_to_journald": "false", "log_to_stderr": "false", "log_to_syslog": "false", "mon_cluster_log_to_file": "true", "mon_cluster_log_to_graylog": "false", "mon_cluster_log_to_graylog_host": "127.0.0.1", "mon_cluster_log_to_graylog_port": "12201", "mon_cluster_log_to_journald": "false", "mon_cluster_log_to_stderr": "false", "mon_cluster_log_to_syslog": "default=false", "mon_cluster_log_to_syslog_facility": "daemon", "mon_cluster_log_to_syslog_level": "info", "rocksdb_log_to_ceph_log": "true",

If I understand correctly I don't see the log in journald because '"log_to_journald": "false"' (in fact, on the Octopus node there is no "log_to_journald" option). Is this the correct default behaviour?

...

...

...

What I'm wondering about, why did you remove the MONs you wanted to upgrade from Ceph? Is that recommended somewhere? Maybe you need to run 'ceph mon add <name> <addr>' as well to let node1 or node2 join the monmap again? I'm not sure, it has been a long time doing this kind of stuff. ;-) But I don't think I ever removed a MON before upgrading it, not even when reinstalling the OS.

No, we didn't follow any recommendations, it seemed reasonable to remove a monitor before reinstalling the OS, a fresh start looked better than a backup/restore of '/var/lib/ceph/'. Maybe we were wrong. For the next upgrades (node3-5) we could think not to do that and see what it happens.

I don't exactly remember, tbh, but I expect that it would suffice to have the /var/lib/ceph/mon/ceph-nodeX directory with the correct permissions present in order for it to re-sync with the other MONs. But please double-check that, I'm really not sure. If I have the time, I will try to test that in a lab environment.

Thank you! I will check it before doing anything strange.

BTW, I remember that another reason for the MONs removal was that we needed to change the hostname (not the IP address) and we taught that a fresh install would be better.

[cut]

...

...

...

...

...

- Inspecting debug logs on the syncing MON could help finding the root cause.

With "syncing MON" you mean the MON which is just added to the cluster, right? Collecting this kind of logs is on the to-do list, but blocked because the "log problem" mentioned bellow.

Yes, that's what I meant. The missing logs are a problem, of course. It does sound familiar, but I don't have a good idea right now except for looking at directory and file permissions, apparmor/selinux things, dmesg, stuff like that.

Yep. My biggest fear right now is that lack of logging from Quincy monitor daemons could mean that they don't work properly and if we restart the Octopus one (or it doesn't want to rejoin the quorum) we will have a not functioning cluster. We need to investigate further into it.

I totally understand your hesitation, before I'd do that I would also try to rule out anything I can. But at the same time it would be surprising, given your cluster shows a healthy quorum right now. But yeah, investigate further before moving forward is the safer choice here.

Probably you are right. I'll be cautiuos.

Thank you for your help! Iztok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io