Hi to all! I'm writing this (long) email to seek the wisdom of the community about a problem we are facing in this moment. As I wrote before we would like to upgrade one of our Ceph cluster from Octopus to Quincy (and then to Squid) and, at the same time, upgrade the O.S. from centos7/rocky8 to rocky9. Our plan is (was?) to start from the MON/MGR nodes, bring those nodes to Quincy and rocky9 and then re-install all the OSD servers (one-by-one). Our initial setup for the MON/MGR nodes was: - node1: centos7+ceph15.2.16 - node2: centos7+ceph15.2.16 - node3: centos7+ceph15.2.16 - node4: rocky8+ceph15.2.17 - node5: rocky8+ceph15.2.17 To begin with we removed node1 and node2 from the cluster ("ceph mon remove mon.node1") and reinstall it with rocky9+ceph17.2.9. As next steps we taught to: 1. upgrade Ceph in-place on node4 and node5 to ceph17.2.6 2. add the freshly installed node1 and node2 to the cluster (with ceph17.2.9) 3. remove, reinstall and re-add node3 to the cluster. 4. remove, reinstall and re-add node4 and node5 to the cluster (to have all rocky9 nodes). The first point went smoothly, but unfortunately the second did not (and we are stopped there). We exported the monmap from the running cluster and created on the freshly (re)installed hosts (node1, node2) the mon data directory. Then we started the ceph-mon daemon on host node1. Once the daemon started it got stuck in probing/electing, the commands to the cluster (like "ceph -s") took a long time to return. If the ceph-mon on node1 was stopped the command returned to work as usual and the cluster reported (as it should) a warning about a monitor down (the node1 we just stopped). Removing "mon.node1" from the cluster returned the health to OK. Clearly everything screams about "network" and/or "time-synchronization". But the networks seems fine, no firewall is in place, ping between nodes is fine (checked also jumbo-frames) and the services are able to communicate with each other. The same about time, the clocks are all synchronized and no CLOCK_SKEW is detected. The current situation from the monitor standpoint is the following: in the cluster: - node3: centos7+ceph15.2.16 - node4: rocky8+ceph17.2.6 - node5: rocky8+ceph17.2.6 out-of-the-cluster: - node1: rocky9+ceph17.2.9 - node2: rocky9+ceph17.2.9 We are unable to decide what is our best course of action so we are searching the wisdom from the community :-) We taught, among other things, to: - Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is the only daemon which wasn't restarted and when queried ("ceph daemon mon.node1 mon_status") has in "extra_probe_peers" the IP addresses of the original removed nodes, which are the same IPs as the nodes we want to add. We aren't sure if this prevent the ceph-mon to join, because we don't understand when/what "extra_probe_peers" is doing. - Remove the node1 from the cluster, set min_mon_release to Quincy and then try to add the new monitors. We are unsure about both approaches because if they don't work we will have a "unstable" cluster with only 2 monitors and no much space to operate. To "complicate" the matter we notice that on node4 and node5 the ceph-mon is not logging anymore to "/var/log/ceph/ceph*.log" even tough a "mon_cluster_log_to_file: true". If I search for open FD there is nothing in /var/log/ceph for the running ceph-mon... Have you some advice or you encounter similar problems in the past? I didn't found anything, but probably I searched with the wrong keywords. I can provide you with more information if needed. Thanks a lot! Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu