Hi! A little update. We were able to make the ceph-mgr back working, but we are still don't know what was wrong. To made it work again we restarted (one at the time) the MONs, after restarting the 2 MON (out of 3) the MGR started working again. We are still investigation why we are unable to add new MONs to the cluster (they are stucked in probing/electing phase), but it is a topic for another thread. Cheers Iztok On 08/04/26 18:14, Iztok Gregori via ceph-users wrote:
Hi to all!
First a little of background. We are in the process of a massive upgrade from Octopus to Quincy (during which there is also the upgrade (reinstall) from CentOS 7/Rocky 8 to Rocky 9).
During the first phase we removed from the cluster 2 monitor/managers and reinstalled them with Rocky 9 and Ceph 17.2.9). The 3 remaining monitors/manager were: - 2 x Rocky 8 with 15.2.17 - 1 x CentOS 7 with 15.2.16
Then we upgraded Ceph from 15.2.17 to 17.9.6 on the 2 nodes with Rocky 8 and restarted the services. So to summarize we have: - 2 x MON Rocky 8 with 17.2.6 - 1 x MON CentOS 7 with 15.2.16
and - 1 x MGR CentOS 7 with 15.2.16
The next step was add the new (reinstalled) nodes as MON in the cluster we encounter the first problem. Once started the new ceph-mon daemon entered the cluster but all the commands ("ceph -s" for example) remained stuck. If we stopped the new ceph-mon daemon the commands restart to work (the status was in warning because the new ceph-mon was down). To regain the quorum we deleted the new ceph-mon.
And here I think we made a mistake, we started the ceph-mgr daemons on the 2x Rocky 8 nodes. So we had: - 2 x MGR Rocky 8 with 17.2.6 (standby) - 1 x MGR CentOS 7 with 15.2.16 (active)
We wrongly assume that the MGR was the culprit and we fail the active MGR. From then on the status of the active MGR is in starting and all the PGS are unknown:
cluster: id: 5bed1bf6-123f-4597-b8b1-d77931228548 health: HEALTH_WARN Reduced data availability: 13825 pgs inactive
services: mon: 3 daemons, quorum n-3-c7,n-2-r8,n-1-r8 (age 91m) mgr: n-1-r8 (active, starting, since 13m), standbys: n-2-r8, n-3-c7 osd: 852 osds: 849 up (since 8d), 849 in (since 7w)
data: pools: 11 pools, 13825 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 13825 unknown
From the logs we see
2026-04-08T16:59:26.244+0200 7fe0596a1700 0 ms_deliver_dispatch: unhandled message 0x5583db0d0c60 mon_map magic: 0 v1 from mon.2 v2:172.19.240.35:3300/0 2026-04-08T16:59:26.807+0200 7fe0596a1700 1 mgr handle_mgr_map Activating! 2026-04-08T16:59:26.808+0200 7fe0596a1700 1 mgr handle_mgr_map I am now activating 2026-04-08T16:59:34.923+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:36.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:38.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:40.924+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:42.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:44.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:46.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:48.926+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:50.925+0200 7fdfd7e70700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs 2026-04-08T16:59:52.926+0200 7fdfd7e70700 1 mgr.server send_report Giving up on OSDs that haven't reported yet, sending potentially incomplete PG state to mon
The mgr dump looks like this:
{ "epoch": 373, "active_gid": 16955846717, "active_name": "n-1-r8", "active_addrs": { "addrvec": [] }, "active_addr": "(unrecognized address family 0)/0", "active_change": "2026-04-08T16:59:26.282185+0200", "active_mgr_features": 4540138320759226367, "available": false, "standbys": [ { "gid": 16955784229, "name": "n-2-r8", "mgr_features": 4540138320759226367, "available_modules": [
We think that "unrecognized address family" could be the problem, because we have a cluster/public network which is on a different interface and could be that is not recognized/binded on mgr startup. But we might be wrong.
Somebody can give us some help?
Thanks in advance Iztok Gregori