On 14/09/2026 10:53, Alex Walender wrote:
Hey there!
I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons.
The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster.
The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again.
I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count).
Hello, We have a somewhat similar issue on a, busy, CephFS cluster with 1500 OSDs (5 MON+MGR, 7 MDS -- 5 of which are also MON+MGR, 52 OSD nodes, largest pool with 8K PGs). This is an upgrade from 17.2.8 to 19.2.6. We did a staggered upgrade to be able to upgrade the 7 MDS during our next scheduled downtime (with "mgr/orchestrator/fail_fs"), since we have 6 active MDS. The MGR, MON & "crash" upgrades went smoothly, but there were issues during the OSD upgrades (after 24+ hours, about 60% of the OSD updated), and the MONs started to stall: 2266 slow ops, oldest one blocked for 153 sec, mon.$MON0 has slow ops and sometime the MONs did not answer at all: 2026-09-16T12:17:26.485+0000 7f7972084640 -1 monclient: get_monmap_and_config failed to get config Pausing the upgrade seemed to worsen the MONs issues ("159104 slow ops, oldest one blocked for 951 sec, mon.$MON0 has slow ops"), so we resumed it to its completion (41 hours for MGR+MON+crash+1500 OSDs). Sometimes the MON issue goes away on its own, or we have to fail/restart MON(s) to get back to a "normal" situation. This continues even now, after all OSDs have been upgraded & only the MDS (+one RGW) remain in the older version until our scheduled downtime: "overall": { "ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable)": 8, "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 1540 } The frequency of the issue is every ~3 hours for this cluster. We see thousands of: "Sep 17 10:25:15 $MON1 ceph-mon[2618906]: ReplicaActive::clear_remote_reservation(): not reserved!" in the MON logs. We had no issue upgrading two smaller clusters (non-staggered upgrades), one with 150 OSDs and another with 800 OSDs (both clusters were already running Squid). Loïc. PS: We find that there are multiple issues with the upgrade process, like: - OSDs in an alternate CRUSH tree were forcibly moved to the "default" branch after their upgrade (on nodes with OSDs in 2 independent CRUSH trees: "default" and "fast" -- no issue on nodes with all OSDs in an alternate CRUSH tree) - the update documentation is lacking regarding the update of the credentials created by "cephadm" itself (bootstrap, admin, deployed services, ...) and, were it not for Eugen B., we probably wouldn't have found "mon_auth_emergency_allowed_ciphers" - progress messages (displayed by "ceph -W cephadm") have a problem with instance counters (all instances of whatever is being updated are displayed as the first instance, there "1/59"), which is a confusing: 2026-09-15T09:11:18.662822+0000 mgr.$MGR1.osnxsd [INF] Upgrade: Updating crash.$NODE7 (1/59) [...] 2026-09-15T09:11:41.166019+0000 mgr.$MGR1.osnxsd [INF] Upgrade: Updating crash.$NODE8 (1/59) (in "src/pybind/mgr/cephadm/upgrade.py", "_upgrade_daemons()": "num" is set to "1" but never incremented in the loop) -- | Loic Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |