Hi, Were you able to find anything more about this? I'm looking at upgrading a bunch of clusters from previous major versions, but do not want to upgrade all RGW services. And also limit the number of osds at the time that are being upgraded, to keep the recovery traffic lower for example. Br, On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote:
Adding the dev list
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I did a quick search on tracker but couldn't find anything related. A customer reported this, and I can confirm the behaviour on a lab cluster. I usually perform a staggered upgrade with --daemon-types and --limit, but not for all daemon types. So I haven't stumbled across this yet myself, but our customer did (I can reproduce with --daemon-types as well). They upgraded from latest Reef to latest Squid and reported that despite providing the --limit parameter, the mons were all upgraded. So I tried to reproduce, and the behaviour is not really clear to me, I'll try to clarify.
# Start with MGRs
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1
Upgrading MGRs with limit works, but it doesn't reflect in the MGR log. Usually, I expect a line like this:
...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade
But there is no such line in the logs. Then I upgrade the rest of the MGRs.
# Continue with MONs
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mon --limit 1
This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. And again, no such line in the log (Hit upgrade limit). What I noticed was a MGR respawn after the first MON had been upgraded successfully. Maybe some state of the upgrade progress gets lost during the respawn? I then upgraded the remaining MON. Then ceph-crash is upgraded successfully.
# Upgrade OSD
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1
And this is the first service that actually reports the upgrade limit:
2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Complete! 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Complete!
This is really irritating and inconsistent: If the orchestrator does honor --limit with other services than OSD, why isn't that visible in the logs? And what's with the MONs? Why 2 out of 3?
Any pointers appreciated! Not sure which Ceph versions might be affected by this, I'll try out a couple more upgrade paths.
Thanks, Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io