Re: Cephadm staggered upgrade --limit parameter not always applied
I'm doing further tests in a small cluster. Wondering a bit about the rotation of keys, since when I later upgrade the RGW service the mon,mgr and osds are restarted due to key rotation as well. I'm wondering if this is intentional? 3 nodes running mon/mgr/monitoring, 3 OSD nodes with additional RGW daemons. Starting at v17.2.7, upgrading all but the RGWs to v19.2.6 # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon_types mgr,mon,osd,crash Upgrade RGWs # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --services rgw.vps-lab1 On 8/26/26 5:37 PM, Eugen Block wrote:
I started to conduct some tests and document in which version the limits worked. I’m not done yet, I’ll update the respective tracker issue when I get there. To perform the staggered update you could just redeploy rgw daemons with a new image after all preceding upgrades have finished (mgr, mon, crash, osd, mds) with ceph orch daemon redeploy rgw.<some-rgw> <image>.
I don’t have the exact syntax at hand right now, but it should be quite similar.
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I haven't found anything explaining this yet, but I also haven't looked for a while. I just retried limiting RGW upgrade (from Reef to Tentacle):
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 --daemon-types rgw --limit 1
But the limit is still ignored:
2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1)
All three RGW daemons have been upgraded automatically. I'm pretty sure that it worked at some point with RGWs, in my notes from one of our customer upgrades where this worked:
ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 --daemon-types rgw --limit 1
I can't tell when it stopped working properly.
Regards, Eugen
Zitat von Andreas Eriksson <norrland@nullbyte.se>:
Hi,
Were you able to find anything more about this? I'm looking at upgrading a bunch of clusters from previous major versions, but do not want to upgrade all RGW services. And also limit the number of osds at the time that are being upgraded, to keep the recovery traffic lower for example.
Br,
On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote:
Adding the dev list
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I did a quick search on tracker but couldn't find anything related. A customer reported this, and I can confirm the behaviour on a lab cluster. I usually perform a staggered upgrade with --daemon-types and --limit, but not for all daemon types. So I haven't stumbled across this yet myself, but our customer did (I can reproduce with --daemon-types as well). They upgraded from latest Reef to latest Squid and reported that despite providing the --limit parameter, the mons were all upgraded. So I tried to reproduce, and the behaviour is not really clear to me, I'll try to clarify.
# Start with MGRs
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1
Upgrading MGRs with limit works, but it doesn't reflect in the MGR log. Usually, I expect a line like this:
...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade
But there is no such line in the logs. Then I upgrade the rest of the MGRs.
# Continue with MONs
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mon --limit 1
This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. And again, no such line in the log (Hit upgrade limit). What I noticed was a MGR respawn after the first MON had been upgraded successfully. Maybe some state of the upgrade progress gets lost during the respawn? I then upgraded the remaining MON. Then ceph-crash is upgraded successfully.
# Upgrade OSD
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1
And this is the first service that actually reports the upgrade limit:
2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Complete! 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Complete!
This is really irritating and inconsistent: If the orchestrator does honor --limit with other services than OSD, why isn't that visible in the logs? And what's with the MONs? Why 2 out of 3?
Any pointers appreciated! Not sure which Ceph versions might be affected by this, I'll try out a couple more upgrade paths.
Thanks, Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yeah, that's what I noticed as well. Every run of a staggered upgrade will rotate keys again and redeploy the respective daemons. I can't say if that's intentional, but I am sure the devs know about it. I guess this process will be further improved in future versions. I also conducted a lot of tests last week with staggered upgrades. I wanted to create an overview which versions respect the --limit parameter and if they report it in the mgr log and/or the 'ceph orch upgrade status' output. But I didn't have the time yet to fully document it. Zitat von Andreas Eriksson <norrland@nullbyte.se>:
I'm doing further tests in a small cluster. Wondering a bit about the rotation of keys, since when I later upgrade the RGW service the mon,mgr and osds are restarted due to key rotation as well.
I'm wondering if this is intentional?
3 nodes running mon/mgr/monitoring, 3 OSD nodes with additional RGW daemons.
Starting at v17.2.7, upgrading all but the RGWs to v19.2.6
# ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon_types mgr,mon,osd,crash
Upgrade RGWs
# ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --services rgw.vps-lab1
On 8/26/26 5:37 PM, Eugen Block wrote:
I started to conduct some tests and document in which version the limits worked. I’m not done yet, I’ll update the respective tracker issue when I get there. To perform the staggered update you could just redeploy rgw daemons with a new image after all preceding upgrades have finished (mgr, mon, crash, osd, mds) with ceph orch daemon redeploy rgw.<some-rgw> <image>.
I don’t have the exact syntax at hand right now, but it should be quite similar.
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I haven't found anything explaining this yet, but I also haven't looked for a while. I just retried limiting RGW upgrade (from Reef to Tentacle):
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 --daemon-types rgw --limit 1
But the limit is still ignored:
2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1)
All three RGW daemons have been upgraded automatically. I'm pretty sure that it worked at some point with RGWs, in my notes from one of our customer upgrades where this worked:
ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 --daemon-types rgw --limit 1
I can't tell when it stopped working properly.
Regards, Eugen
Zitat von Andreas Eriksson <norrland@nullbyte.se>:
Hi,
Were you able to find anything more about this? I'm looking at upgrading a bunch of clusters from previous major versions, but do not want to upgrade all RGW services. And also limit the number of osds at the time that are being upgraded, to keep the recovery traffic lower for example.
Br,
On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote:
Adding the dev list
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I did a quick search on tracker but couldn't find anything related. A customer reported this, and I can confirm the behaviour on a lab cluster. I usually perform a staggered upgrade with --daemon-types and --limit, but not for all daemon types. So I haven't stumbled across this yet myself, but our customer did (I can reproduce with --daemon-types as well). They upgraded from latest Reef to latest Squid and reported that despite providing the --limit parameter, the mons were all upgraded. So I tried to reproduce, and the behaviour is not really clear to me, I'll try to clarify.
# Start with MGRs
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1
Upgrading MGRs with limit works, but it doesn't reflect in the MGR log. Usually, I expect a line like this:
...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade
But there is no such line in the logs. Then I upgrade the rest of the MGRs.
# Continue with MONs
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mon --limit 1
This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. And again, no such line in the log (Hit upgrade limit). What I noticed was a MGR respawn after the first MON had been upgraded successfully. Maybe some state of the upgrade progress gets lost during the respawn? I then upgraded the remaining MON. Then ceph-crash is upgraded successfully.
# Upgrade OSD
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1
And this is the first service that actually reports the upgrade limit:
2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Complete! 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Complete!
This is really irritating and inconsistent: If the orchestrator does honor --limit with other services than OSD, why isn't that visible in the logs? And what's with the MONs? Why 2 out of 3?
Any pointers appreciated! Not sure which Ceph versions might be affected by this, I'll try out a couple more upgrade paths.
Thanks, Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I updated this tracker with my observations: https://tracker.ceph.com/issues/77093 Zitat von Eugen Block <eblock@nde.ag>:
Yeah, that's what I noticed as well. Every run of a staggered upgrade will rotate keys again and redeploy the respective daemons. I can't say if that's intentional, but I am sure the devs know about it. I guess this process will be further improved in future versions.
I also conducted a lot of tests last week with staggered upgrades. I wanted to create an overview which versions respect the --limit parameter and if they report it in the mgr log and/or the 'ceph orch upgrade status' output. But I didn't have the time yet to fully document it.
Zitat von Andreas Eriksson <norrland@nullbyte.se>:
I'm doing further tests in a small cluster. Wondering a bit about the rotation of keys, since when I later upgrade the RGW service the mon,mgr and osds are restarted due to key rotation as well.
I'm wondering if this is intentional?
3 nodes running mon/mgr/monitoring, 3 OSD nodes with additional RGW daemons.
Starting at v17.2.7, upgrading all but the RGWs to v19.2.6
# ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon_types mgr,mon,osd,crash
Upgrade RGWs
# ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --services rgw.vps-lab1
On 8/26/26 5:37 PM, Eugen Block wrote:
I started to conduct some tests and document in which version the limits worked. I’m not done yet, I’ll update the respective tracker issue when I get there. To perform the staggered update you could just redeploy rgw daemons with a new image after all preceding upgrades have finished (mgr, mon, crash, osd, mds) with ceph orch daemon redeploy rgw.<some-rgw> <image>.
I don’t have the exact syntax at hand right now, but it should be quite similar.
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I haven't found anything explaining this yet, but I also haven't looked for a while. I just retried limiting RGW upgrade (from Reef to Tentacle):
reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 --daemon-types rgw --limit 1
But the limit is still ignored:
2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1)
All three RGW daemons have been upgraded automatically. I'm pretty sure that it worked at some point with RGWs, in my notes from one of our customer upgrades where this worked:
ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 --daemon-types rgw --limit 1
I can't tell when it stopped working properly.
Regards, Eugen
Zitat von Andreas Eriksson <norrland@nullbyte.se>:
Hi,
Were you able to find anything more about this? I'm looking at upgrading a bunch of clusters from previous major versions, but do not want to upgrade all RGW services. And also limit the number of osds at the time that are being upgraded, to keep the recovery traffic lower for example.
Br,
On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote:
Adding the dev list
Zitat von Eugen Block <eblock@nde.ag>:
> Hi, > > I did a quick search on tracker but couldn't find anything > related. A customer reported this, and I can confirm the > behaviour on a lab cluster. I usually perform a staggered > upgrade with --daemon-types and --limit, but not for all > daemon types. So I haven't stumbled across this yet myself, > but our customer did (I can reproduce with --daemon-types as > well). They upgraded from latest Reef to latest Squid and > reported that despite providing the --limit parameter, the > mons were all upgraded. So I tried to reproduce, and the > behaviour is not really clear to me, I'll try to clarify. > > # Start with MGRs > > reef1:~ # ceph orch upgrade start --image > quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 > > Upgrading MGRs with limit works, but it doesn't reflect in the > MGR log. Usually, I expect a line like this: > > ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade > > But there is no such line in the logs. Then I upgrade the rest > of the MGRs. > > # Continue with MONs > > reef1:~ # ceph orch upgrade start --image > quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 > > This gets even weirder, the orchestrator upgrades 2 out of 3 > MONs. And again, no such line in the log (Hit upgrade limit). > What I noticed was a MGR respawn after the first MON had been > upgraded successfully. Maybe some state of the upgrade > progress gets lost during the respawn? I then upgraded the > remaining MON. Then ceph-crash is upgraded successfully. > > # Upgrade OSD > > reef1:~ # ceph orch upgrade start --image > quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone > --limit 1 > > And this is the first service that actually reports the upgrade limit: > > 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO > cephadm.upgrade] Upgrade: Updating osd.0 (1/1) > 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 > log_channel(cephadm) log [INF] : Upgrade: Updating osd.0 (1/1) > 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO > root] Hit upgrade limit of 1. Stopping upgrade > 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 > log_channel(cephadm) log [INF] : Hit upgrade limit of 1. > Stopping upgrade > 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO > cephadm.upgrade] Upgrade: Setting container_image for all nvmeof > 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 > log_channel(cephadm) log [INF] : Upgrade: Setting > container_image for all nvmeof > 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO > cephadm.upgrade] Upgrade: Finalizing container_image settings > 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 > log_channel(cephadm) log [INF] : Upgrade: Finalizing > container_image settings > 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO > cephadm.upgrade] Upgrade: Complete! > 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 > log_channel(cephadm) log [INF] : Upgrade: Complete! > > > This is really irritating and inconsistent: If the > orchestrator does honor --limit with other services than OSD, > why isn't that visible in the logs? And what's with the MONs? > Why 2 out of 3? > > Any pointers appreciated! Not sure which Ceph versions might > be affected by this, I'll try out a couple more upgrade paths. > > Thanks, > Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Andreas Eriksson
-
Eugen Block