I have tested a staggered upgrade from 19.2.4 to 19.2.6. Indeed the OSD keys got rotated multiple times, I guess it may always happen when upgrading, even if it's a single daemon type that's later in the sequence. It's not really a problem but it does make everything take quite a bit longer (we have 1000+ OSD's on production clusters and must do a staggered upgrade because we need to manage the MDS failover). I think the devs are aware though. I do not observe this port multiplication however. mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (7h) 42s ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 running (14m) 42s ago 5w 475M - 19.2.6 ff836bb28e7d 45c38fb90d31 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running (7h) 42s ago 5w 675M - 19.2.6 ff836bb28e7d 1d69113ae7ed However I have been observing some strangeness with daemons showing non-standard status: mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (6h) 3m ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 unknown 8m ago 5w 339M - 19.2.6 ff836bb28e7d 98add61a7280 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running 0s ago 5w - - <unknown> <unknown> <unknown> A node failed to finish starting its containers without a clear reason why, it also refused to exit maintenance, and a mon on another node failed and fixed itself without intervention. I have not really determined a clear cause. Most of that fixed itself eventually although I now notice a single crash daemon playing up. Of course we're making it hard because we do a staggered upgrade and we use multiple routed subnets too :) Regards, JM On 8/26/26 12:21, Eugen Block wrote:
I haven't seen anything like this yet:
mgr.reef1.zfjbzv reef1 *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f mgr.reef2.omwexa reef2 *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago 14h 167M - 20.2.4 6facee348180 b24464d5c98b
I just did a staggered upgrade in a test cluster from 18.2.7 to 20.2.4 and I noticed that it goes through redeploying all daemons with a new keyring everytime although they already had been upgraded, for example MGR, MON and OSD when I was trying to upgrade MDS only (--daemon-type MDS).
A 'ceph mgr fail' does not clean up those redundant port entries. Should I create a tracker for that or is this already known?
I haven't tried the same on 19.2.6 yet.
Regards, Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jean-Marie de Boer Specialist SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl E: jean-marie.deboer@surf.nl | M: +31610961787