I think I found at least a clue, apparently it's depending on other services like dashboard. During those upgrade tests I bootstrapped the cluster with the flag "--skip-dashboard" since I didn't need that. I could verify this behavior for different Ceph versions as well as underlying operating systems (openSUSE Leap 16.0 and Ubuntu 24.04). If I bootstrap a cluster with dashboard, the upgrade process won't mess up the orch ps output. I think this is a bug, I'll check the tracker if there's something like this. Zitat von Eugen Block <eblock@nde.ag>:
It got worse:
reef1:~ # ceph orch ps --daemon-type mgr NAME HOST PORTS
STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.reef1.eqokui reef1 *:9283,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 198M - 20.2.4 6facee348180 472d9e5cd38a mgr.reef3.wiuxme reef3 *:8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765,8765 running (3h) 7m ago 23h 144M - 20.2.4 6facee348180 820c2422172f
I started from scratch (purged the previous cluster) and upgraded from:
18.2.2 -> 18.2.5 -> 18.2.6 -> 18.2.7 -> 18.2.8 -> 19.2.0 -> 19.2.1 -> 19.2.2 -> 19.2.3 -> 19.2.4 -> 19.2.5 -> 20.2.0 -> 20.2.1 -> 20.2.3 -> 20.2.4
So these are 14 Upgrades but I'm counting 39 and 27 entries for port 8765, so that doesn't seem to correlate. I have no idea yet where that comes from :-D Maybe there were some leftovers on those VMs that somehow confused something. I have other test clusters and our production cluster as well on either 20.2.4 or 19.2.6 and can't see this effect anywhere else (yet).
Zitat von Jean-Marie de Boer <ceph-users@ceph.io>:
I have tested a staggered upgrade from 19.2.4 to 19.2.6.
Indeed the OSD keys got rotated multiple times, I guess it may always happen when upgrading, even if it's a single daemon type that's later in the sequence. It's not really a problem but it does make everything take quite a bit longer (we have 1000+ OSD's on production clusters and must do a staggered upgrade because we need to manage the MDS failover). I think the devs are aware though.
I do not observe this port multiplication however.
mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (7h) 42s ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 running (14m) 42s ago 5w 475M - 19.2.6 ff836bb28e7d 45c38fb90d31 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running (7h) 42s ago 5w 675M - 19.2.6 ff836bb28e7d 1d69113ae7ed
However I have been observing some strangeness with daemons showing non-standard status:
mgr.ceph-tst-a-01.xqeivr ceph-tst-a-01 *:8443,9283,8765 running (6h) 3m ago 5w 478M - 19.2.6 ff836bb28e7d 251c5d067daf mgr.ceph-tst-b-01.cbupet ceph-tst-b-01 *:8443,9283,8765 unknown 8m ago 5w 339M - 19.2.6 ff836bb28e7d 98add61a7280 mgr.ceph-tst-c-01.xhofkh ceph-tst-c-01 *:8443,9283,8765 running 0s ago 5w - - <unknown> <unknown> <unknown> A node failed to finish starting its containers without a clear reason why, it also refused to exit maintenance, and a mon on another node failed and fixed itself without intervention. I have not really determined a clear cause.
Most of that fixed itself eventually although I now notice a single crash daemon playing up.
Of course we're making it hard because we do a staggered upgrade and we use multiple routed subnets too :)
Regards, JM
On 8/26/26 12:21, Eugen Block wrote:
I haven't seen anything like this yet:
mgr.reef1.zfjbzv reef1 *:9283,8765,8765,8765,8765,8765,8765 running (2m) 109s ago 18h 129M - 20.2.4 6facee348180 ed9ae39b8f5f mgr.reef2.omwexa reef2 *:8765,8765,8765,8765,8765,8765,8765 running (2m) 108s ago 14h 167M - 20.2.4 6facee348180 b24464d5c98b
I just did a staggered upgrade in a test cluster from 18.2.7 to 20.2.4 and I noticed that it goes through redeploying all daemons with a new keyring everytime although they already had been upgraded, for example MGR, MON and OSD when I was trying to upgrade MDS only (--daemon-type MDS).
A 'ceph mgr fail' does not clean up those redundant port entries. Should I create a tracker for that or is this already known?
I haven't tried the same on 19.2.6 yet.
Regards, Eugen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Jean-Marie de Boer Specialist
SURF | Science Park 140 | 1098 XG Amsterdam | www.surf.nl E: jean-marie.deboer@surf.nl | M: +31610961787 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io