Hi, we're facing an issue during upgrades (and sometimes server reboots), it appears to occur when (at leat) one of the MONs has to do a full sync. And I'm wondering if the upgrade procedure could be improved in that regard, I'll come back to that later. First, I'll try to summarize the events. We upgraded to latest Pacific (16.2.15) last week. Note that the filesystems of the MONs are on HDDs (we're planning to move to flash, we already noticed issues during disaster recovery on HDDs last year). According to the logs the MGRs and MONs were upgraded successfully and a quorum was formed successfully at: 2024-04-30T12:57:22.347182+0000 mon.ndeceph03 (mon.0) 1208046 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum ndeceph03,ndeceph01) Then ceph started the first OSD upgrades a few minutes later, the first one at: 2024-04-30T13:00:42.733528+0000 mon.ndeceph03 (mon.0) 101 : cluster [INF] osd.25 marked itself down and dead One OSD by one on ndeceph01 was upgraded: 2024-04-30T13:01:46.517+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.19 (5/7) 2024-04-30T13:01:46.517+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.19 (5/7) 2024-04-30T13:02:24.800+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.22 (6/7) 2024-04-30T13:02:24.800+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.22 (6/7) 2024-04-30T13:02:48.220+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.29 (7/7) 2024-04-30T13:02:48.220+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.29 (7/7) But the mon service on ndeceph02 (also OSD server) was still syncing (for around 6 minutes): 2024-04-30T13:02:33.124+0000 7f1c24444700 1 mon.ndeceph02@2(synchronizing) e58 handle_auth_request failed to assign global_id 2024-04-30T13:08:30.123+0000 7f1c24444700 1 mon.ndeceph02@2(synchronizing) e58 handle_auth_request failed to assign global_id All HDD OSDs from ndeceph02 (not upgraded yet) were complaining about timeouts, and apparently tried to reboot multiple times (wondering why the SSDs didn't complain though): 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f2e1399b700' had timed out after 15.000000954s 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 osd.0 367411 is_healthy false -- internal heartbeat failed 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 osd.0 367411 not healthy; waiting to boot 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f2e1399b700' had timed out after 15.000000954s 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 osd.0 367411 is_healthy false -- internal heartbeat failed 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 osd.0 367411 not healthy; waiting to boot ... 2024-04-30T13:03:59.322+0000 7f2e201b4700 1 osd.0 367414 state: booting -> active 2024-04-30T13:05:22.184+0000 7f2e201b4700 1 osd.0 367433 state: booting -> active 2024-04-30T13:06:39.602+0000 7f2e201b4700 1 osd.0 367447 state: booting -> active 2024-04-30T13:08:02.612+0000 7f2e201b4700 1 osd.0 367454 state: booting -> active During this syncing period the disk utilization of the OS filesystem was at 100%, but ceph kept upgrading other OSD daemons. There are 3 main hosts, failure domain is host, replicated pools with min_size 2, size 3. If the OSDs from one host are struggling (because of disk IO on the filesystem, apparently) and ceph keeps upgrading others, we get inactive PGs. So what I'm wondering about is, how does the orchestrator decide if it's ok to stop one OSD ('ceph osd ok-to-stop <ID>' is the manual command) while others obviously are not healthy and would cause inactive PGs? The cluster did notice slow requests and reported messages like these: 2024-04-30T13:02:58.867499+0000 osd.28 (osd.28) 10529 : cluster [WRN] Monitor daemon marked osd.28 down, but it is still running If some OSDs clearly aren't healthy, I would expect the orchestrator to pause the upgrade. After the mon full sync completed, it started upgrading the OSDs on ndeceph02 as well, but the inactive PGs were only resolved after several more minutes when almost all OSDs from that tree branch had been upgraded: 2024-04-30T13:16:17.536359+0000 mon.ndeceph03 (mon.0) 1461 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 11 pgs peering) During a planned upgrade this issue can be mitigated via staggered upgrades (now that I know what the cause is), upgrading MGRs and MONs first and wait until everything has settled. Then continue with OSDs. But after a reboot there's no way to control that, of course. I helped a customer last year with a mon sync issue so we might be able to improve things a bit until we have flash disks. I don't know how it all behaves in newer ceph versions regarding the upgrade, but I feel like it wouldn't make a difference right now. Is there any way to improve the orchestrator to consider unhealthy OSDs before stopping healthy ones? Thanks! Eugen