I was in the middle of tuning my OSDs when lightning blew me off the Internet. Had to wait 5 days for my ISP to send a tech and replace a fried cable. In the mean time, among other things. I had some serious time drift between servers thanks to the OS upgrades replacing NTP with chrony and me not having thought to re-establish a master in-house timeserver. Ceph tried really hard to keep up with all that, but eventually it was just too much. Now I've got an offline filesystem and apparently it's stuck trying to get back online again. The forensics: [ceph: root@www7 /]# ceph -s cluster: id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f health: HEALTH_WARN failed to probe daemons or devices 1 filesystem is degraded 1/3 mons down, quorum www7,ceph03 services: mon: 3 daemons, quorum www7,ceph03 (age 2m), out of quorum: dell02 mgr: ceph08.tlocfi(active, since 58m), standbys: dell02.odtbqw, www7.rxagfn mds: 1/1 daemons up, 1 standby osd: 7 osds: 7 up (since 12h), 7 in (since 18h); 308 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 0/1 healthy, 1 recovering pools: 22 pools, 681 pgs objects: 125.10k objects, 36 GiB usage: 91 GiB used, 759 GiB / 850 GiB avail pgs: 47772/369076 objects misplaced (12.944%) 373 active+clean 308 active+clean+remapped io: client: 170 B/s rd, 0 op/s rd, 0 op/s wr [ceph: root@www7 /]# ceph health detail HEALTH_WARN 1 filesystem is degraded; 25 client(s) laggy due to laggy OSDs [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceefs is degraded [WRN] MDS_CLIENTS_LAGGY: 25 client(s) laggy due to laggy OSDs mds.ceefs.www7.drnuyi(mds.0): Client 14019719 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14124385 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14144243 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14144375 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14224103 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14224523 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14234194 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14234545 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14236841 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14237837 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14238536 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14244124 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14264236 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14266870 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14294170 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14294434 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14296012 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14304212 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14316057 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14318379 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14325518 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14328956 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14334283 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14336104 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14374237 is laggy; not evicted because some OSD(s) is/are laggy [ceph: root@www7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2.79994 root default -25 0.15999 host ceph01 1 hdd 0.15999 osd.1 up 0.15999 1.00000 -28 1.15999 host ceph03 3 hdd 0.15999 osd.3 up 0.15999 1.00000 5 hdd 1.00000 osd.5 up 1.00000 1.00000 -9 0.15999 host ceph06 2 hdd 0.15999 osd.2 up 0.15999 1.00000 -3 0.15999 host ceph07 6 hdd 0.15999 osd.6 up 0.15999 1.00000 -6 1.00000 host ceph08 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -7 0.15999 host www7 0 hdd 0.15999 osd.0 up 0.15999 1.00000 [ceph: root@www7 /]# ceph pg stat 681 pgs: 373 active+clean, 308 active+clean+remapped; 36 GiB data, 91 GiB used, 759 GiB / 850 GiB avail; 255 B/s rd, 0 op/s; 47772/369073 objects misplaced (12.944%)