Looks like the immediate danger has passed by: [root@gnosis ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN nodown,noout flag(s) set 735 slow ops, oldest one blocked for 3573 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops. services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 268 up, 268 in flags nodown,noout data: pools: 10 pools, 2545 pgs objects: 86.76 M objects, 218 TiB usage: 277 TiB used, 1.5 PiB / 1.8 PiB avail pgs: 2537 active+clean 8 active+clean+scrubbing+deep io: client: 34 MiB/s rd, 24 MiB/s wr, 954 op/s rd, 1.01 kop/s wr I will prepare a new case with info we have collected so far. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Amit Ghadge <amitg.b14@gmail.com> Sent: 20 May 2020 09:44 To: Frank Schilder Subject: Re: [ceph-users] total ceph outage again, need help look like ceph-01 shows in starting, so I think that why command not executed and you also try to disable to scrubbing for temporary. On Wed, May 20, 2020 at 12:57 PM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Dear cephers, I'm sitting with a major ceph outage again. The mon/mgr hosts suffer from a packet storm of ceph traffic between ceph fs clients and the mons. No idea why this is happening. Main problem is, that I can't get through to the cluster. Admin commands hang forever: [root@gnosis ~]# ceph osd set nodown However, "ceph status" returns and shows me that I need to do something: [root@gnosis ~]# ceph status cluster: id: --- health: HEALTH_WARN 2 MDSs report slow metadata IOs 1 MDSs report slow requests 8 osds down services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active, starting), standbys: ceph-02, ceph-03 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 208 up, 216 in; 153 remapped pgs data: pools: 10 pools, 2545 pgs objects: 86.71 M objects, 218 TiB usage: 277 TiB used, 1.5 PiB / 1.8 PiB avail pgs: 2542 active+clean 3 active+clean+scrubbing+deep io: client: 152 MiB/s rd, 72 MiB/s wr, 854 op/s rd, 796 op/s wr Is there any way to get admin commands to the mons with higher priority? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>