[ceph-users] ceph Nautilus lost two disk over night everything hangs

Hello, i run a ceph Nautilus cluster with 9 hosts and 144 OSDs. Last night we lost two disks, so two OSDs (67,90) are down. The two disks are on two different hosts. A third ODS on a third host repotrts slow ops. ceph is repairing at the moment. Pools affected are eg these ones: pool 35 'pxa-rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 192082 lfor 0/27841/27845 flags hashpspool,selfmanaged_snaps stripe_width 0 pg_num_min 128 target_size_ratio 0.0001 application rbd pool 36 'pxa-ec' erasure size 6 min_size 5 crush_rule 7 object_hash rjenkins pg_num 512 pgp_num 512 last_change 192177 lfor 0/172580/172578 flags hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384 pg_num_min 512 target_size_ratio 0.15 application rbd At the mmoment the proxmox-cluster using storage from the seperate ceph cluster hangs. The ppols with date are erasure coded with the following profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 What I do not understand is why access on the virtualization seem to block. Could that be related to min_size of the pools cause this behaviour? How can I find out if this is true or what else is causing the blocking behaviour seen? This is the current status: health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete Degraded data redundancy: 42384/130014984 objects degraded (0.033%), 4 pgs degraded, 5 pgs undersized 15 daemons have recently crashed 150 slow ops, oldest one blocked for 15901 sec, daemons [osd.60,osd.67] have slow ops. services: mon: 3 daemons, quorum ceph2,ceph5,ceph8 (age 4h) mgr: ceph2(active, since 7w), standbys: ceph5, ceph8, ceph-admin mds: cephfsrz:1 {0=ceph6=up:active} 2 up:standby osd: 144 osds: 142 up (since 4h), 142 in (since 5h); 6 remapped pgs task status: scrub status: mds.ceph6: idle data: pools: 15 pools, 2632 pgs objects: 21.70M objects, 80 TiB usage: 139 TiB used, 378 TiB / 517 TiB avail pgs: 0.038% pgs not active 42384/130014984 objects degraded (0.033%) 2623 active+clean 3 active+undersized+degraded+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfill_wait 1 remapped+incomplete io: client: 2.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s rd, 179 op/s wr recovery: 51 MiB/s, 12 objects/s Thanks a lot Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312