Hello Seeking some help to recover these incomplete pgs… I made osd, down then tried but nothing seems working …. Ceph version 18.2.7, we have multiple disk failed in ceph cluster on different nodes… for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do ceph pg map $i | grep -q "acting.*1" && \ ceph-objectstore-tool --data-path /var/lib/ceph/15688cb4-044a-11ec-942e-516035adea04/osd.17 --op mark-complete --pgid $i --force done # for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do ceph pg map $i | grep -q "acting.*1" && \ ceph-objectstore-tool --data-path /var/lib/ceph/15688cb4-044a-11ec-942e-516035adea04/osd.17 --op repair --pgid $i --force done & pg 16.48 is incomplete, acting [20,9,17] pg 16.50 is incomplete, acting [20,27,5] pg 16.68 is incomplete, acting [2,5,27] pg 16.78 is incomplete, acting [23,14,29] pg 17.4f is incomplete, acting [10,30,18] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.44 is incomplete, acting [10,4,21,29,0] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.45 is stuck inactive since forever, current state incomplete, last acting [10,28,24,4,0] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.49 is incomplete, acting [28,10,26,0,15] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4a is incomplete, acting [21,1,23,0,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4b is incomplete, acting [6,12,17,30,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4c is incomplete, acting [5,0,8,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4f is incomplete, acting [26,14,5,7,9] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.51 is incomplete, acting [20,14,28,26,23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.52 is incomplete, acting [10,26,22,30,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.53 is incomplete, acting [12,24,25,23,22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.54 is incomplete, acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.57 is incomplete, acting [15,22,9,0,14] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,0,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,0,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5d is incomplete, acting [30,19,4,17,15] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5e is incomplete, acting [3,9,10,17,16] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5f is remapped+incomplete, acting [22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.60 is incomplete, acting [25,27,23,4,22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.61 is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.62 is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.63 is incomplete, acting [15,4,1,22,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.64 is incomplete, acting [4,7,5,29,31] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.65 is incomplete, acting [3,9,15,10,19] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.6b is remapped+incomplete, acting [1] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.6f is incomplete, acting [30,16,17,18,10] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.72 is remapped+incomplete, acting [19] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.73 is incomplete, acting [12,27,13,16,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.74 is incomplete, acting [17,18,19,4,29] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.75 is incomplete, acting [8,31,20,15,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.76 is incomplete, acting [5,26,18,30,21] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.78 is incomplete, acting [13,26,14,10,16] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7a is incomplete, acting [15,1,16,12,6] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7b is incomplete, acting [2,17,15,29,9] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7c is incomplete, acting [28,19,6,4,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7f is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 22.4b is incomplete, acting [14,7,23] pg 22.51 is incomplete, acting [11,8,2] pg 22.5a is incomplete, acting [31,15,4] pg 34.48 is incomplete, acting [31,19,17] pg 34.51 is incomplete, acting [15,12,20] pg 34.65 is incomplete, acting [30,18,20] pg 34.6d is incomplete, acting [30,12,21] pg 34.7c is incomplete, acting [31,9,3] # ceph -s cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 4 OSD(s) experiencing slow operations in BlueStore 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 1/16510107 objects unfound (0.000%) Reduced data availability: 449 pgs inactive, 444 pgs incomplete Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 3/60411723 objects degraded (0.000%), 1 pg degraded 343 pgs not deep-scrubbed in time 63 slow ops, oldest one blocked for 262107 sec, daemons [osd.1,osd.15,osd.16,osd.18,osd.19,osd.2,osd.21,osd.25,osd.26,osd.27]... have slow ops. services: mon: 7 daemons, quorum test-host04n,test-host05n,test-host02n,test-host03n,test-host06n,test-host07n,test-host08n (age 5h) mgr: test-host04n.azmqik(active, since 5h), standbys: test-host05n.lybhho mds: 3/3 daemons up, 11 standby osd: 32 osds: 32 up (since 18m), 32 in (since 18m); 36 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.51M objects, 55 TiB usage: 155 TiB used, 246 TiB / 401 TiB avail pgs: 0.332% pgs unknown 29.502% pgs not active 3/60411723 objects degraded (0.000%) 1250071/60411723 objects misplaced (2.069%) 1/16510107 objects unfound (0.000%) 1022 active+clean 438 incomplete 16 active+remapped+backfilling 14 active+remapped+backfill_wait 6 remapped+incomplete 5 unknown 3 active+clean+scrubbing+deep 1 active+recovery_unfound+degraded io: recovery: 1.3 GiB/s, 345 objects/s progress: Global Recovery Event (4h) [===================.........] (remaining: 2h) Regards Dev