Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs, # ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396 [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548 Some of the OSDs had gone down on the cluster. Below is the # ceph status # ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2) data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr And the health detail shows as # ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time We've the pools as below # ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated! Regards, Jayanth Reddy