EC 8+3 Pool PGs stuck in remapped+incomplete
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs, # ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396 [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548 Some of the OSDs had gone down on the cluster. Below is the # ceph status # ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2) data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr And the health detail shows as # ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time We've the pools as below # ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated! Regards, Jayanth Reddy
problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy! in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs" But it looks to me that you have way too many PGs which is also super negatively impacting performance. Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly. kind regards, Nino On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)*
5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs,
# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548
Some of the OSDs had gone down on the cluster. Below is the # ceph status
# ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2)
data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded
io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr
And the health detail shows as
# ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time
We've the pools as below
# ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec
We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated!
Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Nino. Would give these initial suggestions a try and let you know at the earliest. Regards, Jayanth Reddy ________________________________ From: Nino Kotur <ninokotur@gmail.com> Sent: Saturday, June 17, 2023 12:16:09 PM To: Jayanth Reddy <jayanthreddy5666@gmail.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] EC 8+3 Pool PGs stuck in remapped+incomplete problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy! in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs" But it looks to me that you have way too many PGs which is also super negatively impacting performance. Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly. kind regards, Nino On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy <jayanthreddy5666@gmail.com<mailto:jayanthreddy5666@gmail.com>> wrote: Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs, # ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396 [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548 Some of the OSDs had gone down on the cluster. Below is the # ceph status # ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2) data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr And the health detail shows as # ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs<http://ceph.com/docs> for 'incomplete') pg 15.985 is remapped+incomplete, acting [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs<http://ceph.com/docs> for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs<http://ceph.com/docs> for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs<http://ceph.com/docs> for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs<http://ceph.com/docs> for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time We've the pools as below # ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated! Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hello Nino / Users, After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery. Thanks, Jayanth Reddy On Sat, Jun 17, 2023 at 12:31 PM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Thanks, Nino.
Would give these initial suggestions a try and let you know at the earliest.
Regards, Jayanth Reddy ------------------------------ *From:* Nino Kotur <ninokotur@gmail.com> *Sent:* Saturday, June 17, 2023 12:16:09 PM *To:* Jayanth Reddy <jayanthreddy5666@gmail.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* Re: [ceph-users] EC 8+3 Pool PGs stuck in remapped+incomplete
problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs
[osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy!
in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs"
But it looks to me that you have way too many PGs which is also super negatively impacting performance.
Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly.
kind regards, Nino
On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)*
5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs,
# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548
Some of the OSDs had gone down on the cluster. Below is the # ceph status
# ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2)
data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded
io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr
And the health detail shows as
# ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time
We've the pools as below
# ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec
We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated!
Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jayanth, Can you post the complete output of “ceph pg <ID> query”? So that we can understand the situation better. Can you get OSD 3 or 4 back into the cluster? If you are sure they cannot rejoin, you may try “ceph osd lost <ID>” (doc says this may result in permanent data lost. I didn’t have a chance to try this myself). Weiwen Hu
在 2023年6月18日，00:26，Jayanth Reddy <jayanthreddy5666@gmail.com> 写道：
Hello Nino / Users,
After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery.
Thanks, Jayanth Reddy
On Sat, Jun 17, 2023 at 12:31 PM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Thanks, Nino.
Would give these initial suggestions a try and let you know at the earliest.
Regards, Jayanth Reddy ------------------------------ *From:* Nino Kotur <ninokotur@gmail.com> *Sent:* Saturday, June 17, 2023 12:16:09 PM *To:* Jayanth Reddy <jayanthreddy5666@gmail.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* Re: [ceph-users] EC 8+3 Pool PGs stuck in remapped+incomplete
problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs
[osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy!
in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs"
But it looks to me that you have way too many PGs which is also super negatively impacting performance.
Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly.
kind regards, Nino
On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)*
5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs,
# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548
Some of the OSDs had gone down on the cluster. Below is the # ceph status
# ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2)
data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded
io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr
And the health detail shows as
# ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time
We've the pools as below
# ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec
We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated!
Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Weiwen, Thank you for the response. I've attached the output for all PGs in state incomplete and remapped+incomplete. Thank you! Thanks, Jayanth Reddy On Sat, Jun 17, 2023 at 11:00 PM 胡 玮文 <huww98@outlook.com> wrote:
Hi Jayanth,
Can you post the complete output of “ceph pg <ID> query”? So that we can understand the situation better.
Can you get OSD 3 or 4 back into the cluster? If you are sure they cannot rejoin, you may try “ceph osd lost <ID>” (doc says this may result in permanent data lost. I didn’t have a chance to try this myself).
Weiwen Hu
在 2023年6月18日，00:26，Jayanth Reddy <jayanthreddy5666@gmail.com> 写道：
Hello Nino / Users,
After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery.
Thanks, Jayanth Reddy
On Sat, Jun 17, 2023 at 12:31 PM Jayanth Reddy < jayanthreddy5666@gmail.com> wrote:
Thanks, Nino.
Would give these initial suggestions a try and let you know at the earliest.
Regards, Jayanth Reddy ------------------------------ *From:* Nino Kotur <ninokotur@gmail.com> *Sent:* Saturday, June 17, 2023 12:16:09 PM *To:* Jayanth Reddy <jayanthreddy5666@gmail.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* Re: [ceph-users] EC 8+3 Pool PGs stuck in remapped+incomplete
problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs
[osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy!
in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs"
But it looks to me that you have way too many PGs which is also super negatively impacting performance.
Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly.
kind regards, Nino
On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy < jayanthreddy5666@gmail.com> wrote:
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)*
5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs,
# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352]
214 15.39d3 remapped+incomplete
[404,221,223,585,38,102,533,471,568,451,195]
404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete
[297,646,212,254,110,169,500,372,623,470,678]
297
[2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678]
548
Some of the OSDs had gone down on the cluster. Below is the # ceph
status
# ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2)
data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded
io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr
And the health detail shows as
# ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive,
5
pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting
[2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time
We've the pools as below
# ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec
We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated!
Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Weiwen, Thank you for the response. I've attached the output for all PGs in state incomplete and remapped+incomplete. Thank you! Thanks, Jayanth Reddy On Sun, Jun 18, 2023 at 4:09 PM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Hello Weiwen,
Thank you for the response. I've attached the output for all PGs in state incomplete and remapped+incomplete. Thank you!
Thanks, Jayanth Reddy
On Sat, Jun 17, 2023 at 11:00 PM 胡 玮文 <huww98@outlook.com> wrote:
Hi Jayanth,
Can you post the complete output of “ceph pg <ID> query”? So that we can understand the situation better.
Can you get OSD 3 or 4 back into the cluster? If you are sure they cannot rejoin, you may try “ceph osd lost <ID>” (doc says this may result in permanent data lost. I didn’t have a chance to try this myself).
Weiwen Hu
在 2023年6月18日，00:26，Jayanth Reddy <jayanthreddy5666@gmail.com> 写道：
Hello Nino / Users,
After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery.
Thanks, Jayanth Reddy
On Sat, Jun 17, 2023 at 12:31 PM Jayanth Reddy < jayanthreddy5666@gmail.com> wrote:
Thanks, Nino.
Would give these initial suggestions a try and let you know at the earliest.
Regards, Jayanth Reddy ------------------------------ *From:* Nino Kotur <ninokotur@gmail.com> *Sent:* Saturday, June 17, 2023 12:16:09 PM *To:* Jayanth Reddy <jayanthreddy5666@gmail.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* Re: [ceph-users] EC 8+3 Pool PGs stuck in remapped+incomplete
problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs
[osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy!
in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs"
But it looks to me that you have way too many PGs which is also super negatively impacting performance.
Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly.
kind regards, Nino
On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy < jayanthreddy5666@gmail.com> wrote:
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)*
5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs,
# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352]
214 15.39d3 remapped+incomplete
[404,221,223,585,38,102,533,471,568,451,195]
404
[2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647]
223 15.d46 remapped+incomplete
[297,646,212,254,110,169,500,372,623,470,678]
297
[2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678]
548
Some of the OSDs had gone down on the cluster. Below is the # ceph
status
# ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped
pgs
flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2)
data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded
io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr
And the health detail shows as
# ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs
inactive, 5
pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting
[2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting
[2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time
We've the pools as below
# ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec
We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated!
Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Weiwen, Thank you for the response. I've attached the output for all PGs in state incomplete and remapped+incomplete. Thank you! Thanks, Jayanth Reddy On Sat, Jun 17, 2023 at 11:00 PM 胡 玮文 <huww98@outlook.com> wrote:
Hi Jayanth,
Can you post the complete output of “ceph pg <ID> query”? So that we can understand the situation better.
Can you get OSD 3 or 4 back into the cluster? If you are sure they cannot rejoin, you may try “ceph osd lost <ID>” (doc says this may result in permanent data lost. I didn’t have a chance to try this myself).
Weiwen Hu
在 2023年6月18日，00:26，Jayanth Reddy <jayanthreddy5666@gmail.com> 写道：
Hello Nino / Users,
After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery.
Thanks, Jayanth Reddy
On Sat, Jun 17, 2023 at 12:31 PM Jayanth Reddy < jayanthreddy5666@gmail.com> wrote:
Thanks, Nino.
Would give these initial suggestions a try and let you know at the earliest.
Regards, Jayanth Reddy ------------------------------ *From:* Nino Kotur <ninokotur@gmail.com> *Sent:* Saturday, June 17, 2023 12:16:09 PM *To:* Jayanth Reddy <jayanthreddy5666@gmail.com> *Cc:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* Re: [ceph-users] EC 8+3 Pool PGs stuck in remapped+incomplete
problem is just that some of your OSDs have too much PGs and pool cannot recover as it cannot create more PGs
[osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
I'd have to guess that the safest thing would be permanently or temporarily adding more storage so that PGs drop below 250, another option is just dropping down the total number of PGs but I don't know if I would perform that action before my pool was healthy!
in case that there is only one OSD that has this number of OSDs but all other OSDs have less than 100-150 than you can just reweight problematic OSD so it rebalances those "too many PGs"
But it looks to me that you have way too many PGs which is also super negatively impacting performance.
Another option is to increase max allowed PGs per OSD to say 350 this should also allow cluster to rebuild honestly even tho this may be easiest option, i'd never do this, performance cost of having over 150 PGs per OSD suffer greatly.
kind regards, Nino
On Sat, Jun 17, 2023 at 8:23 AM Jayanth Reddy < jayanthreddy5666@gmail.com> wrote:
Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)*
5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs,
# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352]
214 15.39d3 remapped+incomplete
[404,221,223,585,38,102,533,471,568,451,195]
404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete
[297,646,212,254,110,169,500,372,623,470,678]
297
[2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678]
548
Some of the OSDs had gone down on the cluster. Below is the # ceph
status
# ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250)
services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2)
data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded
io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr
And the health detail shows as
# ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive,
5
pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting
[2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting
[2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678]
(reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time
We've the pools as below
# ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec
We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated!
Regards, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Jayanth Reddy
-
Nino Kotur
-
胡 玮文