When PGs are degraded they won't scrub, further, if an OSD is involved with recovery of another PG it wont accept scrubs either so that is the likely explanation of your not-scrubbed-in time issue. Its of low concern. Are you sure that recovery is not progressing? I see: "7349/147534197 objects degraded" can you check that again (maybe wait an hour) and see if 7,349 has been reduced. Another thing I'm noticing is that OSD 57 and 79 are the primary for many of the PGs which are degraded. They might could use a service restart. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Mon, Jun 5, 2023 at 12:01 PM Nicola Mori <mori@fi.infn.it> wrote:
Dear Ceph users,
after an outage and recovery of one machine I have several PGs stuck in active+recovering+undersized+degraded+remapped. Furthermore, many PGs have not been (deep-)scrubbed in time. See below for status and health details. It's been like this for two days, with no recovery I/O being reported, so I guess something is stuck in a bad state. I'd need some help in understanding what's going on here and how to fix it. Thanks,
Nicola
---------------------
# ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 2 OSD(s) have spurious read errors Degraded data redundancy: 7349/147534197 objects degraded (0.005%), 22 pgs degraded, 22 pgs undersized 332 pgs not deep-scrubbed in time 503 pgs not scrubbed in time (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT)
services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 2d) mgr: bofur.tklnrn(active, since 32h), standbys: balin.hvunfe, aka.wzystq mds: 2/2 daemons up, 1 standby osd: 104 osds: 104 up (since 37h), 104 in (since 37h); 22 remapped pgs
data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.53M objects, 40 TiB usage: 54 TiB used, 142 TiB / 196 TiB avail pgs: 7349/147534197 objects degraded (0.005%) 2715/147534197 objects misplaced (0.002%) 507 active+clean 20 active+recovering+undersized+degraded+remapped 2 active+recovery_wait+undersized+degraded+remapped
# ceph health detail [WRN] PG_DEGRADED: Degraded data redundancy: 7349/147534197 objects degraded (0.005%), 22 pgs degraded, 22 pgs undersized pg 3.2c is stuck undersized for 37h, current state active+recovery_wait+undersized+degraded+remapped, last acting [79,83,34,37,65,NONE,18,95] pg 3.57 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,99,37,NONE,15,104,55,40] pg 3.76 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,5,37,15,100,33,85,NONE] pg 3.9c is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,86,88,NONE,11,69,20,10] pg 3.106 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,15,89,NONE,36,32,23,64] pg 3.107 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,NONE,64,20,61,92,104,43] pg 3.10c is stuck undersized for 37h, current state active+recovery_wait+undersized+degraded+remapped, last acting [79,34,NONE,95,104,16,69,18] pg 3.11e is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,89,64,46,32,NONE,40,15] pg 3.14e is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,34,69,97,85,NONE,46,62] pg 3.160 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,1,101,84,18,33,NONE,69] pg 3.16a is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,16,59,103,13,38,49,NONE] pg 3.16e is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,0,27,96,55,10,81,NONE] pg 3.170 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [NONE,57,14,46,55,99,15,40] pg 3.19b is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [NONE,79,59,8,32,17,7,90] pg 3.1a0 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [NONE,79,26,50,104,24,97,40] pg 3.1a5 is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,100,61,27,20,NONE,24,85] pg 3.1a8 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,24,NONE,3,55,40,98,45] pg 3.1aa is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,91,48,NONE,24,3,8,85] pg 3.1af is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,NONE,90,33,104,69,26,8] pg 3.1c1 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,95,NONE,53,54,27,18,85] pg 3.1c4 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,69,56,84,95,8,NONE,4] pg 3.1d5 is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,48,NONE,104,34,16,37,89] [WRN] PG_NOT_DEEP_SCRUBBED: 332 pgs not deep-scrubbed in time pg 3.1ff not deep-scrubbed since 2023-05-18T21:06:57.883787+0000 pg 3.1fe not deep-scrubbed since 2023-05-22T19:50:11.497538+0000 pg 3.1fd not deep-scrubbed since 2023-05-22T19:44:12.680598+0000 pg 3.1fc not deep-scrubbed since 2023-05-20T19:56:43.746580+0000 pg 3.1fb not deep-scrubbed since 2023-05-22T18:29:12.794152+0000 pg 3.1f9 not deep-scrubbed since 2023-05-19T08:19:16.636964+0000 pg 3.1f8 not deep-scrubbed since 2023-05-22T21:49:28.891350+0000 pg 3.1f5 not deep-scrubbed since 2023-05-18T21:18:19.636068+0000 pg 3.1f4 not deep-scrubbed since 2023-05-18T18:00:41.241562+0000 pg 3.1f3 not deep-scrubbed since 2023-05-21T01:36:32.735139+0000 pg 3.1f2 not deep-scrubbed since 2023-05-23T03:59:02.154966+0000 pg 3.1f1 not deep-scrubbed since 2023-05-22T21:47:46.419880+0000 pg 3.1f0 not deep-scrubbed since 2023-05-22T19:17:38.327356+0000 pg 3.1ef not deep-scrubbed since 2023-05-19T01:49:04.133392+0000 pg 3.1ee not deep-scrubbed since 2023-05-21T12:25:52.010406+0000 pg 3.1ed not deep-scrubbed since 2023-05-19T20:13:20.675257+0000 pg 3.1eb not deep-scrubbed since 2023-05-18T12:13:53.684650+0000 pg 3.1ea not deep-scrubbed since 2023-05-18T09:45:57.172578+0000 pg 3.1e9 not deep-scrubbed since 2023-05-23T00:26:18.621324+0000 pg 3.1e8 not deep-scrubbed since 2023-05-21T05:15:03.969687+0000 pg 3.1e4 not deep-scrubbed since 2023-05-21T16:21:11.738145+0000 pg 3.1e3 not deep-scrubbed since 2023-05-22T13:13:19.611165+0000 pg 3.1e0 not deep-scrubbed since 2023-05-21T17:43:36.545240+0000 pg 3.1de not deep-scrubbed since 2023-05-18T00:03:49.873073+0000 pg 3.1dd not deep-scrubbed since 2023-05-22T20:30:56.025015+0000 pg 3.1db not deep-scrubbed since 2023-05-22T18:12:44.615539+0000 pg 3.1da not deep-scrubbed since 2023-05-20T21:11:00.060022+0000 pg 3.1d9 not deep-scrubbed since 2023-05-22T19:02:03.292022+0000 pg 3.1d8 not deep-scrubbed since 2023-05-23T17:37:05.320161+0000 pg 3.1d6 not deep-scrubbed since 2023-05-19T15:19:58.293551+0000 pg 3.1d4 not deep-scrubbed since 2023-05-23T02:28:54.392188+0000 pg 3.1d3 not deep-scrubbed since 2023-05-18T06:02:14.181321+0000 pg 3.1d2 not deep-scrubbed since 2023-05-18T11:46:29.582700+0000 pg 3.1d1 not deep-scrubbed since 2023-05-19T08:31:54.033426+0000 pg 3.1cd not deep-scrubbed since 2023-05-21T08:52:41.817826+0000 pg 3.1cc not deep-scrubbed since 2023-05-22T22:51:02.466708+0000 pg 3.1c9 not deep-scrubbed since 2023-05-18T08:06:50.220587+0000 pg 3.1c7 not deep-scrubbed since 2023-05-22T17:07:35.346608+0000 pg 3.1c5 not deep-scrubbed since 2023-05-20T17:09:12.048012+0000 pg 3.1c1 not deep-scrubbed since 2023-05-21T11:39:47.640196+0000 pg 3.1c0 not deep-scrubbed since 2023-05-22T20:22:57.166475+0000 pg 3.1bf not deep-scrubbed since 2023-05-19T19:08:08.313143+0000 pg 3.1be not deep-scrubbed since 2023-05-21T12:28:17.345386+0000 pg 3.1bd not deep-scrubbed since 2023-05-18T19:19:29.002801+0000 pg 3.1bb not deep-scrubbed since 2023-05-19T07:15:53.508751+0000 pg 3.1b8 not deep-scrubbed since 2023-05-19T18:50:27.701909+0000 pg 3.1b6 not deep-scrubbed since 2023-05-19T03:30:55.707248+0000 pg 3.1b5 not deep-scrubbed since 2023-05-20T20:37:48.346272+0000 pg 3.1b4 not deep-scrubbed since 2023-05-23T02:11:04.833784+0000 pg 3.1b3 not deep-scrubbed since 2023-05-18T20:46:40.876590+0000 282 more pgs... [WRN] PG_NOT_SCRUBBED: 503 pgs not scrubbed in time pg 3.1ff not scrubbed since 2023-05-24T23:37:22.323516+0000 pg 3.1fe not scrubbed since 2023-05-25T02:01:18.754476+0000 pg 3.1fd not scrubbed since 2023-05-24T20:31:23.239794+0000 pg 3.1fc not scrubbed since 2023-05-25T00:42:05.670791+0000 pg 3.1fb not scrubbed since 2023-05-24T19:29:29.438626+0000 pg 3.1fa not scrubbed since 2023-05-24T21:50:04.911965+0000 pg 3.1f9 not scrubbed since 2023-05-25T20:44:49.010622+0000 pg 3.1f8 not scrubbed since 2023-05-24T18:17:49.471926+0000 pg 3.1f7 not scrubbed since 2023-05-24T17:27:43.545337+0000 pg 3.1f6 not scrubbed since 2023-05-24T22:16:04.008644+0000 pg 3.1f5 not scrubbed since 2023-05-24T20:14:01.159271+0000 pg 3.1f4 not scrubbed since 2023-05-24T16:20:29.746958+0000 pg 3.1f3 not scrubbed since 2023-05-25T00:45:49.464448+0000 pg 3.1f2 not scrubbed since 2023-05-24T17:37:58.701570+0000 pg 3.1f1 not scrubbed since 2023-05-24T20:21:46.824657+0000 pg 3.1f0 not scrubbed since 2023-05-25T00:59:02.693836+0000 pg 3.1ef not scrubbed since 2023-05-24T21:35:10.061965+0000 pg 3.1ee not scrubbed since 2023-05-24T17:13:37.835095+0000 pg 3.1ed not scrubbed since 2023-05-24T18:17:21.739348+0000 pg 3.1ec not scrubbed since 2023-05-24T17:54:23.365899+0000 pg 3.1eb not scrubbed since 2023-05-24T23:18:31.345229+0000 pg 3.1ea not scrubbed since 2023-05-25T00:25:06.747723+0000 pg 3.1e9 not scrubbed since 2023-05-25T19:27:39.496774+0000 pg 3.1e8 not scrubbed since 2023-05-25T01:31:11.083814+0000 pg 3.1e7 not scrubbed since 2023-05-25T01:43:43.116599+0000 pg 3.1e6 not scrubbed since 2023-05-24T18:26:39.778008+0000 pg 3.1e4 not scrubbed since 2023-05-24T22:18:59.986309+0000 pg 3.1e3 not scrubbed since 2023-05-24T14:34:52.095564+0000 pg 3.1e2 not scrubbed since 2023-05-24T23:56:04.083842+0000 pg 3.1e1 not scrubbed since 2023-05-25T02:00:18.766811+0000 pg 3.1e0 not scrubbed since 2023-05-25T02:01:42.094304+0000 pg 3.1df not scrubbed since 2023-05-24T19:41:59.890557+0000 pg 3.1de not scrubbed since 2023-05-24T23:57:49.463552+0000 pg 3.1dd not scrubbed since 2023-05-25T17:42:33.397660+0000 pg 3.1dc not scrubbed since 2023-05-24T17:34:43.656366+0000 pg 3.1db not scrubbed since 2023-05-24T21:48:10.126232+0000 pg 3.1da not scrubbed since 2023-05-24T17:54:43.136739+0000 pg 3.1d9 not scrubbed since 2023-05-24T20:22:14.256914+0000 pg 3.1d8 not scrubbed since 2023-05-24T23:34:56.555311+0000 pg 3.1d7 not scrubbed since 2023-05-25T18:08:08.689329+0000 pg 3.1d6 not scrubbed since 2023-05-24T20:23:30.301130+0000 pg 3.1d5 not scrubbed since 2023-05-25T20:30:25.691077+0000 pg 3.1d4 not scrubbed since 2023-05-24T21:21:46.923743+0000 pg 3.1d3 not scrubbed since 2023-05-24T18:12:50.468466+0000 pg 3.1d2 not scrubbed since 2023-05-24T20:33:32.376232+0000 pg 3.1d1 not scrubbed since 2023-05-24T20:32:55.981738+0000 pg 3.1d0 not scrubbed since 2023-05-24T18:16:51.195524+0000 pg 3.1cf not scrubbed since 2023-05-24T22:32:00.879058+0000 pg 3.1ce not scrubbed since 2023-05-25T02:46:02.834267+0000 pg 3.1cd not scrubbed since 2023-05-24T21:02:08.288116+0000 453 more pgs... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io