HEALTH_ERR 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal'; noout flag(s) set; 1 scrub errors; Reduced data availability: 218 pgs inactive, 218 pgs incomplete; Possible data damage: 1 pg inconsistent; 267 pgs not deep-scrubbed in time; 37 pgs not scrubbed in time; 116 slow ops, oldest one blocked for 56681 sec, daemons [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.18]... have slow ops.; too many PGs per OSD (317 > max 250) [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.server(mds.0): 40 slow metadata IOs are blocked > 30 secs, oldest blocked for 56623 secs [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.server(mds.0): 98 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.server(mds.0): Behind on trimming (593/128) max_segments: 128, num_segments: 593 [WRN] MGR_MODULE_DEPENDENCY: Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal' Module 'telemetry' has failed dependency: module 'pip' has no attribute '_internal' [WRN] OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [WRN] PG_AVAILABILITY: Reduced data availability: 218 pgs inactive, 218 pgs incomplete pg 47.1c5 is incomplete, acting [6,7,1] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.1c8 is stuck inactive since forever, current state incomplete, last acting [27,12,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.1c9 is incomplete, acting [7,12,9] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.206 is incomplete, acting [1,19,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.20b is incomplete, acting [4,6,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.20c is incomplete, acting [7,0,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.20e is incomplete, acting [0,6,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.217 is incomplete, acting [19,8,1] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.21d is incomplete, acting [19,2,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.21f is incomplete, acting [27,1,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.222 is incomplete, acting [9,10,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.227 is incomplete, acting [7,8,13] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.22b is incomplete, acting [3,9,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.22d is incomplete, acting [28,10,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.22e is incomplete, acting [18,19,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.230 is incomplete, acting [4,30,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.234 is incomplete, acting [10,17,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.243 is incomplete, acting [18,8,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.244 is incomplete, acting [30,0,10] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.24e is incomplete, acting [12,7,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.254 is incomplete, acting [7,23,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.255 is incomplete, acting [33,30,8] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.258 is incomplete, acting [6,0,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.259 is incomplete, acting [8,30,5] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.25d is incomplete, acting [27,7,0] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.25e is incomplete, acting [28,8,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.261 is incomplete, acting [30,18,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.266 is incomplete, acting [8,16,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.26a is incomplete, acting [7,16,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.27a is incomplete, acting [16,9,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.27f is incomplete, acting [27,5,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.280 is incomplete, acting [27,10,2] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.281 is incomplete, acting [10,17,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.284 is incomplete, acting [28,8,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.286 is incomplete, acting [30,7,4] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.28e is incomplete, acting [10,7,12] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.290 is incomplete, acting [3,8,9] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.298 is incomplete, acting [18,10,19] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.299 is incomplete, acting [11,10,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.29a is incomplete, acting [19,17,7] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2a5 is incomplete, acting [6,27,0] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2a7 is incomplete, acting [30,12,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2a9 is incomplete, acting [7,10,13] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2ac is incomplete, acting [9,19,15] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2ae is incomplete, acting [3,4,27] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2af is incomplete, acting [30,19,2] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2b0 is incomplete, acting [30,12,28] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2b3 is incomplete, acting [27,12,6] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2b8 is incomplete, acting [19,17,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2bc is incomplete, acting [8,7,3] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 47.2bf is incomplete, acting [9,23,30] (reducing pool claypool min_size from 2 may help; search ceph.com/docs for 'incomplete') [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 21.280 is active+clean+inconsistent, acting [3,14,7] [WRN] PG_NOT_DEEP_SCRUBBED: 267 pgs not deep-scrubbed in time pg 47.286 not deep-scrubbed since 2021-03-25T22:53:15.065688-0600 pg 21.2bb not deep-scrubbed since 2021-03-26T20:24:27.616173-0600 pg 47.280 not deep-scrubbed since 2021-03-27T21:20:30.399908-0600 pg 47.28e not deep-scrubbed since 2021-03-27T03:44:01.214168-0600 pg 47.28b not deep-scrubbed since 2021-03-27T22:45:21.260059-0600 pg 47.290 not deep-scrubbed since 2021-03-27T03:05:59.228923-0600 pg 47.2a5 not deep-scrubbed since 2021-03-24T01:36:27.398025-0600 pg 47.2a7 not deep-scrubbed since 2021-03-27T21:32:01.793734-0600 pg 21.29c not deep-scrubbed since 2021-03-27T03:08:53.853499-0600 pg 47.2ad not deep-scrubbed since 2021-03-27T22:29:02.738458-0600 pg 47.2a9 not deep-scrubbed since 2021-03-25T20:58:47.775813-0600 pg 47.24d not deep-scrubbed since 2021-03-27T21:15:10.218831-0600 pg 21.274 not deep-scrubbed since 2021-03-27T02:20:16.591959-0600 pg 21.271 not deep-scrubbed since 2021-03-27T23:09:24.527503-0600 pg 47.255 not deep-scrubbed since 2021-03-25T20:14:25.340283-0600 pg 47.25e not deep-scrubbed since 2021-03-27T00:12:19.948412-0600 pg 47.259 not deep-scrubbed since 2021-03-21T01:05:52.666420-0600 pg 47.258 not deep-scrubbed since 2021-03-27T20:33:04.243319-0600 pg 21.262 not deep-scrubbed since 2021-03-27T01:58:50.415874-0600 pg 47.266 not deep-scrubbed since 2021-03-24T23:14:49.566588-0600 pg 47.261 not deep-scrubbed since 2021-03-26T20:41:03.399382-0600 pg 47.269 not deep-scrubbed since 2021-03-26T02:15:44.559049-0600 pg 47.26a not deep-scrubbed since 2021-03-26T01:25:59.521139-0600 pg 47.276 not deep-scrubbed since 2021-03-27T21:33:07.907478-0600 pg 47.27f not deep-scrubbed since 2021-03-24T23:39:08.357812-0600 pg 21.245 not deep-scrubbed since 2021-03-26T19:17:26.180939-0600 pg 47.206 not deep-scrubbed since 2021-03-24T03:59:12.427505-0600 pg 47.20e not deep-scrubbed since 2021-03-26T02:50:03.752634-0600 pg 47.20b not deep-scrubbed since 2021-03-26T20:19:01.735932-0600 pg 47.215 not deep-scrubbed since 2021-03-27T20:17:49.992450-0600 pg 47.217 not deep-scrubbed since 2021-03-27T23:26:09.896032-0600 pg 47.21d not deep-scrubbed since 2021-03-28T00:04:41.815145-0600 pg 47.21f not deep-scrubbed since 2021-03-26T19:50:30.551545-0600 pg 47.21e not deep-scrubbed since 2021-03-26T01:59:49.637250-0600 pg 47.227 not deep-scrubbed since 2021-03-25T03:39:00.406340-0600 pg 47.222 not deep-scrubbed since 2021-03-26T21:23:24.045585-0600 pg 21.218 not deep-scrubbed since 2021-03-27T03:16:35.287831-0600 pg 47.22b not deep-scrubbed since 2021-03-26T03:42:26.835052-0600 pg 47.230 not deep-scrubbed since 2021-03-27T02:49:41.789992-0600 pg 47.1c4 not deep-scrubbed since 2021-03-27T23:07:14.725369-0600 pg 21.1fd not deep-scrubbed since 2021-03-27T01:31:13.297653-0600 pg 47.1cb not deep-scrubbed since 2021-03-27T19:06:05.459813-0600 pg 21.1ef not deep-scrubbed since 2021-03-26T20:29:09.801484-0600 pg 47.1d7 not deep-scrubbed since 2021-03-28T00:01:02.836931-0600 pg 47.1da not deep-scrubbed since 2021-03-26T23:50:21.899786-0600 pg 21.1df not deep-scrubbed since 2021-03-26T01:34:56.338349-0600 pg 47.1e1 not deep-scrubbed since 2021-03-27T19:50:13.164921-0600 pg 47.1e0 not deep-scrubbed since 2021-03-27T01:38:05.517845-0600 pg 47.1f7 not deep-scrubbed since 2021-03-26T23:42:36.370211-0600 pg 47.1f9 not deep-scrubbed since 2021-03-24T23:14:41.816716-0600 217 more pgs... [WRN] PG_NOT_SCRUBBED: 37 pgs not scrubbed in time pg 47.284 not scrubbed since 2021-03-28T21:43:34.141833-0600 pg 47.28e not scrubbed since 2021-03-29T00:36:47.386296-0600 pg 47.2af not scrubbed since 2021-03-28T03:26:31.969098-0600 pg 47.2b0 not scrubbed since 2021-03-29T02:18:12.708308-0600 pg 47.254 not scrubbed since 2021-03-28T19:41:04.896055-0600 pg 47.25e not scrubbed since 2021-03-29T01:32:11.012945-0600 pg 47.259 not scrubbed since 2021-03-27T22:39:37.697021-0600 pg 47.266 not scrubbed since 2021-03-28T03:35:27.190995-0600 pg 47.227 not scrubbed since 2021-03-28T01:16:28.930572-0600 pg 47.234 not scrubbed since 2021-03-28T21:24:34.834498-0600 pg 47.1c5 not scrubbed since 2021-03-29T00:09:05.664124-0600 pg 47.1e0 not scrubbed since 2021-03-28T23:57:25.436078-0600 pg 47.1f9 not scrubbed since 2021-03-29T02:19:18.367248-0600 pg 47.181 not scrubbed since 2021-03-29T01:55:17.182313-0600 pg 47.ac not scrubbed since 2021-03-28T03:01:26.639534-0600 pg 47.b2 not scrubbed since 2021-03-28T19:00:52.306016-0600 pg 47.4f not scrubbed since 2021-03-28T19:47:56.332962-0600 pg 47.6b not scrubbed since 2021-03-28T19:08:20.531682-0600 pg 47.7d not scrubbed since 2021-03-29T00:47:33.754276-0600 pg 47.39 not scrubbed since 2021-03-28T21:02:51.726250-0600 pg 47.38 not scrubbed since 2021-03-28T21:52:56.209769-0600 pg 47.81 not scrubbed since 2021-03-29T03:10:09.500658-0600 pg 47.f8 not scrubbed since 2021-03-28T00:24:32.390309-0600 pg 47.d2 not scrubbed since 2021-03-28T19:01:14.191464-0600 pg 47.c1 not scrubbed since 2021-03-28T20:18:17.342293-0600 pg 47.12e not scrubbed since 2021-03-28T21:08:38.252809-0600 pg 47.121 not scrubbed since 2021-03-28T03:21:15.443939-0600 pg 47.11e not scrubbed since 2021-03-29T02:51:23.634263-0600 pg 47.2ea not scrubbed since 2021-03-28T20:45:30.210125-0600 pg 47.2d7 not scrubbed since 2021-03-29T01:04:27.951777-0600 pg 47.303 not scrubbed since 2021-03-27T22:54:34.374295-0600 pg 47.35a not scrubbed since 2021-03-28T03:19:21.985850-0600 pg 47.35c not scrubbed since 2021-03-27T21:49:11.236313-0600 pg 47.3a2 not scrubbed since 2021-03-28T00:48:35.353609-0600 pg 47.3a5 not scrubbed since 2021-03-29T00:02:18.249404-0600 pg 47.3dd not scrubbed since 2021-03-29T02:45:38.444232-0600 pg 47.3d4 not scrubbed since 2021-03-28T02:04:07.939796-0600 [WRN] SLOW_OPS: 116 slow ops, oldest one blocked for 56681 sec, daemons [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.18]... have slow ops. [WRN] TOO_MANY_PGS: too many PGs per OSD (317 > max 250)