6 PG's stuck not-active, remapped
We recently did some work on the Ceph cluster, and a few disks ended up offline at the same time. There are now 6 PG's that are stuck in a "remapped" state, and this is all of their recovery states: *recovery_state: 0: name: Started/Primary/WaitActingChangeenter_time: 2020-10-21 18:48:02.034430comment: waiting for pg acting set to change1: name: Startedenter_time: 2020-10-21 18:48:01.752957* Any ideas? Mac Wynkoop
As an example, here's the acting and up set of one of the PG's: *up: 0: 1131: 1382: 303: 1324: 1055: 576: 1067: 1408: 161acting: 0: 721: 1502: 21474836473: 21474836474: 245: 486: 327: 1578: 103* So obviously there's a lot of backfilling there... but it seems it's not making any progress. Mac Wynkoop, Senior Datacenter Engineer *NetDepot.com:* Cloud Servers; Delivered Houston | Atlanta | NYC | Colorado Springs 1-844-25-CLOUD Ext 806 On Wed, Oct 21, 2020 at 2:41 PM Mac Wynkoop <mwynkoop@netdepot.com> wrote:
We recently did some work on the Ceph cluster, and a few disks ended up offline at the same time. There are now 6 PG's that are stuck in a "remapped" state, and this is all of their recovery states:
*recovery_state: 0: name: Started/Primary/WaitActingChangeenter_time: 2020-10-21 18:48:02.034430comment: waiting for pg acting set to change1: name: Startedenter_time: 2020-10-21 18:48:01.752957* Any ideas?
Mac Wynkoop
Hi, On 10/21/20 10:01 PM, Mac Wynkoop wrote: *snipsnap*
*up: 0: 1131: 1382: 303: 1324: 1055: 576: 1067: 1408: 161acting: 0: 721: 1502: 21474836473: 21474836474: 245: 486: 327: 1578: 103*
21474836473 is -1 as unsigned integer. This value means that the CRUSH algorithm did not produce enough OSDS to satisfy the PG requirements (e.g. less than three different OSDs for replicated pools with size=3). You mentioned that some disks are currently offline; if they are marked out your current cluster setup might not be sufficient for your crush rules. Bring the disks back online or (in case of sufficient hosts/osds) change the number of iterations in crush before giving up. The pseudo random characteristics may not always be able to select three hosts out of three available ones ;-) Regards, Burkhard
participants (2)
-
Burkhard Linke
-
Mac Wynkoop