On 8/11/20 2:52 AM, Wido den Hollander wrote:

On 11/08/2020 00:40, Michael Thomas wrote:

...

On my relatively new Octopus cluster, I have one PG that has been perpetually stuck in the 'unknown' state.  It appears to belong to the device_health_metrics pool, which was created automatically by the mgr daemon(?).

The OSDs that the PG maps to are all online and serving other PGs. But when I list the PGs that belong to the OSDs from 'ceph pg map', the offending PG is not listed.

# ceph pg dump pgs | grep ^1.0 dumped pgs 1.0            0                   0         0          0        0 0            0           0      0         0       unknown 2020-08-08T09:30:33.251653-0500         0'0         0:0 [] -1                         []              -1  0'0 2020-08-08T09:30:33.251653-0500              0'0 2020-08-08T09:30:33.251653-0500              0

# ceph osd pool stats device_health_metrics pool device_health_metrics id 1    nothing is going on

# ceph pg map 1.0 osdmap e7199 pg 1.0 (1.0) -> up [41,40,2] acting [41,0]

What can be done to fix the PG?  I tried doing a 'ceph pg repair 1.0', but that didn't seem to do anything.

Is it safe to try to update the crush_rule for this pool so that the PG gets mapped to a fresh set of OSDs?

Yes, it would be. But still, it's weird. Mainly as the acting set is so different from the up-set.

You have different CRUSH rules I think?

Marking those OSDs down might work, but otherwise change the crush_rule and see how that goes.

Yes, I do have different crush rules to help map certain types of data to different classes of hardware (EC HDDs, replicated SSDs, replicated nvme). The default crush rule for the device_health_metrics pool was to use replication across any storage device. I changed it to use the replicated nvme crush rule, and now the map looks different: # ceph pg map 1.0 osdmap e7256 pg 1.0 (1.0) -> up [24,22,12] acting [41,0] However, the acting set of OSDs has not changed. --Mike

On 2020-08-22 03:19, Michael Thomas wrote:

...

Yes, I do have different crush rules to help map certain types of data to different classes of hardware (EC HDDs, replicated SSDs, replicated nvme).  The default crush rule for the device_health_metrics pool was to use replication across any storage device.  I changed it to use the replicated nvme crush rule, and now the map looks different:

# ceph pg map 1.0 osdmap e7256 pg 1.0 (1.0) -> up [24,22,12] acting [41,0]

However, the acting set of OSDs has not changed.

A little more info:

ceph status is reporting a slow OSD, which happens to be the primary OSD for the offending PG:

    health: HEALTH_WARN             1 pools have many more objects per pg than average             1 backfillfull osd(s)             2 nearfull osd(s)             Reduced data availability: 1 pg inactive             304 pgs not deep-scrubbed in time             2 pool(s) backfillfull             2294 slow ops, oldest one blocked for 1122032 sec, osd.41 has slow ops

According to documentation: "Up Set The ordered list of OSDs responsible for a particular placement group for a particular epoch according to CRUSH. Normally this is the same as the Acting Set, except when the Acting Set has been explicitly overridden via pg_temp in the OSD Map." Is that indeed the case? Can you see this in the osdmap (ceph osd dump |grep pg_temp)? Are there any upmaps for that PG, i.e. ceph osd dump | grep upmap | grep "1\.0"? If there is one you can try to remove it (ceph osd rm-pg-upmap-items 1.0). If there isn't one you can set one accordingly to your crush rule policy, i.e.: ceph osd pg-upmap-items 1.0 24 some_other_osd We would have a similar issue (hitting a bug in mimic which is now fixed) where this could happen to some PGs. We would then upmap the PG (and later remove it) to fix it. Maybe it works for this as well. Gr. Stefan