Scrub stuck and 'pg has invalid (post-split) stat'
Hello, Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we encounter an issue with a cache pool becoming completely stuck, relevant messages below: pg xx.x has invalid (post-split) stats; must scrub before tier agent can activate In OSD logs, scrubs are starting in a loop without succeeding for all pg of this pool. What we already tried without luck so far: - shutdown / restart OSD - rebalance pg between OSD - raise the memory on OSD - repeer PG Any idea what is causing this? any help will be greatly appreciated Thanks Cédric
Hi, some more details would be helpful, for example what's the pool size of the cache pool? Did you issue a PG split before or during the upgrade? This thread [1] deals with the same problem, the described workaround was to set hit_set_count to 0 and disable the cache layer until that is resolved. Afterwards you could enable the cache layer again. But keep in mind that the code for cache tier is entirely removed in Reef (IIRC). Regards, Eugen [1] https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-adding... Zitat von Cedric <yipikai7@gmail.com>:
Hello,
Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we encounter an issue with a cache pool becoming completely stuck, relevant messages below:
pg xx.x has invalid (post-split) stats; must scrub before tier agent can activate
In OSD logs, scrubs are starting in a loop without succeeding for all pg of this pool.
What we already tried without luck so far:
- shutdown / restart OSD - rebalance pg between OSD - raise the memory on OSD - repeer PG
Any idea what is causing this? any help will be greatly appreciated
Thanks
Cédric _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen, We used to have cache tiering (hdd+ssd) for openstack nova/glance in the past before we move to nvme hardware. But we were not able to evict all objects because it required to shutdown all virtual instances and then do the eviction. So we decided to set the cache mode to "proxy" and leave it as is until we get a time frame to shutdown all instances bound to this ceph cluster (but it never happened). REgards,
participants (3)
-
Cedric
-
Eugen Block
-
florian.leduc@socgen.com