Please don't drop the list from your response. The first question coming to mind is, why do you have a cache-tier if all your pools are on nvme decices anyway? I don't see any benefit here. Did you try the suggested workaround and disable the cache-tier? Zitat von Cedric <yipikai7@gmail.com>:
Thanks Eugen, see attached infos.
Some more details:
- commands that actually hangs: ceph balancer status ; rbd -p vms ls ; rados -p vms_cache cache-flush-evict-all - all scrub running on vms_caches pgs are stall / start in a loop without actually doing anything - all io are 0 both from ceph status or iostat on nodes
On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
some more details would be helpful, for example what's the pool size of the cache pool? Did you issue a PG split before or during the upgrade? This thread [1] deals with the same problem, the described workaround was to set hit_set_count to 0 and disable the cache layer until that is resolved. Afterwards you could enable the cache layer again. But keep in mind that the code for cache tier is entirely removed in Reef (IIRC).
Regards, Eugen
[1] https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-adding...
Zitat von Cedric <yipikai7@gmail.com>:
Hello,
Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we encounter an issue with a cache pool becoming completely stuck, relevant messages below:
pg xx.x has invalid (post-split) stats; must scrub before tier agent can activate
In OSD logs, scrubs are starting in a loop without succeeding for all pg of this pool.
What we already tried without luck so far:
- shutdown / restart OSD - rebalance pg between OSD - raise the memory on OSD - repeer PG
Any idea what is causing this? any help will be greatly appreciated
Thanks
Cédric _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io