Have you tried to dump the stuck ops from that OSD? It could point to a misbehaving client, I believe there was a thread about that recently in this list. I don’t have the exact command right now but check (within cephadm shell) ‚ceph daemon osd.7 help‘ for the ‚dump‘ options. Zitat von Arvid Picciani <aep@exys.org>:
Hi,
today our entire cluster froze. or anything that uses librbd to be specific. ceph version 16.2.10
The message that saved me was "256 slow ops, oldest one blocked for 2893 sec, osd.7 has slow ops" , because it makes it immediately clear that this osd is the issue.
I stopped the osd, which made the cluster available again. Restarting the osd makes it stuck again, although that osd has nothing in the error log, and the underlying ssd is healthy. It's just that one out of 27. There's nothing unique about it. We use the same disk product in other osds, and the host is also running other osds just fine.
How does this happen, and why can the cluster not recover from this automatically? For example by stopping the affected osd or at least having a timeout for ops.
Thanks
-- +4916093821054 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io