Dear Xiubo, could you explain how to enable kernel debug logs (I assume this is on the client)? Thanks, Manuel On Fri, May 13, 2022 at 9:39 AM Xiubo Li <xiubli@redhat.com> wrote:

On 5/12/22 12:06 AM, Stefan Kooman wrote:

...

Hi List,

We have quite a few linux kernel clients for CephFS. One of our customers has been running mainline kernels (CentOS 7 elrepo) for the past two years. They started out with 3.x kernels (default CentOS 7), but upgraded to mainline when those kernels would frequently generate MDS warnings like "failing to respond to capability release". That worked fine until 5.14 kernel. 5.14 and up would use a lot of CPU and *way* more bandwidth on CephFS than older kernels (order of magnitude). After the MDS was upgraded from Nautilus to Octopus that behavior is gone (comparable CPU / bandwidth usage as older kernels). However, the newer kernels are now the ones that give "failing to respond to capability release", and worse, clients get evicted (unresponsive as far as the MDS is concerned). Even the latest 5.17 kernels have that. No difference is observed between using messenger v1 or v2. MDS version is 15.2.16. Surprisingly the latest stable kernels from CentOS 7 work flawlessly now. Although that is good news, newer operating systems come with newer kernels.

Does anyone else observe the same behavior with newish kernel clients?

There have some known bugs, which have been fixed or under fixing recently, even in the mainline and, not sure whether are they related. Such as [1][2][3][4]. More detail please see ceph-client repo testing branch [5].

I have never see the "failing to respond to capability release" issue yet, if you have the MDS logs(debug_mds = 25 and debug_ms = 1) and kernel debug logs will be better to help debug it further, or provide the steps to reproduce it.

[1] https://tracker.ceph.com/issues/55332 [2] https://tracker.ceph.com/issues/55421 [3] https://bugzilla.redhat.com/show_bug.cgi?id=2063929 [4] https://tracker.ceph.com/issues/55377 [5] https://github.com/ceph/ceph-client/commits/testing

Thanks

-- Xiubo

...

Gr. Stefan

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io