CephFS clients waiting for lock when one of them goes slow
Hi all, I have a very small Ceph setup - 3 OSDs, 3 MDS, 3 MON, CephFS with ~20 ceph-fuse clients connected. All run version 14.2.9 (both servers and clients). Some clients work with a relatively large CephFS directory, few tens of thousands of files, with lots of files being added and deleted all the time. Almost every day I see exactly the same pattern when system goes under load: - One of the clients starts running slow (e.g. high CPU or out-of-memory, sometimes with OOM killer intervention) - A number of requests from other clients gets blocked for ~15 seconds waiting for the (r/w/x)lock on other files in the same directory - Objecter requests empty - No requests visibly running in the dump_ops_in_flight output, only those waiting for the lock MDS cache size is 1.5 gb, total capacity - around 1TB, usual load is ~300 [rw]ops, 3-4 mb/s, most of the config is the default one. Have somebody seen similar issues before? Best regards, Petr Belyaev
Hi, 1,5 GB MDS cache for 20 clients is not enough, I would start with increasing that to 4 or 8 GB and see if the problems still occur. Regards, Eugen Zitat von Petr Belyaev <p.belyaev@alohi.com>:
Hi all,
I have a very small Ceph setup - 3 OSDs, 3 MDS, 3 MON, CephFS with ~20 ceph-fuse clients connected. All run version 14.2.9 (both servers and clients).
Some clients work with a relatively large CephFS directory, few tens of thousands of files, with lots of files being added and deleted all the time. Almost every day I see exactly the same pattern when system goes under load:
- One of the clients starts running slow (e.g. high CPU or out-of-memory, sometimes with OOM killer intervention) - A number of requests from other clients gets blocked for ~15 seconds waiting for the (r/w/x)lock on other files in the same directory - Objecter requests empty - No requests visibly running in the dump_ops_in_flight output, only those waiting for the lock
MDS cache size is 1.5 gb, total capacity - around 1TB, usual load is ~300 [rw]ops, 3-4 mb/s, most of the config is the default one.
Have somebody seen similar issues before?
Best regards, Petr Belyaev _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, I've seen this problem quite frequently as of late, running v13.2.10 MDS. It seems to be dependent on the client behavior - a lot of xlock contention on some directory, although it's hard to pin down which client is doing what. The only remedy was to fail over the MDS. 1k - 4k clients 2M requests/replies (not sure what the window is) 40GB of MDS cache 1 active MDS, 1 standby-replay Something tells me that I need multi-active MDS setup, but every MDS crash in that set up results in clearing the journal. It could be that this cluster was upgraded from older releases of Ceph and years of clearing the journal has led to unrecoverable damage. For now we're hanging on with 1 active MDS, and there's plans to move to radosgw.
On Sat, Aug 15, 2020 at 12:32 AM <pchoi@nuro.ai> wrote:
Yes, I've seen this problem quite frequently as of late, running v13.2.10 MDS. It seems to be dependent on the client behavior - a lot of xlock contention on some directory, although it's hard to pin down which client is doing what. The only remedy was to fail over the MDS.
1k - 4k clients 2M requests/replies (not sure what the window is) 40GB of MDS cache 1 active MDS, 1 standby-replay
Something tells me that I need multi-active MDS setup, but every MDS crash in that set up results in clearing the journal. It could be that this cluster was upgraded from older releases of Ceph and years of clearing the journal has led to unrecoverable damage.
backtrace of the crash?
For now we're hanging on with 1 active MDS, and there's plans to move to radosgw. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
pchoi＠nuro.ai
-
Petr Belyaev
-
Yan, Zheng