On Tue, Mar 26, 2024 at 7:30 PM Yongseok Oh <yongseok.oh@linecorp.com> wrote:
Hi,
CephFS is provided as a shared file system service in a private cloud environment of our company, LINE. The number of sessions is approximately more than 5,000, and session evictions occur several times a day. When session eviction occurs, the message 'Cannot send after transport endpoint shutdown' or 'Permission denied' is displayed and file system access is not possible. Our users are very uncomfortable with this issue. In particular, there are no special problems such as network connection or CPU usage. When I access the machine and take a close look, there are no special problems. After this, users feel the inconvenience of having to perform umount/mount tasks and run the application again. In a Kubernetes environment, recovery is a bit more complicated, which causes a lot of frustration.
I tested that by setting the mds_session_blocklist_on_timeout and mds_session_blocklist_on_evict options to false and setting client_reconnect_stale to true on the client side, the file system can be accessed even if eviction occurs. It seemed like there was no major problem accessing the file system as the session was still attached.
What I'm curious about is if I turn on the above option, will there be any other side effects? For example, should I take some action if the integrity of the file is broken or if there is an issue on the mds side? I am asking a question because there are no details regarding this in the official CephFS documentation.
The suggestion you are following is documented here https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-configuring-blockl... As you mentioned rightly, it doesn't talk about the side effects if any. But that said, it's kind of a workaround even if it works. Do we know the exact reason for session eviction from the logs ? It could be any of these things mentioned here https://docs.ceph.com/en/latest/cephfs/eviction/#automatic-client-eviction
Thank you
Yongseok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io