Can setting mds_session_blocklist_on_timeout to false minize the session eviction?
Hi, CephFS is provided as a shared file system service in a private cloud environment of our company, LINE. The number of sessions is approximately more than 5,000, and session evictions occur several times a day. When session eviction occurs, the message 'Cannot send after transport endpoint shutdown' or 'Permission denied' is displayed and file system access is not possible. Our users are very uncomfortable with this issue. In particular, there are no special problems such as network connection or CPU usage. When I access the machine and take a close look, there are no special problems. After this, users feel the inconvenience of having to perform umount/mount tasks and run the application again. In a Kubernetes environment, recovery is a bit more complicated, which causes a lot of frustration. I tested that by setting the mds_session_blocklist_on_timeout and mds_session_blocklist_on_evict options to false and setting client_reconnect_stale to true on the client side, the file system can be accessed even if eviction occurs. It seemed like there was no major problem accessing the file system as the session was still attached. What I'm curious about is if I turn on the above option, will there be any other side effects? For example, should I take some action if the integrity of the file is broken or if there is an issue on the mds side? I am asking a question because there are no details regarding this in the official CephFS documentation. Thank you Yongseok
On Tue, Mar 26, 2024 at 7:30 PM Yongseok Oh <yongseok.oh@linecorp.com> wrote:
Hi,
CephFS is provided as a shared file system service in a private cloud environment of our company, LINE. The number of sessions is approximately more than 5,000, and session evictions occur several times a day. When session eviction occurs, the message 'Cannot send after transport endpoint shutdown' or 'Permission denied' is displayed and file system access is not possible. Our users are very uncomfortable with this issue. In particular, there are no special problems such as network connection or CPU usage. When I access the machine and take a close look, there are no special problems. After this, users feel the inconvenience of having to perform umount/mount tasks and run the application again. In a Kubernetes environment, recovery is a bit more complicated, which causes a lot of frustration.
I tested that by setting the mds_session_blocklist_on_timeout and mds_session_blocklist_on_evict options to false and setting client_reconnect_stale to true on the client side, the file system can be accessed even if eviction occurs. It seemed like there was no major problem accessing the file system as the session was still attached.
What I'm curious about is if I turn on the above option, will there be any other side effects? For example, should I take some action if the integrity of the file is broken or if there is an issue on the mds side? I am asking a question because there are no details regarding this in the official CephFS documentation.
The suggestion you are following is documented here https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-configuring-blockl... As you mentioned rightly, it doesn't talk about the side effects if any. But that said, it's kind of a workaround even if it works. Do we know the exact reason for session eviction from the logs ? It could be any of these things mentioned here https://docs.ceph.com/en/latest/cephfs/eviction/#automatic-client-eviction
Thank you
Yongseok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The solution can be found here. https://ceph.io/en/news/blog/2020/automatic-cephfs-recovery-after-blacklisti... When a session eviction occurs on the client side and then the recovery is performed, the client's caps and dirty data are dropped and the server also releases the client's caps, so there is no problem with metadata consistency. Is there a reason why the auto recovery option is not turned on by default? We are conducting various tests to turn on the auto recovery option by default for the storage we operate.
participants (2)
-
Kotresh Hiremath Ravishankar
-
Yongseok Oh