That's unfortunate. For one, it says "some filesystems are more equal than others". Back in ancient days, when dinosaurs used time-sharing, you could mount a remote fileystem at login and logout, whether explicit or timeout would unmount it. But virtually nobody time-shares anymore, but lots of people hibernate, either because they want to be more energy efficient or to prolong the life of a battery. And unlike a login, awaking from hibernation doesn't run a login script. Offhand, I'm not entirely sure if you can easily add any sort of wakeup script at the user level. My system turns off the office fan when it goes to sleep. The timeout interval is long enough that I'm probably no longer in the room at that point. In theory I could handle unmount/mount there, but I'm uncertain that I have the full range of system services at that level, and, as you pointed out, there might be open files that would expect to remain open when the machine came back up. Hence my wish that the ceph client and servers - especially linked OSD's - could "make their peace". which is to say quiesce their I/O, disconnect from the servers and possibly have Ceph mark the session as hibernating in order to come back up faster/more intelligently. Because it's a bloody royal PITA to have to manually mount Ceph every time I need to edit or run provisioning or use any of the other myriad systems that I keep in Ceph. Tim On 10/28/24 04:54, Burkhard Linke wrote:
Hi,
On 10/26/24 18:45, Tim Holloway wrote:
On the whole, I prefer to use NFS for my clients to use Ceph filesystem. It has the advantage that NFS client/mount is practically guaranteed to be pre-installed on all my client systems.
On the other hand, there are downsides. NFS (Ceph/NFS-Ganesha) has been known to be cranky on my network when it comes to serving resources on occasion, and should the NFS host go down, the NFS client cannot fallback to the alternate NFS host without assistance. Which I should probably add, since I run keepalive anyway, but that's another matter.
My issue, then is that if I do a native/FUSE mount of Ceph FS on my desktop it hurts Ceph. The desktop hibernates when not in use and that causes the network communications between it and the Ceph servers to break until it wakes up again. In the mean time, I get "laggy OSD" errors.
In extreme cases, where the desktop doesn't wake up clean and requires a hardware reset, I think it even ends up with orphan connections, since at one point I recall having to forcible terminate close to 20 connections.
Ideally, then, since systemd is handling hibernation, there should be a hook in the Ceph client to ensure it and the Ceph servers are at peace with each other while it's down. Probably better reporting on the Ceph admin side as to the hostnames of clients with connection issues, but that's just icing.
Do not use a network file system with hibernating machines. It will not work reliably.
Both NFS and especially CephFS can delegate responsibility to the clients. In case of NFSv4 these are the delegates, in case of CephFS it are cephfs capabilities. So the filesystem needs to have a back channel to the clients in case capabilities / delegates have to be revoked.
This won't work if the client is not reachable. Access to the files/directory protected by a capability might not be possible depending on the type of capability granted. E.g. in case of a write capability read access might be stalled; in case of a read capability any other client requesting write access will be blocked.
To make things worse, the cephfs capabilities also cover cached files. So even files not actively used by a client might be blocked....
A hook might reduce the impact of the problem, but it won't resolve it. At least the mountpoint itself will be as active capability on the client. This is the cap list of a client without any active file access and after dropping the caches:
# cat caps total 8193 avail 8192 used 1 reserved 0 min 8192
ino mds issued implemented -------------------------------------------------- 0x1 0 pAsLsXsFs pAsLsXsFs
Waiters: -------- tgid ino need want -----------------------------------------------------
(see /sys/kernel/debug/ceph/<fs id>.<client id>/caps)
TL;DR: do not use cephfs with hibernating clients.
We had the same problem with our desktops a while ago, and decided to switch to a NFS re-export of the CephFS filesystem. This has proven to be much more reliable in case of hibernation. But as you already mentioned NFS also has other problems...
Regards,
Burkhard Linke
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io