Back when I was battline Octopus, I had problems getting ganesha's NFS to work reliably. I resolved this by doing a direct (ceph) mount on my desktop machine instead of an NFS mount. I've since been plagued by ceph "laggy OSD" complaints that appear to be due to a non-responsive client and I'm suspecting that the client in question is the desktop machine when it's suspended while the ceph mount is in effect. So the question is: Should ceph native mounts be used on general client machines which may hibernate or otherwise go offline?
On Tue, Feb 6, 2024 at 12:09 PM Tim Holloway <timh@mousetech.com> wrote:
Back when I was battline Octopus, I had problems getting ganesha's NFS to work reliably. I resolved this by doing a direct (ceph) mount on my desktop machine instead of an NFS mount.
I've since been plagued by ceph "laggy OSD" complaints that appear to be due to a non-responsive client and I'm suspecting that the client in question is the desktop machine when it's suspended while the ceph mount is in effect.
You should not see "laggy OSD" messages due to a client becoming unresponsive.
So the question is: Should ceph native mounts be used on general client machines which may hibernate or otherwise go offline?
The mounts will eventually be evicted (generally) by the MDS if the machine hibernates/suspends. There are mechanisms for the mount to recover (see "recover_session" in the mount.ceph man page). Any dirty data would be lost. As for whether you should have clients that hibernate, it's not ideal. It could conceivably create problems if client machines hibernate longer than the blocklist duration (after eviction by the MDS). -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Unfortunately, it looks like the exact error text has already rolled off my logs. Earlier today something jammed up and I restarted ceph, which returned everything to a healthy state. But I'll take that as an indicator that ceph isn't a good match for sleeper systems. Fortunately, ganesha NFS is now working properly for me since I upgraded Octopus to Pacific and I no longer feel the need for a direct mount. NFS has had to deal with hibernation for years, so I think I can depend on it to deal with it one way or the other. Tim On Tue, 2024-02-06 at 13:00 -0500, Patrick Donnelly wrote:
On Tue, Feb 6, 2024 at 12:09 PM Tim Holloway <timh@mousetech.com> wrote:
Back when I was battline Octopus, I had problems getting ganesha's NFS to work reliably. I resolved this by doing a direct (ceph) mount on my desktop machine instead of an NFS mount.
I've since been plagued by ceph "laggy OSD" complaints that appear to be due to a non-responsive client and I'm suspecting that the client in question is the desktop machine when it's suspended while the ceph mount is in effect.
You should not see "laggy OSD" messages due to a client becoming unresponsive.
So the question is: Should ceph native mounts be used on general client machines which may hibernate or otherwise go offline?
The mounts will eventually be evicted (generally) by the MDS if the machine hibernates/suspends. There are mechanisms for the mount to recover (see "recover_session" in the mount.ceph man page). Any dirty data would be lost.
As for whether you should have clients that hibernate, it's not ideal. It could conceivably create problems if client machines hibernate longer than the blocklist duration (after eviction by the MDS).
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Followup. Desktop system went to sleep overnight. I woke up to this: HEALTH_WARN 1 client(s) laggy due to laggy OSDs; 1 clients failing to respond to capability release; 1 MDSs report slow requests [WRN] MDS_CLIENTS_LAGGY: 1 client(s) laggy due to laggy OSDs mds.ceefs.www2.lzjqgd(mds.0): Client 7513643 is laggy; not evicted because some OSD(s) is/are laggy [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.ceefs.www2.lzjqgd(mds.0): Client a64.mousetech.com: failing to respond to capability release client_id: 7513643 [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.ceefs.www2.lzjqgd(mds.0): 1 slow requests are blocked > 30 secs a64 is the sleeping desktop. I restarted the the ceph target and a whole bunch of stuck pg's popped up, but the whole w2 machine is apparently scrambled and even a reboot hasn't made its processes happy so I'm going to have to go in and fix them one by one. May be unrelated to the original laggy OSD problem, though. Just wanted to mention it for completeness. Also, the desktop has weirdly managed to mount both native ceph and the ceph NFS at the same mount point. Most likely because I'd added a pre/post sleep script to unmount/remount ceph when the system went into or out of suspension and I lost track of it so I haven't fixed it. Again, this is mostly just for background not related to ceph's own internals, although I had wondered if the ceph mount was happening slowly and the suspension wasn't waiting properly for it to complete. On Tue, 2024-02-06 at 13:00 -0500, Patrick Donnelly wrote:
On Tue, Feb 6, 2024 at 12:09 PM Tim Holloway <timh@mousetech.com> wrote:
Back when I was battline Octopus, I had problems getting ganesha's NFS to work reliably. I resolved this by doing a direct (ceph) mount on my desktop machine instead of an NFS mount.
I've since been plagued by ceph "laggy OSD" complaints that appear to be due to a non-responsive client and I'm suspecting that the client in question is the desktop machine when it's suspended while the ceph mount is in effect.
You should not see "laggy OSD" messages due to a client becoming unresponsive.
So the question is: Should ceph native mounts be used on general client machines which may hibernate or otherwise go offline?
The mounts will eventually be evicted (generally) by the MDS if the machine hibernates/suspends. There are mechanisms for the mount to recover (see "recover_session" in the mount.ceph man page). Any dirty data would be lost.
As for whether you should have clients that hibernate, it's not ideal. It could conceivably create problems if client machines hibernate longer than the blocklist duration (after eviction by the MDS).
participants (2)
-
Patrick Donnelly
-
Tim Holloway