kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)
22 Dec
2020
22 Dec
'20
4:37 p.m.
I have a vm on a osd node (which can reach host and other nodes via the
macvtap interface (used by the host and guest)). I just did a simple
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an
archive namespace, inbox is on rbd ssd, fs meta also on ssd)
How can I recover from such lock-up. If I have a similar situation with
an nfs-ganesha mount, I have the option to do a umount -l, and clients
recover quickly without any issues.
Having to reset the vm, is not really an option. What is best way to
resolve this?
Ceph cluster: 14.2.11 (the vm has 14.2.16)
I have in my ceph.conf nothing special, these 2x in the mds section:
mds bal fragment size max = 120000
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387
All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
22 Dec
22 Dec
4:47 p.m.
Just got this during bonnie test, trying to do an ls -l on the cephfs. I
also have this kworker process constantly at 40% when doing this
bonnie++ test.
[35281.101763] INFO: task bash:1169 blocked for more than 120 seconds.
[35281.102064] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[35281.102175] bash D ffffa03fbfc9acc0 0 1169 1167
0x00000004
[35281.102181] Call Trace:
[35281.102275] [<ffffffff84b86d4f>] ? __schedule+0x3af/0x860
[35281.102285] [<ffffffff84b87229>] schedule+0x29/0x70
[35281.102296] [<ffffffff84b84d11>] schedule_timeout+0x221/0x2d0
[35281.102332] [<ffffffff844c6966>] ? finish_wait+0x56/0x70
[35281.102342] [<ffffffff84b85482>] ? mutex_lock+0x12/0x2f
[35281.102381] [<ffffffff846e7ed8>] ? autofs4_wait+0x428/0x920
[35281.102386] [<ffffffff84b875dd>] wait_for_completion+0xfd/0x140
[35281.102407] [<ffffffff844daf40>] ? wake_up_state+0x20/0x20
[35281.102422] [<ffffffff846e902b>] autofs4_expire_wait+0xab/0x160
[35281.102425] [<ffffffff846e6060>] do_expire_wait+0x1e0/0x210
[35281.102429] [<ffffffff846e62b3>] autofs4_d_manage+0x73/0x1c0
[35281.102455] [<ffffffff84658e8a>] follow_managed+0xba/0x310
[35281.102459] [<ffffffff84659e5d>] lookup_fast+0x12d/0x230
[35281.102464] [<ffffffff8465c90d>] path_lookupat+0x16d/0x8d0
[35281.102467] [<ffffffff8465deed>] ? do_last+0x66d/0x1340
[35281.102488] [<ffffffff8464a73a>] ? __check_object_size+0x1ca/0x250
[35281.102499] [<ffffffff84628675>] ? kmem_cache_alloc+0x35/0x1f0
[35281.102503] [<ffffffff8465fc0f>] ? getname_flags+0x4f/0x1a0
[35281.102507] [<ffffffff8465d09b>] filename_lookup+0x2b/0xc0
[35281.102510] [<ffffffff84660da7>] user_path_at_empty+0x67/0xc0
[35281.102513] [<ffffffff84660e11>] user_path_at+0x11/0x20
[35281.102516] [<ffffffff84653603>] vfs_fstatat+0x63/0xc0
[35281.102519] [<ffffffff846539be>] SYSC_newstat+0x2e/0x60
[35281.102529] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
[35281.102533] [<ffffffff84b94ec9>] ?
system_call_after_swapgs+0x96/0x13a
[35281.102536] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
[35281.102539] [<ffffffff84b94ec9>] ?
system_call_after_swapgs+0x96/0x13a
[35281.102543] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
[35281.102546] [<ffffffff84b94ec9>] ?
system_call_after_swapgs+0x96/0x13a
[35281.102549] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
[35281.102552] [<ffffffff84b94ec9>] ?
system_call_after_swapgs+0x96/0x13a
[35281.102555] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
[35281.102558] [<ffffffff84b94ec9>] ?
system_call_after_swapgs+0x96/0x13a
[35281.102561] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
[35281.102565] [<ffffffff84653e7e>] SyS_newstat+0xe/0x10
[35281.102568] [<ffffffff84b94f92>] system_call_fastpath+0x25/0x2a
[35281.102572] [<ffffffff84b94ed5>] ?
system_call_after_swapgs+0xa2/0x13a
-----Original Message-----
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something
like umount -l available?) (help wanted going to production)
I have a vm on a osd node (which can reach host and other nodes via the
macvtap interface (used by the host and guest)). I just did a simple
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an
archive namespace, inbox is on rbd ssd, fs meta also on ssd)
How can I recover from such lock-up. If I have a similar situation with
an nfs-ganesha mount, I have the option to do a umount -l, and clients
recover quickly without any issues.
Having to reset the vm, is not really an option. What is best way to
resolve this?
Ceph cluster: 14.2.11 (the vm has 14.2.16)
I have in my ceph.conf nothing special, these 2x in the mds section:
mds bal fragment size max = 120000
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387
All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:53 p.m.
I can live-migrate the vm in this locked up state to a different host
without any problems.
-----Original Message-----
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something
like umount -l available?) (help wanted going to production)
I have a vm on a osd node (which can reach host and other nodes via the
macvtap interface (used by the host and guest)). I just did a simple
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an
archive namespace, inbox is on rbd ssd, fs meta also on ssd)
How can I recover from such lock-up. If I have a similar situation with
an nfs-ganesha mount, I have the option to do a umount -l, and clients
recover quickly without any issues.
Having to reset the vm, is not really an option. What is best way to
resolve this?
Ceph cluster: 14.2.11 (the vm has 14.2.16)
I have in my ceph.conf nothing special, these 2x in the mds section:
mds bal fragment size max = 120000
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387
All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:03 p.m.
Is there not some genius out there that can shed a ligth on this? ;)
Currently I am not able to reproduce this. Thus it would be nice to have
some procedure at hand that resolves stale cephfs mounts nicely.
-----Original Message-----
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something
like umount -l available?) (help wanted going to production)
I have a vm on a osd node (which can reach host and other nodes via the
macvtap interface (used by the host and guest)). I just did a simple
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an
archive namespace, inbox is on rbd ssd, fs meta also on ssd)
How can I recover from such lock-up. If I have a similar situation with
an nfs-ganesha mount, I have the option to do a umount -l, and clients
recover quickly without any issues.
Having to reset the vm, is not really an option. What is best way to
resolve this?
Ceph cluster: 14.2.11 (the vm has 14.2.16)
I have in my ceph.conf nothing special, these 2x in the mds section:
mds bal fragment size max = 120000
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387
All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:49 p.m.
Hi,
there have been several threads about hanging cephfs mounts, one quite
long thread [1] describes a couple of debugging options but also
mentions to avoid mounting cephfs on OSD nodes in a production
environment.
Do you see blacklisted clients with 'ceph osd blacklist ls'? If the
answer is yes try to unblock that client [2].
The same option ('umount -l') is available on a cephfs client, you can
try that, too. Other options described in [1] are to execute an MDS
failover, but sometimes a reboot of that VM is the only solution left.
Regards,
Eugen
[1]
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028719.html
[2]
https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-un-blocklisting-a…
Zitat von Marc Roos <M.Roos(a)f1-outsourcing.eu>eu>:
Is there not some genius out there that can shed a
ligth on this? ;)
Currently I am not able to reproduce this. Thus it would be nice to have
some procedure at hand that resolves stale cephfs mounts nicely.
-----Original Message-----
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something
like umount -l available?) (help wanted going to production)
I have a vm on a osd node (which can reach host and other nodes via the
macvtap interface (used by the host and guest)). I just did a simple
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an
archive namespace, inbox is on rbd ssd, fs meta also on ssd)
How can I recover from such lock-up. If I have a similar situation with
an nfs-ganesha mount, I have the option to do a umount -l, and clients
recover quickly without any issues.
Having to reset the vm, is not really an option. What is best way to
resolve this?
Ceph cluster: 14.2.11 (the vm has 14.2.16)
I have in my ceph.conf nothing special, these 2x in the mds section:
mds bal fragment size max = 120000
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387
All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1228
days inactive
1228
days old
4 comments
2 participants
participants (2)
-
Eugen Block -
Marc Roos