On 3/2/21 5:16 PM, Jeff Layton wrote:
On Tue, 2021-03-02 at 09:25 +0100, Stefan Kooman wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with
I'm guessing this is a stable series kernel
It's a kernel from 'elrepo', so I'm not sure. I would guess so too.
Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22
-22 == -EINVAL
Looks like a an osdmap parsing error?
Indeed, that is kinda weird isn't it.
Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
The mount helper only recently got v2 support, and that hasn't trickled out into the distros yet. See:
Ah, good to know.
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
What mount options are you passing in? Are you using mon autodiscovery?
We provide the mon address explictly in the /etc/fstab, So something like this: mon1,mon2,mon3,mon4,mon5:/cephfs/dir /client_mountpoint ceph name=client-id,secretfile=/etc/ceph/ceph.client.client-id.cephfs.key,noatime,_netdev 0 2 We are not using any dns based discovery of monitors if that is what you mean. Note: We tried with nautilus packages before (14.2.16) and got the same result.
v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid argument", the ms_mode=legacy does work, but fails with the same errors. Gr. Stefan