cephfs: unable to mount share with 5.11 mainline, ceph 15.2.9, MDS 14.1.16
Hi, On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22 Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported). Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port? Thanks, Stefan
On Tue, 2021-03-02 at 09:25 +0100, Stefan Kooman wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with
I'm guessing this is a stable series kernel
Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22
-22 == -EINVAL Looks like a an osdmap parsing error?
Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
The mount helper only recently got v2 support, and that hasn't trickled out into the distros yet. See: https://github.com/ceph/ceph/pull/38788
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
What mount options are you passing in? Are you using mon autodiscovery? v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2. -- Jeff Layton <jlayton@redhat.com>
On 3/2/21 5:16 PM, Jeff Layton wrote:
On Tue, 2021-03-02 at 09:25 +0100, Stefan Kooman wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with
I'm guessing this is a stable series kernel
It's a kernel from 'elrepo', so I'm not sure. I would guess so too.
Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22
-22 == -EINVAL
Looks like a an osdmap parsing error?
Indeed, that is kinda weird isn't it.
Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
The mount helper only recently got v2 support, and that hasn't trickled out into the distros yet. See:
Ah, good to know.
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
What mount options are you passing in? Are you using mon autodiscovery?
We provide the mon address explictly in the /etc/fstab, So something like this: mon1,mon2,mon3,mon4,mon5:/cephfs/dir /client_mountpoint ceph name=client-id,secretfile=/etc/ceph/ceph.client.client-id.cephfs.key,noatime,_netdev 0 2 We are not using any dns based discovery of monitors if that is what you mean. Note: We tried with nautilus packages before (14.2.16) and got the same result.
v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid argument", the ms_mode=legacy does work, but fails with the same errors. Gr. Stefan
On Tue, 2021-03-02 at 17:44 +0100, Stefan Kooman wrote:
On 3/2/21 5:16 PM, Jeff Layton wrote:
On Tue, 2021-03-02 at 09:25 +0100, Stefan Kooman wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with
I'm guessing this is a stable series kernel
It's a kernel from 'elrepo', so I'm not sure. I would guess so too.
Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22
-22 == -EINVAL
Looks like a an osdmap parsing error?
Indeed, that is kinda weird isn't it.
Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
The mount helper only recently got v2 support, and that hasn't trickled out into the distros yet. See:
Ah, good to know.
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
What mount options are you passing in? Are you using mon autodiscovery?
We provide the mon address explictly in the /etc/fstab, So something like this:
mon1,mon2,mon3,mon4,mon5:/cephfs/dir /client_mountpoint ceph name=client-id,secretfile=/etc/ceph/ceph.client.client-id.cephfs.key,noatime,_netdev 0 2
We are not using any dns based discovery of monitors if that is what you mean.
Note: We tried with nautilus packages before (14.2.16) and got the same result.
v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid argument", the ms_mode=legacy does work, but fails with the same errors.
That needs different values. See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... You can try passing in a specific mon address and port, like: 192.168.21.22:3300:/cephfs/dir/ ...and then pass in ms_mode=crc or something similar. That said, what you're doing should be working, so this sounds like a regression. I presume you're able to mount with earlier kernels? What's the latest kernel version that you have that works? -- Jeff Layton <jlayton@redhat.com>
On 3/2/21 6:00 PM, Jeff Layton wrote:
v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid argument", the ms_mode=legacy does work, but fails with the same errors.
That needs different values. See:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
You can try passing in a specific mon address and port, like:
192.168.21.22:3300:/cephfs/dir/
...and then pass in ms_mode=crc or something similar.
That works, as in I don't get a mount error (ms_mode=prefer-crc) and added the port 3300 explictly, but same error.
That said, what you're doing should be working, so this sounds like a regression. I presume you're able to mount with earlier kernels? What's the latest kernel version that you have that works?
Previous one was 4.18 ... elrepo only has 5.4 / 5.11 available now AFAIK. I'll try to test some ubuntu kernel ppa's as I can choose what version to use. I'll keep you posted. Gr. Stefan
On 3/2/21 6:00 PM, Jeff Layton wrote:
v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid argument", the ms_mode=legacy does work, but fails with the same errors.
That needs different values. See:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
You can try passing in a specific mon address and port, like:
192.168.21.22:3300:/cephfs/dir/
...and then pass in ms_mode=crc or something similar.
That said, what you're doing should be working, so this sounds like a regression. I presume you're able to mount with earlier kernels? What's the latest kernel version that you have that works?
5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) with a cluster that has ms_bind_ipv4=false works. Port 3300 ms_mode=prefer-crc and ms_mode=crc work. I have tested with 5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) port 3300 and ms_mode=crc as well as ms_mode=prefer-crc and that works when cluster is running with ms_bind_ipv4=false. So the "fix" is to have this config option set: ceph config set global ms_bind_ipv4 false 5.10 kernel (5.10.19-1-lts Arch Linux) works with a cluster that is IPv6 only but has ms_bind_ipv4=true. So it's "broken" since 5.11. So, we have done more reading / researching on the ms_bind_ip{4,6} options: - https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus#Restart_the_OSD_daemo... - https://github.com/rook/rook/issues/6266 ^^ Describe that you have to disable bind to IPv4. - https://github.com/ceph/ceph/pull/13317 ^^ this PR is not completely correct: **Note:** You may use IPv6 addresses instead of IPv4 addresses, but you must set ``ms bind ipv6`` to ``true``. ^^ That is not enough as we have learned, and starts to give trouble with 5.11 linux cephfs client. And from this documentation: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4... we learned that dual stack is not possible for any current stable release, but might be possible with latest code. So the takeaway is that the linux kernel client needs fixing to be able to support dual stack clusters in the future (multiple v1 / v2 address families), and, that until then you should run with ms_bind_ipv4=false for IPv6 only clusters. I'll make a PR to clear up the documenation. Do you want me to create a tracker for the kernel client? I will happily test your changes. Thanks, Stefan
On Wed, Mar 3, 2021 at 11:15 AM Stefan Kooman <stefan@bit.nl> wrote:
On 3/2/21 6:00 PM, Jeff Layton wrote:
v2 support in the kernel is keyed on the ms_mode= mount option, so that has to be passed in if you're connecting to a v2 port. Until the mount helpers get support for that option you'll need to specify the address and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid argument", the ms_mode=legacy does work, but fails with the same errors.
That needs different values. See:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
You can try passing in a specific mon address and port, like:
192.168.21.22:3300:/cephfs/dir/
...and then pass in ms_mode=crc or something similar.
That said, what you're doing should be working, so this sounds like a regression. I presume you're able to mount with earlier kernels? What's the latest kernel version that you have that works?
5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) with a cluster that has ms_bind_ipv4=false works. Port 3300 ms_mode=prefer-crc and ms_mode=crc work.
I have tested with 5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) port 3300 and ms_mode=crc as well as ms_mode=prefer-crc and that works when cluster is running with ms_bind_ipv4=false. So the "fix" is to have this config option set: ceph config set global ms_bind_ipv4 false
Right. According to your original post that was already the case: "ms_bind_ipv6=trie, ms_bind_ipv4=false".
5.10 kernel (5.10.19-1-lts Arch Linux) works with a cluster that is IPv6 only but has ms_bind_ipv4=true. So it's "broken" since 5.11.
So, we have done more reading / researching on the ms_bind_ip{4,6} options:
- https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus#Restart_the_OSD_daemo...
- https://github.com/rook/rook/issues/6266
^^ Describe that you have to disable bind to IPv4.
- https://github.com/ceph/ceph/pull/13317
^^ this PR is not completely correct:
**Note:** You may use IPv6 addresses instead of IPv4 addresses, but you must set ``ms bind ipv6`` to ``true``.
^^ That is not enough as we have learned, and starts to give trouble with 5.11 linux cephfs client.
And from this documentation: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4... we learned that dual stack is not possible for any current stable release, but might be possible with latest code. So the takeaway is that the linux kernel client needs fixing to be able to support dual stack clusters in the future (multiple v1 / v2 address families), and, that until then you should run with ms_bind_ipv4=false for IPv6 only clusters.
I don't think we do any dual stack testing, whether in userspace or (certainly!) with the kernel client.
I'll make a PR to clear up the documenation. Do you want me to create a tracker for the kernel client? I will happily test your changes.
Sure. You are correct that the kernel client needs a bit a work as we haven't considered dual stack configurations there at all. Thanks, Ilya
On 3/3/21 1:16 PM, Ilya Dryomov wrote:
I have tested with 5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) port 3300 and ms_mode=crc as well as ms_mode=prefer-crc and that works when cluster is running with ms_bind_ipv4=false. So the "fix" is to have this config option set: ceph config set global ms_bind_ipv4 false
Right. According to your original post that was already the case: "ms_bind_ipv6=trie, ms_bind_ipv4=false".
Indeed, I wrote that. That was not correct. We *did* have set that on a test cluster, but those changes have never propagated to production.
And from this documentation: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4... we learned that dual stack is not possible for any current stable release, but might be possible with latest code. So the takeaway is that the linux kernel client needs fixing to be able to support dual stack clusters in the future (multiple v1 / v2 address families), and, that until then you should run with ms_bind_ipv4=false for IPv6 only clusters.
I don't think we do any dual stack testing, whether in userspace or (certainly!) with the kernel client.
Yeah, there are also quite some combinations possible, especially with dual stack and separate public / cluster networks, even with different IP stack maybe? Would be good to know (as in the broader community) what is being tested / supported with regards to networking. Nowadays separate public/cluster is not advised anymore (which is a good thing, thanks Wido) but maybe this should also be made clear for IP families. IMHO it would be good to test both IPv4 and IPv6.
I'll make a PR to clear up the documenation. Do you want me to create a tracker for the kernel client? I will happily test your changes.
Sure. You are correct that the kernel client needs a bit a work as we haven't considered dual stack configurations there at all.
Check, I'll do that and come back with a tracker ID. Thanks, Stefan
On 3/3/21 1:16 PM, Ilya Dryomov wrote:
Sure. You are correct that the kernel client needs a bit a work as we haven't considered dual stack configurations there at all.
https://tracker.ceph.com/issues/49581 Gr. Stefan
On 3/3/21 1:16 PM, Ilya Dryomov wrote:
And from this documentation: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4... we learned that dual stack is not possible for any current stable release, but might be possible with latest code. So the takeaway is that the linux kernel client needs fixing to be able to support dual stack clusters in the future (multiple v1 / v2 address families), and, that until then you should run with ms_bind_ipv4=false for IPv6 only clusters.
I don't think we do any dual stack testing, whether in userspace or (certainly!) with the kernel client.
I'll make a PR to clear up the documenation. Do you want me to create a tracker for the kernel client? I will happily test your changes.
Sure. You are correct that the kernel client needs a bit a work as we haven't considered dual stack configurations there at all.
I added another tracker as it is related to this thread: https://tracker.ceph.com/issues/49584 ^^^ do _not_ try this on a IPv6 only production cluster with ms_bind_ipv4=true or you will regret it ;-). Gr. Stefan
On Tue, Mar 2, 2021 at 9:26 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22 Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
Hi Stefan, Those "another match of type 1" errors suggest that you have two different v1 addresses for some of or all OSDs and MDSes in osdmap and mdsmap respectively. What is the output of "ceph osd dump" and "ceph fs dump"? Thanks, Ilya
On 3/2/21 5:42 PM, Ilya Dryomov wrote:
On Tue, Mar 2, 2021 at 9:26 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22 Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
Hi Stefan,
Those "another match of type 1" errors suggest that you have two different v1 addresses for some of or all OSDs and MDSes in osdmap and mdsmap respectively.
What is the output of "ceph osd dump" and "ceph fs dump"?
That's a lot of output, so I trimmed it: --- snip --- osd.0 up in weight 1 up_from 98071 up_thru 98719 down_at 98068 last_clean_interval [96047,98067) [v2:[2001:7b8:80:1:0:1:2:1]:6848/505534,v1:[2001:7b8:80:1:0:1:2:1]:6854/505534,v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534] [v2:[2001:7b8:80:1:0:1:2:1]:6872/505534,v1:[2001:7b8:80:1:0:1:2:1]:6878/505534,v2:0.0.0.0:6886/505534,v1:0.0.0.0:6892/505534] exists,up 93e7d17f-2c7a-4acd-93c0-586dbb7cc6d7 -- snap --- -- snip --- [mds.mds1{0:229930080} state up:active seq 144042 addr [v2:[2001:7b8:80:1:0:1:3:1]:6800/2234186180,v1:[2001:7b8:80:1:0:1:3:1]:6801/2234186180,v2:0.0.0.0:6802/2234186180,v1:0.0.0.0:6803/2234186180]] Standby daemons: [mds.mds2{-1:229977514} state up:standby seq 2 addr [v2:[2001:7b8:80:3:0:2c:3:2]:6800/2983725953,v1:[2001:7b8:80:3:0:2c:3:2]:6801/2983725953,v2:0.0.0.0:6802/2983725953,v1:0.0.0.0:6803/2983725953]] --- snap --- We only have a public address network in use, and IPv6 only, So not sure how we could have multiple v1 addresses for OSDs and MDSes. Thanks, Stefan
On Tue, Mar 2, 2021 at 6:02 PM Stefan Kooman <stefan@bit.nl> wrote:
On 3/2/21 5:42 PM, Ilya Dryomov wrote:
On Tue, Mar 2, 2021 at 9:26 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22 Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
Hi Stefan,
Those "another match of type 1" errors suggest that you have two different v1 addresses for some of or all OSDs and MDSes in osdmap and mdsmap respectively.
What is the output of "ceph osd dump" and "ceph fs dump"?
That's a lot of output, so I trimmed it:
--- snip --- osd.0 up in weight 1 up_from 98071 up_thru 98719 down_at 98068 last_clean_interval [96047,98067) [v2:[2001:7b8:80:1:0:1:2:1]:6848/505534,v1:[2001:7b8:80:1:0:1:2:1]:6854/505534,v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534]
Where did "v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534" come from? This is what confuses the kernel client: it sees two addresses of the same type and doesn't know which one to pick. Instead of blindly picking the first one (or some other dubious heuristic) it just denies the osdmap.
[mds.mds1{0:229930080} state up:active seq 144042 addr [v2:[2001:7b8:80:1:0:1:3:1]:6800/2234186180,v1:[2001:7b8:80:1:0:1:3:1]:6801/2234186180,v2:0.0.0.0:6802/2234186180,v1:0.0.0.0:6803/2234186180]]
Same for the mdsmap. Were you using ipv6 with the kernel client before upgrading to 5.11? What is output of "ceph daemon osd.0 config get ms_bind_ipv4" on the osd0 node? Thanks, Ilya
On 3/2/21 6:54 PM, Ilya Dryomov wrote:
--- snip --- osd.0 up in weight 1 up_from 98071 up_thru 98719 down_at 98068 last_clean_interval [96047,98067) [v2:[2001:7b8:80:1:0:1:2:1]:6848/505534,v1:[2001:7b8:80:1:0:1:2:1]:6854/505534,v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534]
Where did "v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534" come from? This is what confuses the kernel client: it sees two addresses of the same type and doesn't know which one to pick. Instead of blindly picking the first one (or some other dubious heuristic) it just denies the osdmap.
[mds.mds1{0:229930080} state up:active seq 144042 addr [v2:[2001:7b8:80:1:0:1:3:1]:6800/2234186180,v1:[2001:7b8:80:1:0:1:3:1]:6801/2234186180,v2:0.0.0.0:6802/2234186180,v1:0.0.0.0:6803/2234186180]]
Same for the mdsmap.
Were you using ipv6 with the kernel client before upgrading to 5.11?
Yes, exclusively. So dual stack was not something that was possible before nautilus. We always did set "ms_bind_ipv6=true". But good find, I missed that "0.0.0.0" part. Yeah, that must be it.
What is output of "ceph daemon osd.0 config get ms_bind_ipv4" on the osd0 node?
ceph daemon osd.0 config get ms_bind_ipv4 { "ms_bind_ipv4": "true" } And ceph daemon mds.mds1 config get ms_bind_ipv4 { "ms_bind_ipv4": "true" } for that matter. Now I'm typing this I'm like, hmm, there was an issue with this on the mailinglist related to this, as in where you would need to disable ipv4 explicitly. I believe it was a peering issue between PGs. OK, so this is probably it. And up to kernel 5.11 (I'll test 5.10 as well) this would not be a problem. I should be able te reproduce on a couple of test clusters that are configured similarly, and be able to test if setting ms_bind_ipv4=false on OSDs / MDSs fixes the issue. I would like to have the heuristic to prefer IPv6 over IPv4 when filtering addresses (as that is default / common behavior for most if not all dual stack systems) ;-). I'll do testing and let you know. Gr. Stefan
On 3/2/21 7:17 PM, Stefan Kooman wrote:
What is output of "ceph daemon osd.0 config get ms_bind_ipv4" on the osd0 node?
ceph daemon osd.0 config get ms_bind_ipv4 { "ms_bind_ipv4": "true" }
And
ceph daemon mds.mds1 config get ms_bind_ipv4 { "ms_bind_ipv4": "true" }
for that matter.
Now I'm typing this I'm like, hmm, there was an issue with this on the mailinglist related to this, as in where you would need to disable ipv4 explicitly. I believe it was a peering issue between PGs.
OK, so this is probably it. And up to kernel 5.11 (I'll test 5.10 as well) this would not be a problem.
I should be able te reproduce on a couple of test clusters that are configured similarly, and be able to test if setting ms_bind_ipv4=false on OSDs / MDSs fixes the issue.
I would like to have the heuristic to prefer IPv6 over IPv4 when filtering addresses (as that is default / common behavior for most if not all dual stack systems) ;-).
Hmm, we looked up the documentation, and it seems lacking this whole options in master / latest: https://docs.ceph.com/en/latest/rados/configuration/ms-ref/ The ms_bind_ipv6 option states the following: Description Enable to bind daemons to IPv6 addresses instead of IPv4. Not required if you specify a daemon or cluster So, it says: _instead_. We have configured ms_bind_ipv6, so it should not, according to documentation, bind to IPv4, but to IPv6 only. So either the documenation is not correct, or the current behaviour of the daemons is not correct. Anyways, I'll come back if disabling binding to IPv4 works. Gr. Stefan
participants (3)
-
Ilya Dryomov
-
Jeff Layton
-
Stefan Kooman