On Tue, 2021-03-02 at 17:44 +0100, Stefan Kooman wrote: That needs different values. See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… You can try passing in a specific mon address and port, like: 192.168.21.22:3300:/cephfs/dir/ ...and then pass in ms_mode=crc or something similar. That said, what you're doing should be working, so this sounds like a regression. I presume you're able to mount with earlier kernels? What's the latest kernel version that you have that works? -- Jeff Layton <jlayton(a)redhat.com>

On 3/2/21 6:00 PM, Jeff Layton wrote: 5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) with a cluster that has ms_bind_ipv4=false works. Port 3300 ms_mode=prefer-crc and ms_mode=crc work. I have tested with 5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) port 3300 and ms_mode=crc as well as ms_mode=prefer-crc and that works when cluster is running with ms_bind_ipv4=false. So the "fix" is to have this config option set: ceph config set global ms_bind_ipv4 false 5.10 kernel (5.10.19-1-lts Arch Linux) works with a cluster that is IPv6 only but has ms_bind_ipv4=true. So it's "broken" since 5.11. So, we have done more reading / researching on the ms_bind_ip{4,6} options: - https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus#Restart_the_OSD_daem… - https://github.com/rook/rook/issues/6266 ^^ Describe that you have to disable bind to IPv4. - https://github.com/ceph/ceph/pull/13317 ^^ this PR is not completely correct: **Note:** You may use IPv6 addresses instead of IPv4 addresses, but you must set ``ms bind ipv6`` to ``true``. ^^ That is not enough as we have learned, and starts to give trouble with 5.11 linux cephfs client. And from this documentation: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… we learned that dual stack is not possible for any current stable release, but might be possible with latest code. So the takeaway is that the linux kernel client needs fixing to be able to support dual stack clusters in the future (multiple v1 / v2 address families), and, that until then you should run with ms_bind_ipv4=false for IPv6 only clusters. I'll make a PR to clear up the documenation. Do you want me to create a tracker for the kernel client? I will happily test your changes. Thanks, Stefan

On 3/3/21 1:16 PM, Ilya Dryomov wrote: Indeed, I wrote that. That was not correct. We *did* have set that on a test cluster, but those changes have never propagated to production. Yeah, there are also quite some combinations possible, especially with dual stack and separate public / cluster networks, even with different IP stack maybe? Would be good to know (as in the broader community) what is being tested / supported with regards to networking. Nowadays separate public/cluster is not advised anymore (which is a good thing, thanks Wido) but maybe this should also be made clear for IP families. IMHO it would be good to test both IPv4 and IPv6. Check, I'll do that and come back with a tracker ID. Thanks, Stefan

On 3/3/21 1:16 PM, Ilya Dryomov wrote: I added another tracker as it is related to this thread: https://tracker.ceph.com/issues/49584 ^^^ do _not_ try this on a IPv6 only production cluster with ms_bind_ipv4=true or you will regret it ;-). Gr. Stefan

On 3/2/21 5:42 PM, Ilya Dryomov wrote: That's a lot of output, so I trimmed it: --- snip --- osd.0 up in weight 1 up_from 98071 up_thru 98719 down_at 98068 last_clean_interval [96047,98067) [v2:[2001:7b8:80:1:0:1:2:1]:6848/505534,v1:[2001:7b8:80:1:0:1:2:1]:6854/505534,v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534] [v2:[2001:7b8:80:1:0:1:2:1]:6872/505534,v1:[2001:7b8:80:1:0:1:2:1]:6878/505534,v2:0.0.0.0:6886/505534,v1:0.0.0.0:6892/505534] exists,up 93e7d17f-2c7a-4acd-93c0-586dbb7cc6d7 -- snap --- -- snip --- [mds.mds1{0:229930080} state up:active seq 144042 addr [v2:[2001:7b8:80:1:0:1:3:1]:6800/2234186180,v1:[2001:7b8:80:1:0:1:3:1]:6801/2234186180,v2:0.0.0.0:6802/2234186180,v1:0.0.0.0:6803/2234186180]] Standby daemons: [mds.mds2{-1:229977514} state up:standby seq 2 addr [v2:[2001:7b8:80:3:0:2c:3:2]:6800/2983725953,v1:[2001:7b8:80:3:0:2c:3:2]:6801/2983725953,v2:0.0.0.0:6802/2983725953,v1:0.0.0.0:6803/2983725953]] --- snap --- We only have a public address network in use, and IPv6 only, So not sure how we could have multiple v1 addresses for OSDs and MDSes. Thanks, Stefan

On 3/2/21 6:54 PM, Ilya Dryomov wrote: Yes, exclusively. So dual stack was not something that was possible before nautilus. We always did set "ms_bind_ipv6=true". But good find, I missed that "0.0.0.0" part. Yeah, that must be it. ceph daemon osd.0 config get ms_bind_ipv4 { "ms_bind_ipv4": "true" } And ceph daemon mds.mds1 config get ms_bind_ipv4 { "ms_bind_ipv4": "true" } for that matter. Now I'm typing this I'm like, hmm, there was an issue with this on the mailinglist related to this, as in where you would need to disable ipv4 explicitly. I believe it was a peering issue between PGs. OK, so this is probably it. And up to kernel 5.11 (I'll test 5.10 as well) this would not be a problem. I should be able te reproduce on a couple of test clusters that are configured similarly, and be able to test if setting ms_bind_ipv4=false on OSDs / MDSs fixes the issue. I would like to have the heuristic to prefer IPv6 over IPv4 when filtering addresses (as that is default / common behavior for most if not all dual stack systems) ;-). I'll do testing and let you know. Gr. Stefan