On Tue, Mar 2, 2021 at 6:02 PM Stefan Kooman <stefan@bit.nl> wrote:
On 3/2/21 5:42 PM, Ilya Dryomov wrote:
On Tue, Mar 2, 2021 at 9:26 AM Stefan Kooman <stefan@bit.nl> wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: ceph: corrupt mdsmap Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22 Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported).
Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port?
Hi Stefan,
Those "another match of type 1" errors suggest that you have two different v1 addresses for some of or all OSDs and MDSes in osdmap and mdsmap respectively.
What is the output of "ceph osd dump" and "ceph fs dump"?
That's a lot of output, so I trimmed it:
--- snip --- osd.0 up in weight 1 up_from 98071 up_thru 98719 down_at 98068 last_clean_interval [96047,98067) [v2:[2001:7b8:80:1:0:1:2:1]:6848/505534,v1:[2001:7b8:80:1:0:1:2:1]:6854/505534,v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534]
Where did "v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534" come from? This is what confuses the kernel client: it sees two addresses of the same type and doesn't know which one to pick. Instead of blindly picking the first one (or some other dubious heuristic) it just denies the osdmap.
[mds.mds1{0:229930080} state up:active seq 144042 addr [v2:[2001:7b8:80:1:0:1:3:1]:6800/2234186180,v1:[2001:7b8:80:1:0:1:3:1]:6801/2234186180,v2:0.0.0.0:6802/2234186180,v1:0.0.0.0:6803/2234186180]]
Same for the mdsmap. Were you using ipv6 with the kernel client before upgrading to 5.11? What is output of "ceph daemon osd.0 config get ms_bind_ipv4" on the osd0 node? Thanks, Ilya