Can't bind mon to v1 port in Octopus.
Hi, I have a lab single node cluster with octopus installed via ceph-ansible. Both v1 and v2 were enabled in ceph-ansible vars with the correct suffixes. The configuration was generated correctly and both ports were included in the mon array. [global] cluster network = 172.16.6.0/24 fsid = bb204a5c-957d-4a06-a372-redacted mon_host = [v2:172.16.6.210:3300/0,v1:172.16.6.210:6789/0] mon initial members = aio1 mon_pg_warn_max_per_osd = 0 osd pool default crush rule = -1 osd_pool_default_min_size = 1 osd_pool_default_size = 1 public network = 172.16.6.0/24 I can also see that `ms_bind_msgr1` is enabled in the live config. root@aio1 ~ # ceph daemon mon.aio1 config show | grep msgr "mon_warn_on_msgr2_not_enabled": "true", "ms_bind_msgr1": "true", "ms_bind_msgr2": "true", However only v2 is binding netstat -tlnp |grep mon tcp 0 0 172.16.6.210:3300 0.0.0.0:* LISTEN 2039098/ceph-mon I have a client that only speaks v1 (ceph-csi) that can't talk to the v2 port 2020-06-15T09:49:51.330+0100 7f8776038700 -1 --2- v2:172.16.6.210:3300/0 >> conn(0x563bfd6b2000 0x563bde5ff600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer is using msgr V1 protocol 2020-06-15T09:49:52.258+0100 7f8776038700 -1 --2- v2:172.16.6.210:3300/0 >> conn(0x563bfd6b2000 0x563bde5ff600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer is using msgr V1 protocol What could be the reason for mon not binding to port 6789? Thanks Miguel
Just to add that a dump of that live config does indeed not show the v1 port. So it seems to be ignoring the config. I tried the alternative mon host syntaxes without success. root@aio1 ~ # ceph mon dump dumped monmap epoch 2 epoch 2 fsid bb204a5c-957d-4a06-a372-redacted last_changed 2020-06-09T00:08:28.409322+0100 created 2020-06-09T00:08:28.409322+0100 min_mon_release 15 (octopus) 0: v2:172.16.6.210:3300/0 mon.aio1
After some more testing it seems that ceph just does no pickup on some of ceph.conf changes after bootstrapped. It was possible to bind to the v1 port using `ceph mon set-addrs aio1 [v2:172.16.6.210:3300,v1:172.16.6.210:6789]` It was defo not an issue with OS syscalls or permissions, just ceph not picking up on new config after a restart.
My understanding is that MONs only configure themselves from the config file at first startup. After that all MONs use the monmap to learn about themselves, and their peers. As such; adding an address to the config file for a running MON, even if you restart / reboot, would not achieve the expect changes, as it doesn't modify the monmap. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: mafonso@gmail.com [mailto:mafonso@gmail.com] Sent: Thursday, June 18, 2020 12:56 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Can't bind mon to v1 port in Octopus. After some more testing it seems that ceph just does no pickup on some of ceph.conf changes after bootstrapped. It was possible to bind to the v1 port using `ceph mon set-addrs aio1 [v2:172.16.6.210:3300,v1:172.16.6.210:6789]` It was defo not an issue with OS syscalls or permissions, just ceph not picking up on new config after a restart. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
DHilsbos@performair.com
-
mafonso@gmail.com
-
Miguel Afonso