[ceph-users] msgr2 not used on OSDs in some Nautilus clusters

15 Nov 2019

      I've upgraded 7 of our clusters to Nautilus (14.2.4) and noticed that on some of the clusters (3 out of 7) the OSDs aren't using msgr2 at all.  Here's the output for osd.0 on 2 clusters of each type:

### Cluster 1 (v1 only):
# ceph osd find 0 | jq -r '.addrs'
{
  "addrvec": [
    {
      "type": "v1",
      "addr": "10.26.0.33:6809",
      "nonce": 4185021
    }
  ]
}

### Cluster 2 (v1 only):
# ceph osd find 0 | jq -r '.addrs'
{
  "addrvec": [
    {
      "type": "v1",
      "addr": "10.197.0.243:6801",
      "nonce": 3802140
    }
  ]
}

### Cluster 3 (v1 & v2):
# ceph osd find 0 | jq -r '.addrs'
{
  "addrvec": [
    {
      "type": "v2",
      "addr": "10.32.0.36:6802",
      "nonce": 3167
    },
    {
      "type": "v1",
      "addr": "10.32.0.36:6804",
      "nonce": 3167
    }
  ]
}

### Cluster 4 (v1 & v2):
# ceph osd find 0 | jq -r '.addrs'
{
  "addrvec": [
    {
      "type": "v2",
      "addr": "10.36.0.12:6820",
      "nonce": 3150
    },
    {
      "type": "v1",
      "addr": "10.36.0.12:6827",
      "nonce": 3150
    }
  ]
}

All of the mon nodes have the same msgr settings of:

# ceph daemon mon.$(hostname -s) config show | grep msgr
    "mon_warn_on_msgr2_not_enabled": "true",
    "ms_bind_msgr1": "true",
    "ms_bind_msgr2": "true",
    "ms_msgr2_encrypt_messages": "false",
    "ms_msgr2_sign_messages": "false",

What could be causing this?  All of the clusters are listening on port 3300 for v2 and 6789 for v1.  I can even connect to port 3300 on the mon nodes from the OSD nodes.

Thanks,
Bryan

[ceph-users] msgr2 not used on OSDs in some Nautilus clusters

Bryan Stillwell