On 25/03/2021 20:56, Stefan Kooman wrote:
On 3/25/21 8:47 PM, Simon Oosthoek wrote:
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN 1310/ceph-mgr tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN 1310/ceph-mgr tcp6 0 0 :::8443 :::* LISTEN 1310/ceph-mgr tcp6 0 0 :::9283 :::* LISTEN 1310/ceph-mgr unix 2 [ ACC ] STREAM LISTENING 26205 1564/master private/tlsmgr unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr /var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-(
Ok, but that is easily fixable:
ceph config set osd.$id public_addr your_ip_here
Or you can put that in the ceph.conf for the OSDs on each storage server.
Do you have a cluster network as well? If so you should set that IP too.
Only when you run IPv6 only and have not yet set ms_bind_ipv4=false you should not do this. In that case you first have to make sure you set ms_bind_ipv4=false.
As soon as your OSDs are bound to their correct IP again they can peer with each other and it will fix itself.
@Ceph devs: a 14.2.19 with a fix for this issue would avoid other people running into this issue.
Gr. Stefan
Hoi Stefan tnx, I only have one network (25Gbit should be enough), after fixing the mon/mgr nodes and the one OSD node that I upgraded, the cluster seems to be recovering. At first I understood Dan's fix to put the mgr's address in all nodes' configs, but after watching the errors, I changed it to the node's own address on each node... I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19 will be available quickly. Cheers /Simon