Hi Frédéric, The curiosity of Albert's cluster is that (msgr) v1 and v2 are present on the mons, as well as on the osds backend. But v2 is absent on the public OSD and MDS network The specific point is that the public network has been changed. At first, I thought it was the order of declaration of my_host (v1 before v2) but apparently, that's not it. Le mer. 17 juil. 2024 à 09:21, Frédéric Nass <frederic.nass@univ-lorraine.fr> a écrit :
Hi David,
Redeploying 2 out of 3 MONs a few weeks back (to have them using RocksDB to be ready for Quincy) prevented some clients from connecting to the cluster and mounting cephfs volumes.
Before the redeploy, these clients were using port 6789 (v1) explicitly as connections wouldn't work with port 3300 (v2). After the redeploy, removing port 6789 from mon_ips fixed the situation.
Seems like msgr v2 activation did only occur after all 3 MONs were redeployed and used RocksDB. Not sure why this happened though.
@Albert, if this cluster has been upgrade several times, you might want to check /var/lib/ceph/$(ceph fsid)/kv_backend, redeploy the MONS if leveldb, make sure all clients use the new mon_host syntax in ceph.conf ([v2:<cthulhu1_ip>:3300,v1:<cthulhu1_ip>:6789],etc.]) and check their ability to connect to port 3300.
Cheers, Frédéric.
----- Le 16 Juil 24, à 17:53, David david.casier@aevoo.fr a écrit :
Albert,
The network is ok.
However, strangely, the osd and mds did not activate msgr v2 (msgr v2 was activated on mon).
It is possible to bypass by adding the "ms_mode=legacy" option but you need to find out why msgr v2 is not activated
Le mar. 16 juil. 2024 à 15:18, Albert Shih <Albert.Shih@obspm.fr> a écrit :
Le 16/07/2024 à 15:04:05+0200, David C. a écrit Hi,
I think it's related to your network change.
I though about it but in that case why the old (and before upgrade) server works ?
Can you send me the return of "ceph report" ?
Nothing related to the old subnet. (see attach file)
Regards.
JAS -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 16 juil. 2024 15:14:21 CEST
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io