[ceph-users] "Signature check failed" from certain clients

hey folks, I spent the past 2 hours digging through the forums and similar sources with no luck.. I use ceph storage for docker stacks, and this issue has taken the whole thing down as I cannot mount their volumes back... Starting yesterday, some of my nodes cannot mount the filesystem and it just hangs on the mount command, while the logs are full of the messages below. It doesn't matter which MDS node is active for some clients it just works, and for others it doesn't. Any hints? 2019-08-15 02:20:55.230 7f6e80e47700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13406 cs=3203 l=0).handle_message_footer Signature check failed 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 Message signature does not match contents. 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1Signature on message: 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 sig: 1045888080092928376 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1Locally calculated signature: 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 sig_check:13982737427498638198 2019-08-15 02:20:56.254 7f6e7fe45700 0 Signature failed. 2019-08-15 02:20:56.254 7f6e7fe45700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13410 cs=3205 l=0).handle_message_footer Signature check failed 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 Message signature does not match contents. 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1Signature on message: 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 sig: 1045888080092928376 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1Locally calculated signature: 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 sig_check:13982737427498638198 2019-08-15 02:20:57.246 7f6e80646700 0 Signature failed. 2019-08-15 02:20:57.246 7f6e80646700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13414 cs=3207 l=0).handle_message_footer Signature check failed Cheers, Peter