Server not responding to keepalive - cephadm 24.04
Setting up a new cluster on fresh ubuntu 24.04 hosts using cephadm. The first 5 hosts all added without issue, but the next N hosts all throw the same error when adding through the dashboard or through ceph orch add... All hosts have ssh access, docker and all base requirements confirmed. Command: ceph orch host add foxtrot 10.1.20.16 Error: Error EINVAL: check-host failed: Unable to write foxtrot:/var/lib/ceph/260974b7-fbb6-11ef-911a-107c6152e887/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11: Server not responding to keepalive There is a long stack trace that references asyncssh and the /lib/python3.9/site-packages/asyncssh/sftp.py file. Happy to share but not sure best way in the mailing list. Thanks for any help and advice. -Reid -- Drawbridge Digital is proud to be a certified Veteran-Owned Small Business. The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this email and delete the message and any attachments from your computer.
Hi, you can use any pastebin to share verbose logs. I tried to find something in the code related to the message (Server not responding to keepalive), but couldn't find anything. Maybe it's not Ceph related? But I'm also not a dev, so I might just misunderstand. Zitat von Reid Kelley <reid@drawbridgedigital.com>:
Setting up a new cluster on fresh ubuntu 24.04 hosts using cephadm. The first 5 hosts all added without issue, but the next N hosts all throw the same error when adding through the dashboard or through ceph orch add... All hosts have ssh access, docker and all base requirements confirmed.
Command: ceph orch host add foxtrot 10.1.20.16
Error: Error EINVAL: check-host failed: Unable to write foxtrot:/var/lib/ceph/260974b7-fbb6-11ef-911a-107c6152e887/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11: Server not responding to keepalive
There is a long stack trace that references asyncssh and the /lib/python3.9/site-packages/asyncssh/sftp.py file. Happy to share but not sure best way in the mailing list.
Thanks for any help and advice. -Reid
--
Drawbridge Digital is proud to be a certified Veteran-Owned Small Business.
The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited.
If you have received this communication in error, please notify the sender immediately by replying to this email and delete the message and any attachments from your computer. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm pretty sure it's an asyncssh error. I've never seen the issue, but seems related. https://github.com/ronf/asyncssh/issues/725 On Fri, Mar 28, 2025, 11:43 Eugen Block <eblock@nde.ag> wrote:
Hi,
you can use any pastebin to share verbose logs. I tried to find something in the code related to the message (Server not responding to keepalive), but couldn't find anything. Maybe it's not Ceph related? But I'm also not a dev, so I might just misunderstand.
Zitat von Reid Kelley <reid@drawbridgedigital.com>:
Setting up a new cluster on fresh ubuntu 24.04 hosts using cephadm. The first 5 hosts all added without issue, but the next N hosts all throw the same error when adding through the dashboard or through ceph orch add... All hosts have ssh access, docker and all base requirements confirmed.
Command: ceph orch host add foxtrot 10.1.20.16
Error: Error EINVAL: check-host failed: Unable to write
foxtrot:/var/lib/ceph/260974b7-fbb6-11ef-911a-107c6152e887/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11:
Server not responding to keepalive
There is a long stack trace that references asyncssh and the /lib/python3.9/site-packages/asyncssh/sftp.py file. Happy to share but not sure best way in the mailing list.
Thanks for any help and advice. -Reid
--
Drawbridge Digital is proud to be a certified Veteran-Owned Small Business.
The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited.
If you have received this communication in error, please notify the sender immediately by replying to this email and delete the message and any attachments from your computer. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, we solved this today and sharing here. The issue was an MTU mismatch on our end between these subsequent servers and the switch ports. Cephadm is working just fine, humans...not so much. -- Drawbridge Digital is proud to be a certified Veteran-Owned Small Business. The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this email and delete the message and any attachments from your computer.
participants (3)
-
Curt
-
Eugen Block
-
Reid Kelley