Hi Jan, When this happens, does it happen to a single OSD only? Like other OSDs on the same node are still working fine? Any useful information system-wise (dmesg) when it happens? Could it be that the ms_async_rdma_local_gid for this OSD is no longer available / has changed? [1] Does your setup matche the config options (systemd unit files, /etc/security/limits.conf, ceph config) mentioned in this article [2] and the use of RoCE with Jumbo Frames disabled [3][4]? As you might know already, you'll get limited support from the community when it comes to RDMA issues due to the limited number of users of Ceph with RDMA for low-latency networking. I'm not even sure RDMA in Ceph got beyond the experimental phase. Maybe someone can shed some light on this. If your workloads really need RDMA, I would advise you to push any ms async log levels to debug (and any other RDMA related log levels if any) and create a Ceph tracker. Sorry I can't help much when it comes to RDMA as I never used it myself. Regards, Frédéric. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EHGZ2XFTNWB... [2] https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-proje... [3] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5JD4ATRXKMM... [4] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ETNIF52ULSF... ----- Le 3 Juin 25, à 11:10, Jan Marek jmarek@jcu.cz a écrit :
Hello,
we are using CEPH version 19.2.0...
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io