Hello, we had sometimes problem with OSD in our all-flash NVMe CEPH cluster. Our cluster is made by cephadm, using Debian bookworm as a "basic" system and podman as a container solution. We are using two-ports Mellanox 100Gbps adapters: # lspci | grep Mellanox 43:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 43:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] We are using TCP/IP for front-end communication and RDMA for internal communication. Sometimes (after cca 2-3 month of normal working) one OSD container from whole machine have a problem to connect to other OSD processes, but it is still connected to MON process, which goes to the situation, when it is "half-dead" - it cannot connects to other OSD, other OSD cannot connect to it, but it reports to the MON process, that it is still alive. We have experience, that this situation can resolve by restart container of this OSD. We have script, which is running every 5 minute and which searching in logs for this pattern: start_waiting_for_healthy If is this pattern found, we restart of container with appropriate OSD daemon. After restart this container, CEPH cluster goes quickly to HEALTHY state. I'm adding log, where this situation occured, up to restarting container. It was an osd.12 container with PID 3920171. In my opinion there is some problem to open infiniband connection - some problems in infiniband stack... :-( I'm open to you questions, although I cannot do some special operations, because it is our production cluster... Many thanks for this software :-) Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Hello, we are using CEPH version 19.2.0... Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Hi Jan, When this happens, does it happen to a single OSD only? Like other OSDs on the same node are still working fine? Any useful information system-wise (dmesg) when it happens? Could it be that the ms_async_rdma_local_gid for this OSD is no longer available / has changed? [1] Does your setup matche the config options (systemd unit files, /etc/security/limits.conf, ceph config) mentioned in this article [2] and the use of RoCE with Jumbo Frames disabled [3][4]? As you might know already, you'll get limited support from the community when it comes to RDMA issues due to the limited number of users of Ceph with RDMA for low-latency networking. I'm not even sure RDMA in Ceph got beyond the experimental phase. Maybe someone can shed some light on this. If your workloads really need RDMA, I would advise you to push any ms async log levels to debug (and any other RDMA related log levels if any) and create a Ceph tracker. Sorry I can't help much when it comes to RDMA as I never used it myself. Regards, Frédéric. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EHGZ2XFTNWB... [2] https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-proje... [3] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5JD4ATRXKMM... [4] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ETNIF52ULSF... ----- Le 3 Juin 25, à 11:10, Jan Marek jmarek@jcu.cz a écrit :
Hello,
we are using CEPH version 19.2.0...
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frédéric, thanks for reply. My reply in text... Dne st, čen 04, 2025 at 12:38:39 CEST napsal(a) Frédéric Nass:
Hi Jan,
When this happens, does it happen to a single OSD only?
Yes, only one OSD.
Like other OSDs on the same node are still working fine?
Yes, other OSDs in this machine works and perform well.
Any useful information system-wise (dmesg) when it happens?
I've tried to check it, but there was nothink interesting...
Could it be that the ms_async_rdma_local_gid for this OSD is no longer available / has changed? [1]
It cannot be changed - it's statically set for every machine in the cluster. It is derived from IP address of network card and must be the same for every OSD in certain machine...
Does your setup matche the config options (systemd unit files, /etc/security/limits.conf, ceph config) mentioned in this article [2] and the use of RoCE with Jumbo Frames disabled [3][4]?
We have enabled Jumbo frames - we need it, and we didn't enable unlimited memlock... Thanks for pointing me this docs...
As you might know already, you'll get limited support from the community when it comes to RDMA issues due to the limited number of users of Ceph with RDMA for low-latency networking. I'm not even sure RDMA in Ceph got beyond the experimental phase. Maybe someone can shed some light on this.
We mentioned, that all-flash CEPH cluster with images of "disks" of virtual machines for ProxMox with databses and so on, will profit from low latency of RDMA traffic... Our normal traffic to this cluster is around 100MB/s read and 40MB/s write with practically zero latency...
If your workloads really need RDMA, I would advise you to push any ms async log levels to debug (and any other RDMA related log levels if any) and create a Ceph tracker.
We will try this, thanks.
Sorry I can't help much when it comes to RDMA as I never used it myself.
Regards, Frédéric.
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EHGZ2XFTNWB... [2] https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-proje... [3] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5JD4ATRXKMM... [4] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ETNIF52ULSF...
----- Le 3 Juin 25, à 11:10, Jan Marek jmarek@jcu.cz a écrit :
Hello,
we are using CEPH version 19.2.0...
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
I really question the use of rdma with Ceph...at least currently. Sure pure rdma latency ( 0.005 ms ?) is much better than tcp (0.03-0.05 ms), so it will be better when say you are serving i/o from a single raw nvme drive, but in an SDS system like Ceph where OSD latency is around 0.3 ms read, 1 ms write, we are already an order of magnitude higher than tcp, so using rdma rather than tcp will not have a noticeable impact and may not be worth the hassle. This ofcourse may change in the future if OSD latencies drop, but this will not be any time soon. On 03/06/2025 12:07, Jan Marek wrote:
Hello,
we had sometimes problem with OSD in our all-flash NVMe CEPH cluster.
Our cluster is made by cephadm, using Debian bookworm as a "basic" system and podman as a container solution.
We are using two-ports Mellanox 100Gbps adapters:
# lspci | grep Mellanox 43:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 43:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
We are using TCP/IP for front-end communication and RDMA for internal communication.
Sometimes (after cca 2-3 month of normal working) one OSD container from whole machine have a problem to connect to other OSD processes, but it is still connected to MON process, which goes to the situation, when it is "half-dead" - it cannot connects to other OSD, other OSD cannot connect to it, but it reports to the MON process, that it is still alive.
We have experience, that this situation can resolve by restart container of this OSD.
We have script, which is running every 5 minute and which searching in logs for this pattern:
start_waiting_for_healthy
If is this pattern found, we restart of container with appropriate OSD daemon.
After restart this container, CEPH cluster goes quickly to HEALTHY state.
I'm adding log, where this situation occured, up to restarting container. It was an osd.12 container with PID 3920171.
In my opinion there is some problem to open infiniband connection - some problems in infiniband stack... :-(
I'm open to you questions, although I cannot do some special operations, because it is our production cluster...
Many thanks for this software :-)
Sincerely Jan Marek
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Frédéric Nass
-
Jan Marek
-
Maged Mokhtar