Hi, do you map RBDs on your storage nodes? I read somewhere that this might be just an informative message that a client had to resend its data. Zitat von Roland Giesler <roland@giesler.za.net>:
From https://forum.proxmox.com/threads/hyperconverged-cluster-logging-seemingly-r...
We have 4 nodes (dual Xeon CPU's, 256G RAM, 4 NVMe SSD's, 4 HDD's and dual Melanox 25Gb/s sfp's) in a cluster. I have started noticing apparently random crc errors in the osd logs.
Node B, osd.6 2025-10-23T10:32:59.808+0200 7f22a75bf700 0 bad crc in data 3330350463 != exp 677417498 from v1:192.168.131.4:0/3121668685 192.168.131.4 is node D
Node B, osd.7 2025-10-23T09:35:12.995+0200 7fbcdbcd7700 0 bad crc in data 3922083958 != exp 3479198006 from v1:192.168.131.2:0/2732728486 192.168.131.2 is node B, which is the node osd.7 is on.
and so there are others on other nodes and osd's. From what I understand this means that data copied from some other osd to this one that logs the error fails the crc test. However, I have taken (as a test) one of these ssd's out of the cluster and it tests just fine. I put it back and no crc errors are logged for it.
Question: Can something else we causing this? A network connector? It seems pretty random so me, so how can I trace this sources of this?
It was suggested to me by die vendor tech support that turning offloading off on the nic: /ethtool -K vmbr0 rx off tx off tso off gso off gro off lro off./ That doesn't seem to make a difference however.
The next step would be to check for a MTU/jumbo mismatch.
Some details: ceph version 17.2.7
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 62.29883 root default -3 15.17824 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.93149 osd.0 up 1.00000 1.00000 28 ssd 3.30690 osd.28 up 1.00000 1.00000 29 ssd 3.49309 osd.29 up 1.00000 1.00000 -7 16.10413 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.93149 osd.6 up 1.00000 1.00000 7 ssd 0.93149 osd.7 up 1.00000 1.00000 25 ssd 3.49309 osd.25 up 1.00000 1.00000 41 ssd 3.30690 osd.41 up 1.00000 1.00000 -10 15.84383 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 8 ssd 0.93149 osd.8 up 1.00000 1.00000 9 ssd 0.93149 osd.9 up 1.00000 1.00000 24 ssd 3.49309 osd.24 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 15.17264 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 12 ssd 3.30690 osd.12 up 1.00000 1.00000 13 ssd 0.93149 osd.13 up 1.00000 1.00000 19 ssd 3.49309 osd.19 up 1.00000 1.00000
The spinners have their RocksDB on the nvme's, for extra performance, but I have not noticed any crc errors prior to installing the 4TB Samsungs in each node.
Suggestions are welcome.
thanks
Roland _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io