crc errors on 4 node cluster

3 Nov 2025

      From 
https://forum.proxmox.com/threads/hyperconverged-cluster-logging-seemingly-r...

We have 4 nodes (dual Xeon CPU's, 256G RAM, 4 NVMe SSD's, 4 HDD's and 
dual Melanox 25Gb/s sfp's) in a cluster. I have started noticing 
apparently random crc errors in the osd logs.

Node B, osd.6
2025-10-23T10:32:59.808+0200 7f22a75bf700  0 bad crc in data 3330350463 
!= exp 677417498 from v1:192.168.131.4:0/3121668685
192.168.131.4 is node D

Node B, osd.7
2025-10-23T09:35:12.995+0200 7fbcdbcd7700  0 bad crc in data 3922083958 
!= exp 3479198006 from v1:192.168.131.2:0/2732728486
192.168.131.2 is node B, which is the node osd.7 is on.

and so there are others on other nodes and osd's. From what I understand 
this means that data copied from some other osd to this one that logs 
the error fails the crc test. However, I have taken (as a test) one of 
these ssd's out of the cluster and it tests just fine. I put it back and 
no crc errors are logged for it.

Question: Can something else we causing this? A network connector? It 
seems pretty random so me, so how can I trace this sources of this?

It was suggested to me by die vendor tech support that turning 
offloading off on the nic: /ethtool -K vmbr0 rx off tx off tso off gso 
off gro off lro off./  That doesn't seem to make a difference however.

The next step would be to check for a MTU/jumbo mismatch.

Some details: ceph version 17.2.7

# ceph osd tree
ID   CLASS  WEIGHT    TYPE NAME  STATUS  REWEIGHT  PRI-AFF
  -1         62.29883  root default
  -3         15.17824      host FT1-NodeA
   2    hdd   1.86029          osd.2           up   1.00000 1.00000
   3    hdd   1.86029          osd.3           up   1.00000 1.00000
   4    hdd   1.86029          osd.4           up   1.00000 1.00000
   5    hdd   1.86589          osd.5           up   1.00000 1.00000
   0    ssd   0.93149          osd.0           up   1.00000 1.00000
  28    ssd   3.30690          osd.28          up   1.00000 1.00000
  29    ssd   3.49309          osd.29          up   1.00000 1.00000
  -7         16.10413      host FT1-NodeB
  10    hdd   1.86029          osd.10          up   1.00000 1.00000
  11    hdd   1.86029          osd.11          up   1.00000 1.00000
  26    hdd   1.86029          osd.26          up   1.00000 1.00000
  27    hdd   1.86029          osd.27          up   1.00000 1.00000
   6    ssd   0.93149          osd.6           up   1.00000 1.00000
   7    ssd   0.93149          osd.7           up   1.00000 1.00000
  25    ssd   3.49309          osd.25          up   1.00000 1.00000
  41    ssd   3.30690          osd.41          up   1.00000 1.00000
-10         15.84383      host FT1-NodeC
  14    hdd   1.59999          osd.14          up   1.00000 1.00000
  15    hdd   1.86029          osd.15          up   1.00000 1.00000
  16    hdd   1.86029          osd.16          up   1.00000 1.00000
  17    hdd   1.86029          osd.17          up   1.00000 1.00000
   8    ssd   0.93149          osd.8           up   1.00000 1.00000
   9    ssd   0.93149          osd.9           up   1.00000 1.00000
  24    ssd   3.49309          osd.24          up   1.00000 1.00000
  43    ssd   3.30690          osd.43          up   1.00000 1.00000
-13         15.17264      host FT1-NodeD
  20    hdd   1.86029          osd.20          up   1.00000 1.00000
  21    hdd   1.86029          osd.21          up   1.00000 1.00000
  22    hdd   1.86029          osd.22          up   1.00000 1.00000
  23    hdd   1.86029          osd.23          up   1.00000 1.00000
  12    ssd   3.30690          osd.12          up   1.00000 1.00000
  13    ssd   0.93149          osd.13          up   1.00000 1.00000
  19    ssd   3.49309          osd.19          up   1.00000 1.00000

The spinners have their RocksDB on the nvme's, for extra performance, 
but I have not noticed any crc errors prior to installing the 4TB 
Samsungs in each node.

Suggestions are welcome.

thanks

Roland

Roland Giesler

Roland Giesler

Bailey Allison

Eugen Block

tags

participants (3)