Hello all, I am trying to troubleshoot a ceph cluster version 18.2.2 having users reporting slow and blocked reads and writes. When running "ceph status" I am seeing many warnings about its health state: cluster: id: cc881230-e0dd-11ee-aa9e-37c4e4e5e14b health: HEALTH_WARN 6 clients failing to respond to capability release 2 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming Too many repaired reads on 11 OSDs Degraded data redundancy: 2 pgs degraded 105 pgs not deep-scrubbed in time 109 pgs not scrubbed in time 1 mgr modules have recently crashed 12 slow ops, oldest one blocked for 97678 sec, daemons [osd.11,osd.12,osd.15,osd.16,osd.19,osd.20,osd.28,osd.3,osd.32,osd.34]... have slow ops. services: mon: 3 daemons, quorum file03-xx,file04-xx,file05-xx (age 17h) mgr: file03-xx.xxxxxx(active, since 2w), standbys: file04-xx.xxxxxx mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 17h), 44 in (since 39h); 492 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 2065 pgs objects: 66.44M objects, 140 TiB usage: 281 TiB used, 304 TiB / 586 TiB avail pgs: 16511162/134215883 objects misplaced (12.302%) 1508 active+clean 487 active+remapped+backfill_wait 53 active+clean+scrubbing+deep 8 active+clean+scrubbing 5 active+remapped+backfilling 2 active+recovering+degraded+repair 2 active+recovering+repair io: recovery: 47 MiB/s, 37 objects/s When checking the output of `ceph -w` I am flooded with crc error messages like the examples below: 2024-04-24T19:15:40.430334+0000 osd.32 [ERR] 3.566 full-object read crc 0xa5da7fa != expected 0xffffffff on 3:66a8d8f5:::10001c72400.00000007:head 2024-04-24T19:15:40.430507+0000 osd.39 [ERR] 3.270 full-object read crc 0xa1bc3a1e != expected 0xffffffff on 3:0e44aa2f:::1000265a625.00000003:head 2024-04-24T19:15:40.494249+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.529771+0000 osd.32 [ERR] 3.566 full-object read crc 0xa5da7fa != expected 0xffffffff on 3:66a8d8f5:::10001c72400.00000007:head 2024-04-24T19:15:40.582128+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.583350+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.662945+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.698197+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.699389+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.769191+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.834344+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.835513+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head I suspect this is the main issue affecting the cluster health state and performance so I am trying to address this first. The "expected 0xffffffff" crc seems like a bug to me and I found an open ticket (https://tracker.ceph.com/issues/53240) with similar error messages but I am not sure this is related to my case. Could someone point me to the steps to solve these errors? Cheers, -- Fabio