Hello Ceph Users, * Problem: we get the following errors when using krbd, we are using rbd for vms. * Workaround: by switching to librbd the errors disappear. * Software: ** Kernel: 6.8.8-2 (parameters: intel_iommu=on iommu=pt pcie_aspm.policy=performance) ** Ceph: 18.2.2 Description/Details: Errors from using krbd with ceph. Side-effects: [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000015af2284 data crc 1221767919 != exp. 282251377 [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000066b200ab data crc 3817026135 != exp. 3925662391 [Wed Aug 21 03:04:17 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:17 2024] libceph: osd13 (1)10.1.4.13:6809 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000008a131738 data crc 2612835980 != exp. 917302924 [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000005160776b data crc 2965872045 != exp. 563323792 [Wed Aug 21 03:04:21 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: osd6 (1)10.1.4.12:6843 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000007e548354 data crc 1265032637 != exp. 2426281931 [Wed Aug 21 03:06:44 2024] libceph: osd0 (1)10.1.4.11:6835 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000009214d802 data crc 2596010853 != exp. 1221875667 [Wed Aug 21 03:06:44 2024] libceph: osd10 (1)10.1.4.12:6809 bad crc/signature [Wed Aug 21 03:06:47 2024] libceph: read_partial_message 000000000f9edc73 data crc 1326019705 != exp. 3079604517 [Wed Aug 21 03:06:47 2024] libceph: osd3 (1)10.1.4.11:6803 bad crc/signature [Wed Aug 21 03:06:50 2024] libceph: read_partial_message 000000004769da61 data crc 3421275194 != exp. 4183754554 [Wed Aug 21 03:06:50 2024] libceph: osd8 (1)10.1.4.12:6835 bad crc/signature [Wed Aug 21 03:06:51 2024] libceph: read_partial_message 0000000044db9a59 data crc 2603270708 != exp. 4150529351 [Wed Aug 21 03:06:51 2024] libceph: osd14 (1)10.1.4.13:6806 bad crc/signature Description/Details 2: vms get problems with buffer i/o errors on rbd-backed virtual disks: Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=885s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=847s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Add. Sense: I/O process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 CDB: Write(10) 2a 00 34 87 48 08 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 CDB: Write (10) 2a 00 34 81 52 70 00 00 58 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281032 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 880890480 op 0x1: (WRITE) flags 0x103000 phys_seg 11 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: EXT4-fs warning (device sdb1): ext4_end_bio:343: I/0 error 10 writing to inode 27525908 starting Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111054, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: buffer_io_error: 21 callbacks suppressed Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on device sdbl, logical block 110159873 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on dev sdbl, logical block 110111055, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=875s Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111056, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111057, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111058, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 CDB: Write(10) 2a 00 34 87 48 10 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111059, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281040 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111060, lost async page write Thanks for helping out. Greetings Jonas