[Ceph-users] Multiple OSD crashes

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)