Hmm, I'm getting a bit confused. Could you also send the output of "ceph osd pool ls detail".
File ceph-osd-pool-ls-detail.txt attached.
Did you look at the disk/controller cache settings? I don't have disk controllers on Ceph machines. The hard disk is directly attached to the motherboard via SATA cable. But there can be a on chip disk controller on the motherboard, I'm not sure. If your worry is fsync persistence, I have thoroughly tested database fsync reliability on Ceph RBD with hundreds of transactions per second and remove network cable and restart the database machine, etc. while inserts going on. and I did not lose a single transaction. I simulated this many times and persistence on my Ceph cluster was perfect (i.e not a single loss).
I think you should start a deep-scrub with "ceph pg deep-scrub 3.b" and record the output of "ceph -w | grep '3\.b'" (note the single quotes).
The error messages you included in one of your first e-mails are only on 1 out of 3 scrub errors (3 lines for 1 error). We need to find all 3 errors.
I ran again the "ceph pg deep-scrub 3.b", here is the whole output of ceph -w: 2020-11-02 22:33:48.224392 osd.0 [ERR] 3.b shard 2 soid 3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate had a missing info key 2020-11-02 22:33:48.224396 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head : failed to pick suitable object info 2020-11-02 22:35:30.087042 osd.0 [ERR] 3.b deep-scrub 3 errors Btw, I'm very grateful for your perseverance on this. Best regards Sagara