Hello folks! I hope you are doing well :) I have a general question about XFS and LVM backend OSD performance and possible effects if they are used together in the same pool. I built a cluster 5 years ago with Nautilus and I used the XFS backend for OSD's. After 5 years they reached me back with ERR state and I see 10K++ slow ops, 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down due to compactions problem. At the first check-up I found some drives were replaced by others and they used LVM backend for the replaced drives without using wal+db on nvme. In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM drives without wal+db. We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 workload. RGW index stored on SSD pool. 10 nodes: with - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD Total HDD raw size 2.8PiB, SSD size 26TiB I started fixing all the problems one by one and I'm gonna recreate these LVM drives without wal+db and I wonder 2 questions: 1- Are there any speed or latency differences on XFS and LVM backend OSD's for 16TB 7200rpm NL-SAS drives. 2- Mixing XFS and LVM backend on the same cluster does have any negative effect or problems? Best regards: Extra note: If you wonder please check the LSBLK output for 1/10 server: NODE-01# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 14.6T 0 disk ├─sda1 8:1 0 100M 0 part /var/lib/ceph/osd/ceph-180 └─sda2 8:2 0 14.6T 0 part sdb 8:16 0 14.6T 0 disk ├─sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-181 └─sdb2 8:18 0 14.6T 0 part sdc 8:32 0 14.6T 0 disk ├─sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-182 └─sdc2 8:34 0 14.6T 0 part sdd 8:48 0 14.6T 0 disk ├─sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-183 └─sdd2 8:50 0 14.6T 0 part sde 8:64 0 14.6T 0 disk ├─sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-185 └─sde2 8:66 0 14.6T 0 part sdf 8:80 0 14.6T 0 disk └─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5 253:2 0 14.6T 0 lvm sdg 8:96 0 14.6T 0 disk ├─sdg1 8:97 0 100M 0 part /var/lib/ceph/osd/ceph-186 └─sdg2 8:98 0 14.6T 0 part sdh 8:112 0 14.6T 0 disk ├─sdh1 8:113 0 100M 0 part /var/lib/ceph/osd/ceph-187 └─sdh2 8:114 0 14.6T 0 part sdi 8:128 0 14.6T 0 disk ├─sdi1 8:129 0 100M 0 part /var/lib/ceph/osd/ceph-188 └─sdi2 8:130 0 14.6T 0 part sdj 8:144 0 14.6T 0 disk ├─sdj1 8:145 0 100M 0 part /var/lib/ceph/osd/ceph-189 └─sdj2 8:146 0 14.6T 0 part sdk 8:160 0 14.6T 0 disk ├─sdk1 8:161 0 100M 0 part /var/lib/ceph/osd/ceph-190 └─sdk2 8:162 0 14.6T 0 part sdl 8:176 0 14.6T 0 disk ├─sdl1 8:177 0 100M 0 part /var/lib/ceph/osd/ceph-191 └─sdl2 8:178 0 14.6T 0 part sdm 8:192 0 14.6T 0 disk ├─sdm1 8:193 0 100M 0 part /var/lib/ceph/osd/ceph-192 └─sdm2 8:194 0 14.6T 0 part sdn 8:208 0 14.6T 0 disk ├─sdn1 8:209 0 100M 0 part /var/lib/ceph/osd/ceph-193 └─sdn2 8:210 0 14.6T 0 part sdo 8:224 0 14.6T 0 disk ├─sdo1 8:225 0 100M 0 part /var/lib/ceph/osd/ceph-194 └─sdo2 8:226 0 14.6T 0 part sdp 8:240 0 14.6T 0 disk ├─sdp1 8:241 0 100M 0 part /var/lib/ceph/osd/ceph-195 └─sdp2 8:242 0 14.6T 0 part sdq 65:0 0 14.6T 0 disk ├─sdq1 65:1 0 100M 0 part /var/lib/ceph/osd/ceph-196 └─sdq2 65:2 0 14.6T 0 part sdr 65:16 0 14.6T 0 disk ├─sdr1 65:17 0 100M 0 part /var/lib/ceph/osd/ceph-197 └─sdr2 65:18 0 14.6T 0 part sds 65:32 0 14.6T 0 disk ├─sds1 65:33 0 100M 0 part /var/lib/ceph/osd/ceph-198 └─sds2 65:34 0 14.6T 0 part sdt 65:48 0 894.3G 0 disk ├─sdt1 65:49 0 100M 0 part /var/lib/ceph/osd/ceph-199 └─sdt2 65:50 0 894.2G 0 part sdu 65:64 0 894.3G 0 disk └─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607 253:3 0 894.3G 0 lvm sdv 65:80 0 894.3G 0 disk └─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4 253:1 0 894.3G 0 lvm sdw 65:96 0 14.6T 0 disk └─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845 253:0 0 14.6T 0 lvm sdx 65:112 0 223.5G 0 disk ├─sdx1 65:113 0 1M 0 part └─sdx2 65:114 0 223.5G 0 part rbd0 252:0 0 15G 0 disk rbd1 252:16 0 6T 0 disk nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md nvme1n1 259:3 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md