Hello Anthony. Good to hear from you. You miss understood somehow, I don't use filestore, I have XFS based bluestore and LVM based bluestore mixed in the same Nautilus cluster. Yes NVME's are mdadm raid1. I'm %100 agree with you and when I'm gonna rebuild, I will pin 1/2 to Nvme1 and 2/2 to Nvme2 without mirror. I build this system with raid1 nvme because it was one time project and I wanted to be safe rather than fast. Anthony D'Atri <anthony.datri@gmail.com>, 30 Eyl 2024 Pzt, 22:10 tarihinde şunu yazdı:
BlueStore vs Filestore doesn’t matter beyond each OSD. Filestore is very deprecated so you’ll want to redeploy any Filestore OSDs when you can. `ceph osd metadata` can survey.
I’ve had multiple issues over time with the MG spinners fwiw. For what SAS spinners cost with some effort you can get SATA SSDs for legacy systems. Do you have those NVMe drives for WAL+DB mirrored? You generally don’t want to go higher than 10:1
On Sep 30, 2024, at 3:00 PM, Özkan Göksu <ozkangksu@gmail.com> wrote:
Hello folks! I hope you are doing well :)
I have a general question about XFS and LVM backend OSD performance and possible effects if they are used together in the same pool.
I built a cluster 5 years ago with Nautilus and I used the XFS backend for OSD's. After 5 years they reached me back with ERR state and I see 10K++ slow ops, 29 incomplete PG, 2 inconsistent PG, 34300 unfound objects and 1 osd down due to compactions problem.
At the first check-up I found some drives were replaced by others and they used LVM backend for the replaced drives without using wal+db on nvme. In the cluster I have mostly XFS backend drives wal+db on nvme and some LVM drives without wal+db.
We have Cephfs and RBD pools on SSD drives and 8+2 EC pool for RGW S3 workload. RGW index stored on SSD pool. 10 nodes: with - 21 x 16TB Toshiba MG08CA16TEY, Firmware:EJ09 | 8+2 EC RGW DATA POOL - 3 x 960GB MZILS960HEHP/007 Firm: GXL0 | Rep 2 RGW index pool - 2 x PM1725B 1.6T PCI-E NVME | 50G WAL+DB for 21x HDD
Total HDD raw size 2.8PiB, SSD size 26TiB
I started fixing all the problems one by one and I'm gonna recreate these LVM drives without wal+db and I wonder 2 questions: 1- Are there any speed or latency differences on XFS and LVM backend OSD's for 16TB 7200rpm NL-SAS drives. 2- Mixing XFS and LVM backend on the same cluster does have any negative effect or problems?
Best regards:
Extra note: If you wonder please check the LSBLK output for 1/10 server: NODE-01# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 14.6T 0 disk ├─sda1 8:1 0 100M 0 part /var/lib/ceph/osd/ceph-180 └─sda2 8:2 0 14.6T 0 part sdb 8:16 0 14.6T 0 disk ├─sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-181 └─sdb2 8:18 0 14.6T 0 part sdc 8:32 0 14.6T 0 disk ├─sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-182 └─sdc2 8:34 0 14.6T 0 part sdd 8:48 0 14.6T 0 disk ├─sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-183 └─sdd2 8:50 0 14.6T 0 part sde 8:64 0 14.6T 0 disk ├─sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-185 └─sde2 8:66 0 14.6T 0 part sdf 8:80 0 14.6T 0 disk
└─ceph--ef5bd394--8dc9--46a8--a244--0c5d3c1400e3-osd--block--b69c0802--9634--43a5--b4a9--0f36cd8690c5
253:2 0 14.6T 0 lvm sdg 8:96 0 14.6T 0 disk ├─sdg1 8:97 0 100M 0 part /var/lib/ceph/osd/ceph-186 └─sdg2 8:98 0 14.6T 0 part sdh 8:112 0 14.6T 0 disk ├─sdh1 8:113 0 100M 0 part /var/lib/ceph/osd/ceph-187 └─sdh2 8:114 0 14.6T 0 part sdi 8:128 0 14.6T 0 disk ├─sdi1 8:129 0 100M 0 part /var/lib/ceph/osd/ceph-188 └─sdi2 8:130 0 14.6T 0 part sdj 8:144 0 14.6T 0 disk ├─sdj1 8:145 0 100M 0 part /var/lib/ceph/osd/ceph-189 └─sdj2 8:146 0 14.6T 0 part sdk 8:160 0 14.6T 0 disk ├─sdk1 8:161 0 100M 0 part /var/lib/ceph/osd/ceph-190 └─sdk2 8:162 0 14.6T 0 part sdl 8:176 0 14.6T 0 disk ├─sdl1 8:177 0 100M 0 part /var/lib/ceph/osd/ceph-191 └─sdl2 8:178 0 14.6T 0 part sdm 8:192 0 14.6T 0 disk ├─sdm1 8:193 0 100M 0 part /var/lib/ceph/osd/ceph-192 └─sdm2 8:194 0 14.6T 0 part sdn 8:208 0 14.6T 0 disk ├─sdn1 8:209 0 100M 0 part /var/lib/ceph/osd/ceph-193 └─sdn2 8:210 0 14.6T 0 part sdo 8:224 0 14.6T 0 disk ├─sdo1 8:225 0 100M 0 part /var/lib/ceph/osd/ceph-194 └─sdo2 8:226 0 14.6T 0 part sdp 8:240 0 14.6T 0 disk ├─sdp1 8:241 0 100M 0 part /var/lib/ceph/osd/ceph-195 └─sdp2 8:242 0 14.6T 0 part sdq 65:0 0 14.6T 0 disk ├─sdq1 65:1 0 100M 0 part /var/lib/ceph/osd/ceph-196 └─sdq2 65:2 0 14.6T 0 part sdr 65:16 0 14.6T 0 disk ├─sdr1 65:17 0 100M 0 part /var/lib/ceph/osd/ceph-197 └─sdr2 65:18 0 14.6T 0 part sds 65:32 0 14.6T 0 disk ├─sds1 65:33 0 100M 0 part /var/lib/ceph/osd/ceph-198 └─sds2 65:34 0 14.6T 0 part sdt 65:48 0 894.3G 0 disk ├─sdt1 65:49 0 100M 0 part /var/lib/ceph/osd/ceph-199 └─sdt2 65:50 0 894.2G 0 part sdu 65:64 0 894.3G 0 disk
└─ceph--e115ee52--429b--4631--92f8--a4282cb43938-osd--block--f4375a1a--6a0e--4797--a496--0b9ba0448607
253:3 0 894.3G 0 lvm sdv 65:80 0 894.3G 0 disk
└─ceph--bb838dc5--4eab--4461--a0cf--6439fa12706a-osd--block--f5d007a0--86b2--43ff--8750--0b24c4f823e4
253:1 0 894.3G 0 lvm sdw 65:96 0 14.6T 0 disk
└─ceph--d2858c65--6b29--43e8--ad4f--d92b004c2f37-osd--block--905c999e--dd4e--463c--a0e0--93a69ccc7845
253:0 0 14.6T 0 lvm sdx 65:112 0 223.5G 0 disk ├─sdx1 65:113 0 1M 0 part └─sdx2 65:114 0 223.5G 0 part rbd0 252:0 0 15G 0 disk rbd1 252:16 0 6T 0 disk nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md nvme1n1 259:3 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md ├─md0p3 259:6 0 50G 0 md ├─md0p4 259:7 0 50G 0 md ├─md0p5 259:8 0 50G 0 md ├─md0p6 259:9 0 50G 0 md ├─md0p7 259:10 0 50G 0 md ├─md0p8 259:11 0 50G 0 md ├─md0p9 259:12 0 50G 0 md ├─md0p10 259:13 0 50G 0 md ├─md0p11 259:14 0 50G 0 md ├─md0p12 259:15 0 50G 0 md ├─md0p13 259:16 0 50G 0 md ├─md0p14 259:17 0 50G 0 md ├─md0p15 259:18 0 50G 0 md ├─md0p16 259:19 0 50G 0 md ├─md0p17 259:20 0 50G 0 md ├─md0p18 259:21 0 50G 0 md └─md0p19 259:22 0 50G 0 md _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io