Hi, Adrian, Adrian Sevcenco wrote:
Hi! i'm trying to estimate the size requiremenent for a wal+db volume for 26 TB HDD while trying to minimize the administrative hassle for configuration (meaning that i would prefer a wal+db volume not separate db from wal) (RGW usage, 3 node cluster)
I have a similar setup - servers with 2xNVMe and some HDDs. I would recommend aiming for more smaller servers instead of three large ones, though.
so, for a jbod that can go up to 44 drives (not all drives present, at most up to 12) i have 2x 6.4TB NVME AFAIU for metadata (db) device, i need raid as losing metadata, means losing the associated OSD .. did i get this right?
Yes. But I think the cluster setup _should_ be planned for losing an OSD or even an entire server. So I would not bother using RAID-1 for metadata.
So, what would the the best practice to map db+wal to hdd OSD? should i do a mdraid from the 2 nvme and that to split in 12 partitions?
What I did is to put the system to both NVMes on RAID-1 partitions: /boot/efi - 128 MB or something like that) / - I used 200 GB, but it is probably overkill swap - I used 32 GB Then I created a partition covering the rest of the free space on each NVMe and used both of them as physical volumes for a single LVM volume group: vgcreate nvme_vg /dev/nvme0n1p4 /dev/nvme1n1p4 Now I can create LVM-based mirrored logical volumes for local applications, should I ever need them, and non-mirrored LVs for Ceph metadata. Something like this: for i in `seq -w 01 06`; do lvcreate -n ceph_$i -L 100G nvme_vg /dev/nvme0n1p4; done for i in `seq -w 11 16`; do lvcreate -n ceph_$i -L 100G nvme_vg /dev/nvme1n1p4; done ceph-volume lvm prepare --bluestore --block.db /dev/nvme_vg/ceph_01 --data /dev/sda ceph-volume lvm prepare --bluestore --block.db /dev/nvme_vg/ceph_11 --data /dev/sdb ceph-volume lvm prepare --bluestore --block.db /dev/nvme_vg/ceph_02 --data /dev/sdc ceph-volume lvm prepare --bluestore --block.db /dev/nvme_vg/ceph_12 --data /dev/sdd ... ceph-volume lvm activate --all The alternative would be to have a mirrored /boot only, and put everything else on a NVMe-based LVM VG, using mirrored LVs for root and swap. This way root FS would be easily resizable. I don't have experience with this (I am not even sure whether Anaconda can install AlmaLinux onto mirrored LVs), so I went for more traditional md-raid instead of LVM mirror for root, swap, and /boot/efi.
should i split the nvme ssd in 12 namespaces and make individual mdraid for each and that map to osds?
I don't think it would make a measurable difference to use NVMe namespaces.
For a 6.4 TB NVME (divided to 12) and 26 TB OSD, the db/wal would be ~533 GB, so around 2.05% .. how terrible is this number for RGW usage (i get that the recommended is 4%)
At what size the data is in danger if db is too small and when it become safe but only with performance degradation?
I think block.db can safely overflow to the main data area (with a performance degradation, of course). Hope this helps, but as always, your mileage may vary :-) -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall