Hi Anthony, Anthony D'Atri wrote:
swap - I used 32 GB
If you need any swap, what you really need is more physmem. Don’t provision any swap at all. This isn’t 1985.
Well. Why keep lot of boot-time-only code sitting in RAM instead of having it gradually paged out? https://chrisdown.name/2018/01/02/in-defence-of-swap.html _Some_ swap space is a nice bonus. Of course, when it gets used for page-ins as well as page-outs during regular use, more RAM is needed instead.
Then I created a partition covering the rest of the free space on each NVMe and used both of them as physical volumes for a single LVM volume group:
Sharing the boot volume with data is not an ideal strategy. I have a customer who got themselves into an outage doing that. You have a zillion SAS/SATA slots empty, put a pair of SSDs into each system for boot/OS, mirror them with MD, and don’t use them for data.
For my workload, the bottleneck (if any) are the HDDs, so NVMes have lots of idle time, even with some WAL and OMAP traffic hitting them. On one of my clusters, the nodes are used both as Ceph OSDs _and_ as KVM hosts running many instances (with volumes on Ceph RBD on the very same cluster). No problem with that. Even small servers these days have insane amounts of CPU power (especially relative to what is needed for HDD-based RGW traffic), so why keep them idle and consuming electricity most of the time?
My sense is that LVM mirroring is mostly for temporary use while migrating devices, though it may actually use MD under the hood.
(I meant LVM/DM RAID, no LVM mirroring aka CONFIG_DM_MIRROR in the kernel, sorry). These days lots of RAID code is shared with DM-RAID and MD, the difference is mainly in on-disk metadata: in DM-RAID there are none, but they can be provided by user-space LVM tools, which would then setup RAID-1 device-mapper node. See, for example, here: https://elixir.bootlin.com/linux/v6.16.7/source/drivers/md/Kconfig https://elixir.bootlin.com/linux/v6.16.7/source/drivers/md/
I tend to create an MD metadevice and create LVs on top of that.
Me too, but for this cluster it can be nice to have a resizable RAID-based root. In the end I did not went for it, though. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall