Advice on sizing WAL/DB cluster for Optane and SATA SSD disks.
Hi, I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage. On each node, I have: - 1 x 512Gb M.2 SSD (for Proxmox/boot volume) - 1 x 960GB Intel Optane 905P (for Ceph WAL/DB) - 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD) I'm using the Proxmox "pveceph" command to setup the OSDs. By default this seems to pick 10% of the OSD size for the DB volume, and 1% of the OSD size for the WAL volume. This means after four drives, I ran out of space: # pveceph osd create /dev/sde -db_dev /dev/nvme0n1
create OSD on /dev/sde (bluestore) creating block.db on '/dev/nvme0n1' Rounding up size to full physical extent 178.85 GiB lvcreate 'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded' error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has insufficient free space (45784 extents): 45786 required.
Anyway, I assume that means I need to tune my DB and WAL volumes down from the defaults. What advice to you have in terms of making best use of the available space, between WAL and DB? What is the impact of having WAL and DB smaller than 1% and 10% of OSD size respectively? Thanks, Victor
WAL is 1G (you can allocate 2 to be sure), DB should always be 30G. And this doesn't depend on the size of the data partition :-) 14 марта 2020 г. 22:50:37 GMT+03:00, Victor Hooi <victorhooi@yahoo.com> пишет:
Hi,
I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage.
On each node, I have:
- 1 x 512Gb M.2 SSD (for Proxmox/boot volume) - 1 x 960GB Intel Optane 905P (for Ceph WAL/DB) - 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD)
I'm using the Proxmox "pveceph" command to setup the OSDs.
By default this seems to pick 10% of the OSD size for the DB volume, and 1% of the OSD size for the WAL volume.
This means after four drives, I ran out of space:
# pveceph osd create /dev/sde -db_dev /dev/nvme0n1
create OSD on /dev/sde (bluestore) creating block.db on '/dev/nvme0n1' Rounding up size to full physical extent 178.85 GiB lvcreate
'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded'
error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has insufficient free space (45784 extents): 45786 required.
Anyway, I assume that means I need to tune my DB and WAL volumes down from the defaults.
What advice to you have in terms of making best use of the available space, between WAL and DB?
What is the impact of having WAL and DB smaller than 1% and 10% of OSD size respectively?
Thanks, Victor _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- With best regards, Vitaliy Filippov
Hi, Vitaliy - Sure, I can use those absolute values (30GB for DB, 2GB for WAL) you suggested. Currently - Proxmox is defaulting to a 178.85 GB partition for the DB/WAL. (It seems to put the DB and WAL on the same partition). Using your calculations, with 6 x OSDs per host - that means a total of 180GB for DB, 12GB for WAL = 192GB in total. (Optane drive is 960GB in capacity). Question 1 - Are there any advantages to using larger DB partition than 30GB, or larger WAL than 2GB? (Just thinking how to best use the entire Optane drive if possible). Question 2 - How do I check the WAL size in Ceph? (Proxmox seems to be putting the WAL on the same partition as the DB, but I don't know where its size is specified). Thanks, Victor On Mon, Mar 16, 2020 at 12:05 AM Виталий Филиппов <vitalif@yourcmc.ru> wrote:
WAL is 1G (you can allocate 2 to be sure), DB should always be 30G. And this doesn't depend on the size of the data partition :-)
14 марта 2020 г. 22:50:37 GMT+03:00, Victor Hooi <victorhooi@yahoo.com> пишет:
Hi,
I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage.
On each node, I have:
- 1 x 512Gb M.2 SSD (for Proxmox/boot volume) - 1 x 960GB Intel Optane 905P (for Ceph WAL/DB) - 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD)
I'm using the Proxmox "pveceph" command to setup the OSDs.
By default this seems to pick 10% of the OSD size for the DB volume, and 1% of the OSD size for the WAL volume.
This means after four drives, I ran out of space:
# pveceph osd create /dev/sde -db_dev /dev/nvme0n1
create OSD on /dev/sde (bluestore) creating block.db on '/dev/nvme0n1' Rounding up size to full physical extent 178.85 GiB lvcreate 'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded' error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has insufficient free space (45784 extents): 45786 required.
Anyway, I assume that means I need to tune my DB and WAL volumes down from the defaults.
What advice to you have in terms of making best use of the available space, between WAL and DB?
What is the impact of having WAL and DB smaller than 1% and 10% of OSD size respectively?
Thanks, Victor ------------------------------ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- With best regards, Vitaliy Filippov
Hi Victor, 1) RocksDB doesn't put L4 on the fast device if it's less than ~ 286 GB, so no. But, anyway, there's usually no L4, so 30 GB is usually sufficient. I had ~17 GB block.dbs even for 8 TB hard drives used for RBD... RGW probably uses slightly more if stored objects are small... but you're still unlikely to overflow a 30 GB partition with 2 TB OSDs. 2) WAL is the on-disk mirror of RocksDB memtables, its size is defined by bluestore_rocksdb_options. Default is max_write_buffer_number=4, write_buffer_size=256MB, thus 1GB. You don't even need to split wal and db partitions if they're on the same device. Calculations are here: https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing
Hi,
Vitaliy - Sure, I can use those absolute values (30GB for DB, 2GB for WAL) you suggested.
Currently - Proxmox is defaulting to a 178.85 GB partition for the DB/WAL. (It seems to put the DB and WAL on the same partition).
Using your calculations, with 6 x OSDs per host - that means a total of 180GB for DB, 12GB for WAL = 192GB in total. (Optane drive is 960GB in capacity).
Question 1 - Are there any advantages to using larger DB partition than 30GB, or larger WAL than 2GB? (Just thinking how to best use the entire Optane drive if possible).
Question 2 - How do I check the WAL size in Ceph? (Proxmox seems to be putting the WAL on the same partition as the DB, but I don't know where its size is specified).
Thanks, Victor
On 3/16/2020 3:25 PM, vitalif@yourcmc.ru wrote:
Hi Victor,
1) RocksDB doesn't put L4 on the fast device if it's less than ~ 286 GB, so no. But, anyway, there's usually no L4, so 30 GB is usually sufficient. I had ~17 GB block.dbs even for 8 TB hard drives used for RBD... RGW probably uses slightly more if stored objects are small... but you're still unlikely to overflow a 30 GB partition with 2 TB OSDs.
As already mentioned by Janne and given plenty of available space at NVMes I would recommend to have some spare space above 30GB at DB. In my lab I observed up to 100% transient excess under peak load when running some (pretty artificial!) benchmarks. IMO 64GB is the perfect size for combined WAL/DB volume for almost any OSD except ones that have huge main device behind and handle heavy RGW load. 300+ GB is required in that case.
2) WAL is the on-disk mirror of RocksDB memtables, its size is defined by bluestore_rocksdb_options. Default is max_write_buffer_number=4, write_buffer_size=256MB, thus 1GB. You don't even need to split wal and db partitions if they're on the same device. Calculations are here: https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing
Hi,
Vitaliy - Sure, I can use those absolute values (30GB for DB, 2GB for WAL) you suggested.
Currently - Proxmox is defaulting to a 178.85 GB partition for the DB/WAL. (It seems to put the DB and WAL on the same partition).
Using your calculations, with 6 x OSDs per host - that means a total of 180GB for DB, 12GB for WAL = 192GB in total. (Optane drive is 960GB in capacity).
Question 1 - Are there any advantages to using larger DB partition than 30GB, or larger WAL than 2GB? (Just thinking how to best use the entire Optane drive if possible).
Question 2 - How do I check the WAL size in Ceph? (Proxmox seems to be putting the WAL on the same partition as the DB, but I don't know where its size is specified).
Thanks, Victor
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den sön 15 mars 2020 kl 14:06 skrev Виталий Филиппов <vitalif@yourcmc.ru>:
WAL is 1G (you can allocate 2 to be sure), DB should always be 30G. And this doesn't depend on the size of the data partition :-)
DB should be either 3, 30 or 300 depending on how much you can spare on the fast devices. 30 is probably good in this case though. https://github.com/facebook/rocksdb/wiki/Leveled-Compaction Some say using 2x (ie, 60 in this case) helps when temp space is needed for compaction, and since the 960G will only see WAL/DB for 6 drives for now, it would only consume 180G, so you might aswell go for 60G WAL+DB, and still be able to fix 2x as many data drives in using the same sized WAL/DBs too. -- May the most significant bit of your life be positive.
participants (5)
-
Igor Fedotov
-
Janne Johansson
-
Victor Hooi
-
vitalif＠yourcmc.ru
-
Виталий Филиппов