14 Mar
2020
14 Mar
'20
10:50 p.m.
Hi,
I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage.
On each node, I have:
- 1 x 512Gb M.2 SSD (for Proxmox/boot volume)
- 1 x 960GB Intel Optane 905P (for Ceph WAL/DB)
- 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD)
I'm using the Proxmox "pveceph" command to setup the OSDs.
By default this seems to pick 10% of the OSD size for the DB volume, and 1%
of the OSD size for the WAL volume.
This means after four drives, I ran out of space:
# pveceph osd create /dev/sde -db_dev /dev/nvme0n1
create OSD on /dev/sde (bluestore)
creating block.db on '/dev/nvme0n1'
Rounding up size to full physical extent 178.85 GiB
lvcreate
'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded'
error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has
insufficient free space (45784 extents): 45786 required.
Anyway, I assume that means I need to tune my DB and WAL volumes down from
the defaults.
What advice to you have in terms of making best use of the available space,
between WAL and DB?
What is the impact of having WAL and DB smaller than 1% and 10% of OSD size
respectively?
Thanks,
Victor
15 Mar
15 Mar
4:05 p.m.
WAL is 1G (you can allocate 2 to be sure), DB should always be 30G. And this doesn't
depend on the size of the data partition :-)
14 марта 2020 г. 22:50:37 GMT+03:00, Victor Hooi <victorhooi(a)yahoo.com> пишет:
Hi,
I'm building a 4-node Proxmox cluster, with Ceph for the VM disk
storage.
On each node, I have:
- 1 x 512Gb M.2 SSD (for Proxmox/boot volume)
- 1 x 960GB Intel Optane 905P (for Ceph WAL/DB)
- 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD)
I'm using the Proxmox "pveceph" command to setup the OSDs.
By default this seems to pick 10% of the OSD size for the DB volume,
and 1%
of the OSD size for the WAL volume.
This means after four drives, I ran out of space:
# pveceph osd create /dev/sde -db_dev /dev/nvme0n1
--
With best regards,
Vitaliy Filippov
create OSD on /dev/sde (bluestore)
creating block.db on '/dev/nvme0n1'
Rounding up size to full physical extent 178.85 GiB
lvcreate
'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded'
error: Volume group
"ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has
insufficient free space (45784 extents): 45786 required.
Anyway, I assume that means I need to tune my DB and WAL volumes down
from
the defaults.
What advice to you have in terms of making best use of the available
space,
between WAL and DB?
What is the impact of having WAL and DB smaller than 1% and 10% of OSD
size
respectively?
Thanks,
Victor
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:49 p.m.
Hi,
Vitaliy - Sure, I can use those absolute values (30GB for DB, 2GB for WAL)
you suggested.
Currently - Proxmox is defaulting to a 178.85 GB partition for the DB/WAL.
(It seems to put the DB and WAL on the same partition).
Using your calculations, with 6 x OSDs per host - that means a total of
180GB for DB, 12GB for WAL = 192GB in total. (Optane drive is 960GB in
capacity).
Question 1 - Are there any advantages to using larger DB partition than
30GB, or larger WAL than 2GB? (Just thinking how to best use the entire
Optane drive if possible).
Question 2 - How do I check the WAL size in Ceph? (Proxmox seems to be
putting the WAL on the same partition as the DB, but I don't know where its
size is specified).
Thanks,
Victor
On Mon, Mar 16, 2020 at 12:05 AM Виталий Филиппов <vitalif(a)yourcmc.ru>
wrote:
WAL is 1G (you can allocate 2 to be sure), DB should
always be 30G. And
this doesn't depend on the size of the data partition :-)
14 марта 2020 г. 22:50:37 GMT+03:00, Victor Hooi <victorhooi(a)yahoo.com>
пишет:
Hi,
I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage.
On each node, I have:
- 1 x 512Gb M.2 SSD (for Proxmox/boot volume)
- 1 x 960GB Intel Optane 905P (for Ceph WAL/DB)
- 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD)
I'm using the Proxmox "pveceph" command to setup the OSDs.
By default this seems to pick 10% of the OSD size for the DB volume, and 1%
of the OSD size for the WAL volume.
This means after four drives, I ran out of space:
# pveceph osd create /dev/sde -db_dev /dev/nvme0n1
--
With best regards,
Vitaliy Filippov
create OSD on /dev/sde (bluestore)
creating block.db on '/dev/nvme0n1'
Rounding up size to full physical extent 178.85 GiB
lvcreate
'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded'
error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has
insufficient free space (45784 extents): 45786 required.
Anyway, I assume that means I need to tune my DB and WAL volumes down from
the defaults.
What advice to you have in terms of making best use of the available space,
between WAL and DB?
What is the impact of having WAL and DB smaller than 1% and 10% of OSD size
respectively?
Thanks,
Victor
------------------------------
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 Mar
16 Mar
3:25 p.m.
Hi Victor,
1) RocksDB doesn't put L4 on the fast device if it's less than ~ 286 GB,
so no. But, anyway, there's usually no L4, so 30 GB is usually
sufficient. I had ~17 GB block.dbs even for 8 TB hard drives used for
RBD... RGW probably uses slightly more if stored objects are small...
but you're still unlikely to overflow a 30 GB partition with 2 TB OSDs.
2) WAL is the on-disk mirror of RocksDB memtables, its size is defined
by bluestore_rocksdb_options. Default is max_write_buffer_number=4,
write_buffer_size=256MB, thus 1GB. You don't even need to split wal and
db partitions if they're on the same device. Calculations are here:
https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing
> Hi,
>
> Vitaliy - Sure, I can use those absolute values (30GB for DB, 2GB for
> WAL) you suggested.
>
> Currently - Proxmox is defaulting to a 178.85 GB partition for the
> DB/WAL. (It seems to put the DB and WAL on the same partition).
>
> Using your calculations, with 6 x OSDs per host - that means a total
> of 180GB for DB, 12GB for WAL = 192GB in total. (Optane drive is 960GB
> in capacity).
>
> Question 1 - Are there any advantages to using larger DB partition
> than 30GB, or larger WAL than 2GB? (Just thinking how to best use the
> entire Optane drive if possible).
>
> Question 2 - How do I check the WAL size in Ceph? (Proxmox seems to be
> putting the WAL on the same partition as the DB, but I don't know
> where its size is specified).
>
> Thanks,
> Victor
6:03 p.m.
On 3/16/2020 3:25 PM, vitalif(a)yourcmc.ru wrote:
Hi Victor,
1) RocksDB doesn't put L4 on the fast device if it's less than ~ 286
GB, so no. But, anyway, there's usually no L4, so 30 GB is usually
sufficient. I had ~17 GB block.dbs even for 8 TB hard drives used for
RBD... RGW probably uses slightly more if stored objects are small...
but you're still unlikely to overflow a 30 GB partition with 2 TB OSDs.
As already mentioned by Janne and given plenty of available space at
NVMes I would recommend to have some spare space above 30GB at DB. In my
lab I observed up to 100% transient excess under peak load when running
some (pretty artificial!) benchmarks.
IMO 64GB is the perfect size for combined WAL/DB volume for almost any
OSD except ones that have huge main device behind and handle heavy RGW
load. 300+ GB is required in that case.
>
> 2) WAL is the on-disk mirror of RocksDB memtables, its size is defined
> by bluestore_rocksdb_options. Default is max_write_buffer_number=4,
> write_buffer_size=256MB, thus 1GB. You don't even need to split wal
> and db partitions if they're on the same device. Calculations are
> here: https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing
>
>> Hi,
>>
>> Vitaliy - Sure, I can use those absolute values (30GB for DB, 2GB for
>> WAL) you suggested.
>>
>> Currently - Proxmox is defaulting to a 178.85 GB partition for the
>> DB/WAL. (It seems to put the DB and WAL on the same partition).
>>
>> Using your calculations, with 6 x OSDs per host - that means a total
>> of 180GB for DB, 12GB for WAL = 192GB in total. (Optane drive is 960GB
>> in capacity).
>>
>> Question 1 - Are there any advantages to using larger DB partition
>> than 30GB, or larger WAL than 2GB? (Just thinking how to best use the
>> entire Optane drive if possible).
>>
>> Question 2 - How do I check the WAL size in Ceph? (Proxmox seems to be
>> putting the WAL on the same partition as the DB, but I don't know
>> where its size is specified).
>>
>> Thanks,
>> Victor
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:55 a.m.
Den sön 15 mars 2020 kl 14:06 skrev Виталий Филиппов <vitalif(a)yourcmc.ru>ru>:
WAL is 1G (you can allocate 2 to be sure), DB should
always be 30G. And
this doesn't depend on the size of the data partition :-)
DB should be either 3, 30 or 300 depending on how much you can spare on the
fast devices. 30 is probably good in this case though.
https://github.com/facebook/rocksdb/wiki/Leveled-Compaction
Some say using 2x (ie, 60 in this case) helps when temp space is needed for
compaction, and since the 960G will only see WAL/DB for 6 drives for now,
it would only consume
180G, so you might aswell go for 60G WAL+DB, and still be able to fix 2x as
many data drives in using the same sized WAL/DBs too.
--
May the most significant bit of your life be positive.
1510
days inactive
1512
days old
5 comments
5 participants
participants (5)
-
Igor Fedotov -
Janne Johansson -
Victor Hooi -
vitalif＠yourcmc.ru -
Виталий Филиппов