How to use ceph-volume to create multiple OSDs per NVMe disk, and with fixed WAL/DB partition on another device?
I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5). Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB). CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare. My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions). However, I am struggling to get the right ceph-volume command to achieve this. Thanks to a very kind Redditor, I was able to get close: /dev/nvme0n1 is an Optane device (900GB). /dev/nvme2n1 is an Intel NVMe SSD (4TB). ``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1 Total OSDs: 4 Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ``` This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions). Is there any way to specify the WAL/DB partition size in the above? And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? (Or if there's an easier way to achieve the above layout, please let me know). That being said - I also just saw this ceph-users thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZMX... It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share?
There's an option for the block.db size in ceph-volume batch command: --block-db-size BLOCK_DB_SIZE Set (or override) the "bluestore_block_db_size" value, in bytes Zitat von victorhooi@yahoo.com:
I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5).
Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB).
CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare.
My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions).
However, I am struggling to get the right ceph-volume command to achieve this.
Thanks to a very kind Redditor, I was able to get close:
/dev/nvme0n1 is an Optane device (900GB).
/dev/nvme2n1 is an Intel NVMe SSD (4TB).
``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1
Total OSDs: 4
Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1
Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ```
This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions).
Is there any way to specify the WAL/DB partition size in the above?
And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk?
(Or if there's an easier way to achieve the above layout, please let me know).
That being said - I also just saw this ceph-users thread:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZMX...
It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Nov 06, 2020 at 10:15:52AM -0000, victorhooi@yahoo.com wrote:
I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5).
Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB).
CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare.
My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions).
However, I am struggling to get the right ceph-volume command to achieve this.
Thanks to a very kind Redditor, I was able to get close:
/dev/nvme0n1 is an Optane device (900GB).
/dev/nvme2n1 is an Intel NVMe SSD (4TB).
``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1
Total OSDs: 4
Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1
Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ```
This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions).
Is there any way to specify the WAL/DB partition size in the above?
And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? Is there is particular reason you want to run ceph-volume multiple times? The batch subcommand can handle that in one go without the need to explicitly specify any sizes as another reply proposed (though that will work nicely).
Something like this should get you there: ceph-volume lvm batch --osds-per-device 4 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --db-devices /dev/nvme0n1 This of course makes assumption regarding device names, adjust accordingly. Another option to size the volumes on the Optane drive would be to rely on the *slots arguments of the batch subcommand. See either ceph-volume lvm batch --help or https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#implicit-sizing
(Or if there's an easier way to achieve the above layout, please let me know).
That being said - I also just saw this ceph-users thread:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZMX...
It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have a similar setup and have been running some large concurrent benchmarks and I am seeing that running multiple OSDs per NVME doesn't really make a lot of difference. In fact, it actually increases the write amplification if you have write-heavy workloads, so performance degrades over time. Also, if you have powerful (and enough) CPU and memory on the system, ingest throttling and replication becomes the bottleneck and not the NVME writes. Although, it might make a difference if you have read heavy workloads. I haven't tested that enough. Regards, Shridhar On Wed, 11 Nov 2020 at 00:27, Jan Fajerski <jfajerski@suse.com> wrote:
I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5).
Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB).
CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare.
My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of
On Fri, Nov 06, 2020 at 10:15:52AM -0000, victorhooi@yahoo.com wrote: the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions).
However, I am struggling to get the right ceph-volume command to achieve
this.
Thanks to a very kind Redditor, I was able to get close:
/dev/nvme0n1 is an Optane device (900GB).
/dev/nvme2n1 is an Intel NVMe SSD (4TB).
``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices
/dev/nvme0n1
Total OSDs: 4
Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1
Type Path
LV Size % of device
---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25%
---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25%
---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25%
---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ```
This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions).
Is there any way to specify the WAL/DB partition size in the above?
And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? Is there is particular reason you want to run ceph-volume multiple times? The batch subcommand can handle that in one go without the need to explicitly specify any sizes as another reply proposed (though that will work nicely).
Something like this should get you there: ceph-volume lvm batch --osds-per-device 4 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --db-devices /dev/nvme0n1
This of course makes assumption regarding device names, adjust accordingly.
Another option to size the volumes on the Optane drive would be to rely on the *slots arguments of the batch subcommand. See either ceph-volume lvm batch --help or https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#implicit-sizing
(Or if there's an easier way to achieve the above layout, please let me
know).
That being said - I also just saw this ceph-users thread:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZMX...
It talks there about "osd op num shards" and "osd op num threads per
shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Quoting in your message looks kind of messy so forgive me if I’m propagating that below. Honestly I agree that the Optanes will give diminishing returns at best for all but the most extreme workloads (which will probably want to use NVMoF natively anyway).
This does split up the NVMe disk into 4 OSDs, and creates WAL/DB
partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions).
Because you want to use the rest of the space on the Optane for something else?
It talks there about "osd op num shards" and "osd op num threads per
shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share?
I’d like to see that too, since I’m curious if this is not still limited by per-OSD serialization. Those options have been around for years haven’t they? I’m also curious how splitting an NVMe device into multipe OSDs could affect write amplification. — aad
participants (5)
-
Anthony D'Atri
-
Eugen Block
-
Jan Fajerski
-
victorhooi＠yahoo.com
-
Void Star Nill