28 Jun
2023
28 Jun
'23
3:03 p.m.
Hi,
is it a problem that the device class for all my disks is SSD even all of
these disks are NVME disks? If it is just a classification for ceph, so I
can have pools on SSDs and NVMEs separated I don't care. But maybe ceph
handles NVME disks differently internally?
I've added them via
ceph-volume lvm create --bluestore --data /dev/nvme2n1
and they only show up as ssd
root@a0423f621aaa:~# ceph osd metadata osd.0
{
"id": 0,
"arch": "x86_64",
...
"bluefs": "1",
"bluefs_dedicated_db": "0",
"bluefs_dedicated_wal": "0",
"bluefs_single_shared_device": "1",
"bluestore_bdev_access_mode": "blk",
"bluestore_bdev_block_size": "4096",
"bluestore_bdev_dev_node": "/dev/dm-2",
"bluestore_bdev_devices": "nvme0n1",
"bluestore_bdev_driver": "KernelDevice",
"bluestore_bdev_partition_path": "/dev/dm-2",
"bluestore_bdev_rotational": "0",
"bluestore_bdev_size": "1920378863616",
"bluestore_bdev_support_discard": "1",
"bluestore_bdev_type": "ssd",
"ceph_release": "pacific",
"ceph_version": "ceph version 16.2.13
(5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)",
"ceph_version_short": "16.2.13",
"ceph_version_when_created": "ceph version 16.2.13
(5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)",
"cpu": "Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz",
"created_at": "2023-06-20T14:03:35.167741Z",
"default_device_class": "ssd",
"device_ids":
"nvme0n1=SAMSUNG_MZQLB1T9HAJR-00007_S439NF0M506164",
"device_paths":
"nvme0n1=/dev/disk/by-path/pci-0000:5e:00.0-nvme-1",
"devices": "nvme0n1",
"distro": "ubuntu",
"distro_description": "Ubuntu 20.04.6 LTS",
"distro_version": "20.04",
...
"journal_rotational": "0",
"kernel_description": "#169-Ubuntu SMP Tue Jun 6 22:23:09 UTC
2023",
"kernel_version": "5.4.0-152-generic",
"mem_swap_kb": "0",
"mem_total_kb": "196668116",
"network_numa_unknown_ifaces": "back_iface,front_iface",
"objectstore_numa_node": "0",
"objectstore_numa_nodes": "0",
"os": "Linux",
"osd_data": "/var/lib/ceph/osd/ceph-0",
"osd_objectstore": "bluestore",
"osdspec_affinity": "",
"rotational": "0"
}
Cheers
Boris
28 Jun
28 Jun
3:19 p.m.
On 6/28/23 14:03, Boris Behrens wrote:
is it a problem that the device class for all my disks
is SSD even all of
these disks are NVME disks? If it is just a classification for ceph, so I
can have pools on SSDs and NVMEs separated I don't care. But maybe ceph
handles NVME disks differently internally?
No. When creating the OSD ceph-volume looks at
/sys/class/block/DEVICE/queue/rotational
to determine if it's an HDD (file contains 1) or not (file contains 0).
If you need to distinguish between SSD and NVMe you can manually assign
another device class to the OSDs.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
3:27 p.m.
Hi,
is it a problem that the device class for all my disks is SSD even all
of
these disks are NVME disks? If it is just a classification for ceph, so
I
can have pools on SSDs and NVMEs separated I don't care. But maybe ceph
handles NVME disks differently internally?
I am not entirely sure if it is still with newer versions of ceph, but what I understood
from previous discussions is that it does not really pay of to use nvme as an osd and they
are mostly used for speeding up writes to slower osds (hdd)
https://yourcmc.ru/wiki/Ceph_performance
5:02 p.m.
That page has mixed info.
What would we use instead? SATA / SAS that are progressively withering in the market, less
performance for the same money? Why pay extra for an HBA just to use legacy media?
You can use NVMe for WAL+DB, with more complexity. You’ll get faster metadata and lower
latency on some small writes, but in the end the bulk drives are the bottleneck.
All-NVMe chassis are the only way, legacy interfaces are a false economy.
Ceph doesn’t seem to automatically class nvme devices but one can easily change the device
class on an OSD
> On Jun 28, 2023, at 8:29 AM, Marc <Marc(a)f1-outsourcing.eu> wrote:
>
>
>>
>> Hi,
>> is it a problem that the device class for all my disks is SSD even all
>> of
>> these disks are NVME disks? If it is just a classification for ceph, so
>> I
>> can have pools on SSDs and NVMEs separated I don't care. But maybe ceph
>> handles NVME disks differently internally?
>>
>
> I am not entirely sure if it is still with newer versions of ceph, but what I
understood from previous discussions is that it does not really pay of to use nvme as an
osd and they are mostly used for speeding up writes to slower osds (hdd)
>
> https://yourcmc.ru/wiki/Ceph_performance
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:14 p.m.
Even when you factor in density, iops, and the cost of an HBA?
SAS is mostly dead, manufacturers are beginning to drop SATA from their roadmaps.
> On Jun 28, 2023, at 10:24 AM, Marc <Marc(a)f1-outsourcing.eu> wrote:
>
>
>
>>
>> What would we use instead? SATA / SAS that are progressively withering
>> in the market, less performance for the same money? Why pay extra for an
>> HBA just to use legacy media?
>
> I am still buying sas/sata ssd's, these are for me still ~half price of the nvme
equivalent.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 Jun
29 Jun
1:21 p.m.
So basically it does not matter unless I want to have that split up.
Thanks for all the answers.
I am still lobbying to phase out SATA SSDs and replace them with NVME
disks. :)
Am Mi., 28. Juni 2023 um 18:14 Uhr schrieb Anthony D'Atri <
aad(a)dreamsnake.net>gt;:
Even when you factor in density, iops, and the cost of
an HBA?
SAS is mostly dead, manufacturers are beginning to drop SATA from their
roadmaps.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
On Jun 28, 2023, at 10:24 AM, Marc
<Marc(a)f1-outsourcing.eu> wrote:
the
nvme equivalent.
What would we use instead? SATA / SAS that are progressively withering
in the market, less performance for the same money? Why pay extra for an
HBA just to use legacy media?
I am still buying sas/sata ssd's, these are for me still ~half price of
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
326
days inactive
327
days old
6 comments
5 participants
participants (5)
-
Anthony D'Atri -
Anthony D'Atri -
Boris Behrens -
Marc -
Robert Sander