device class for nvme disk is ssd
Hi, is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally? I've added them via ceph-volume lvm create --bluestore --data /dev/nvme2n1 and they only show up as ssd root@a0423f621aaa:~# ceph osd metadata osd.0 { "id": 0, "arch": "x86_64", ... "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-2", "bluestore_bdev_devices": "nvme0n1", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_partition_path": "/dev/dm-2", "bluestore_bdev_rotational": "0", "bluestore_bdev_size": "1920378863616", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "ssd", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "ceph_version_short": "16.2.13", "ceph_version_when_created": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "cpu": "Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz", "created_at": "2023-06-20T14:03:35.167741Z", "default_device_class": "ssd", "device_ids": "nvme0n1=SAMSUNG_MZQLB1T9HAJR-00007_S439NF0M506164", "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "devices": "nvme0n1", "distro": "ubuntu", "distro_description": "Ubuntu 20.04.6 LTS", "distro_version": "20.04", ... "journal_rotational": "0", "kernel_description": "#169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023", "kernel_version": "5.4.0-152-generic", "mem_swap_kb": "0", "mem_total_kb": "196668116", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_node": "0", "objectstore_numa_nodes": "0", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "osdspec_affinity": "", "rotational": "0" } Cheers Boris
On 6/28/23 14:03, Boris Behrens wrote:
is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally?
No. When creating the OSD ceph-volume looks at /sys/class/block/DEVICE/queue/rotational to determine if it's an HDD (file contains 1) or not (file contains 0). If you need to distinguish between SSD and NVMe you can manually assign another device class to the OSDs. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally?
I am not entirely sure if it is still with newer versions of ceph, but what I understood from previous discussions is that it does not really pay of to use nvme as an osd and they are mostly used for speeding up writes to slower osds (hdd) https://yourcmc.ru/wiki/Ceph_performance
That page has mixed info. What would we use instead? SATA / SAS that are progressively withering in the market, less performance for the same money? Why pay extra for an HBA just to use legacy media? You can use NVMe for WAL+DB, with more complexity. You’ll get faster metadata and lower latency on some small writes, but in the end the bulk drives are the bottleneck. All-NVMe chassis are the only way, legacy interfaces are a false economy. Ceph doesn’t seem to automatically class nvme devices but one can easily change the device class on an OSD
On Jun 28, 2023, at 8:29 AM, Marc <Marc@f1-outsourcing.eu> wrote:
Hi, is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally?
I am not entirely sure if it is still with newer versions of ceph, but what I understood from previous discussions is that it does not really pay of to use nvme as an osd and they are mostly used for speeding up writes to slower osds (hdd)
https://yourcmc.ru/wiki/Ceph_performance _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Even when you factor in density, iops, and the cost of an HBA? SAS is mostly dead, manufacturers are beginning to drop SATA from their roadmaps.
On Jun 28, 2023, at 10:24 AM, Marc <Marc@f1-outsourcing.eu> wrote:
What would we use instead? SATA / SAS that are progressively withering in the market, less performance for the same money? Why pay extra for an HBA just to use legacy media?
I am still buying sas/sata ssd's, these are for me still ~half price of the nvme equivalent.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So basically it does not matter unless I want to have that split up. Thanks for all the answers. I am still lobbying to phase out SATA SSDs and replace them with NVME disks. :) Am Mi., 28. Juni 2023 um 18:14 Uhr schrieb Anthony D'Atri < aad@dreamsnake.net>:
Even when you factor in density, iops, and the cost of an HBA?
SAS is mostly dead, manufacturers are beginning to drop SATA from their roadmaps.
On Jun 28, 2023, at 10:24 AM, Marc <Marc@f1-outsourcing.eu> wrote:
What would we use instead? SATA / SAS that are progressively withering in the market, less performance for the same money? Why pay extra for an HBA just to use legacy media?
I am still buying sas/sata ssd's, these are for me still ~half price of the nvme equivalent.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
participants (5)
-
Anthony D'Atri
-
Anthony D'Atri
-
Boris Behrens
-
Marc
-
Robert Sander