Adding dev@ceph.io On Thu, 21 Nov 2019, Muhammad Ahmad wrote:
While trying to research how crush maps are used/modified I stumbled upon these device classes. https://ceph.io/community/new-luminous-crush-device-classes/
I wanted to highlight that having nvme as a separate class will eventually break and should be removed.
There is already a push within the industry to consolidate future command sets and NVMe will likely be it. In other words, NVMe HDDs are not too far off. In fact, the recent October OCP F2F discussed this topic in detail.
If the classification is based on performance then command set (SATA/SAS/NVMe) is probably not the right classification.
I opened a PR that does this: https://github.com/ceph/ceph/pull/31796 I can't remember seeing 'nvme' as a device class on any real cluster; the exceptoin is my basement one, and I think the only reason it ended up that way was because I deployed bluestore *very* early on (with ceph-disk) and the is_nvme() detection helper doesn't work with LVM. That's my theory at least.. can anybody with bluestore on NVMe devices confirm? Does anybody see class 'nvme' devices in their cluster? Thanks! sage
On 11/21/19 4:25 PM, Sage Weil wrote:
Adding dev@ceph.io
On Thu, 21 Nov 2019, Muhammad Ahmad wrote:
While trying to research how crush maps are used/modified I stumbled upon these device classes. https://ceph.io/community/new-luminous-crush-device-classes/
I wanted to highlight that having nvme as a separate class will eventually break and should be removed.
There is already a push within the industry to consolidate future command sets and NVMe will likely be it. In other words, NVMe HDDs are not too far off. In fact, the recent October OCP F2F discussed this topic in detail.
If the classification is based on performance then command set (SATA/SAS/NVMe) is probably not the right classification. I opened a PR that does this:
https://github.com/ceph/ceph/pull/31796
I can't remember seeing 'nvme' as a device class on any real cluster; the exceptoin is my basement one, and I think the only reason it ended up that way was because I deployed bluestore *very* early on (with ceph-disk) and the is_nvme() detection helper doesn't work with LVM. That's my theory at least.. can anybody with bluestore on NVMe devices confirm? Does anybody see class 'nvme' devices in their cluster?
Thanks! sage
Here's what we've got on the new performance nodes with Intel NVMe drives: ID CLASS WEIGHT TYPE NAME -1 64.00000 root default -3 64.00000 rack localrack -2 8.00000 host o03 0 ssd 1.00000 osd.0 1 ssd 1.00000 osd.1 2 ssd 1.00000 osd.2 3 ssd 1.00000 osd.3 4 ssd 1.00000 osd.4 5 ssd 1.00000 osd.5 6 ssd 1.00000 osd.6 7 ssd 1.00000 osd.7 Mark
On 11/21/19 4:46 PM, Mark Nelson wrote:
On 11/21/19 4:25 PM, Sage Weil wrote:
Adding dev@ceph.io
On Thu, 21 Nov 2019, Muhammad Ahmad wrote:
While trying to research how crush maps are used/modified I stumbled upon these device classes. https://ceph.io/community/new-luminous-crush-device-classes/
I wanted to highlight that having nvme as a separate class will eventually break and should be removed.
There is already a push within the industry to consolidate future command sets and NVMe will likely be it. In other words, NVMe HDDs are not too far off. In fact, the recent October OCP F2F discussed this topic in detail.
If the classification is based on performance then command set (SATA/SAS/NVMe) is probably not the right classification. I opened a PR that does this:
https://github.com/ceph/ceph/pull/31796
I can't remember seeing 'nvme' as a device class on any real cluster; the exceptoin is my basement one, and I think the only reason it ended up that way was because I deployed bluestore *very* early on (with ceph-disk) and the is_nvme() detection helper doesn't work with LVM. That's my theory at least.. can anybody with bluestore on NVMe devices confirm? Does anybody see class 'nvme' devices in their cluster?
Thanks! sage
Here's what we've got on the new performance nodes with Intel NVMe drives:
ID CLASS WEIGHT TYPE NAME -1 64.00000 root default -3 64.00000 rack localrack -2 8.00000 host o03 0 ssd 1.00000 osd.0 1 ssd 1.00000 osd.1 2 ssd 1.00000 osd.2 3 ssd 1.00000 osd.3 4 ssd 1.00000 osd.4 5 ssd 1.00000 osd.5 6 ssd 1.00000 osd.6 7 ssd 1.00000 osd.7
Mark
I should probably clarify that this cluster was built with cbt though! Mark
We ssd device class on rook-ceph built clusters on m5 instances (devices appear as nvme) On Thu, Nov 21, 2019 at 2:48 PM Mark Nelson <mnelson@redhat.com> wrote:
On 11/21/19 4:46 PM, Mark Nelson wrote:
On 11/21/19 4:25 PM, Sage Weil wrote:
Adding dev@ceph.io
On Thu, 21 Nov 2019, Muhammad Ahmad wrote:
While trying to research how crush maps are used/modified I stumbled upon these device classes. https://ceph.io/community/new-luminous-crush-device-classes/
I wanted to highlight that having nvme as a separate class will eventually break and should be removed.
There is already a push within the industry to consolidate future command sets and NVMe will likely be it. In other words, NVMe HDDs are not too far off. In fact, the recent October OCP F2F discussed this topic in detail.
If the classification is based on performance then command set (SATA/SAS/NVMe) is probably not the right classification. I opened a PR that does this:
https://github.com/ceph/ceph/pull/31796
I can't remember seeing 'nvme' as a device class on any real cluster; the exceptoin is my basement one, and I think the only reason it ended up that way was because I deployed bluestore *very* early on (with ceph-disk) and the is_nvme() detection helper doesn't work with LVM. That's my theory at least.. can anybody with bluestore on NVMe devices confirm? Does anybody see class 'nvme' devices in their cluster?
Thanks! sage
Here's what we've got on the new performance nodes with Intel NVMe drives:
ID CLASS WEIGHT TYPE NAME -1 64.00000 root default -3 64.00000 rack localrack -2 8.00000 host o03 0 ssd 1.00000 osd.0 1 ssd 1.00000 osd.1 2 ssd 1.00000 osd.2 3 ssd 1.00000 osd.3 4 ssd 1.00000 osd.4 5 ssd 1.00000 osd.5 6 ssd 1.00000 osd.6 7 ssd 1.00000 osd.7
Mark
I should probably clarify that this cluster was built with cbt though!
Mark _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
The nvme devices show up as ssd, so I have to manually reclassify them on my cluster. Sent from my iPhone. Typos are Apple's fault.
On Nov 21, 2019, at 4:10 PM, Kyle Bader <kyle.bader@gmail.com> wrote:
We ssd device class on rook-ceph built clusters on m5 instances (devices appear as nvme)
On Thu, Nov 21, 2019 at 2:48 PM Mark Nelson <mnelson@redhat.com> wrote:
On 11/21/19 4:46 PM, Mark Nelson wrote: On 11/21/19 4:25 PM, Sage Weil wrote:
Adding dev@ceph.io
On Thu, 21 Nov 2019, Muhammad Ahmad wrote:
While trying to research how crush maps are used/modified I stumbled upon these device classes. https://ceph.io/community/new-luminous-crush-device-classes/
I wanted to highlight that having nvme as a separate class will eventually break and should be removed.
There is already a push within the industry to consolidate future command sets and NVMe will likely be it. In other words, NVMe HDDs are not too far off. In fact, the recent October OCP F2F discussed this topic in detail.
If the classification is based on performance then command set (SATA/SAS/NVMe) is probably not the right classification. I opened a PR that does this:
https://github.com/ceph/ceph/pull/31796
I can't remember seeing 'nvme' as a device class on any real cluster; the exceptoin is my basement one, and I think the only reason it ended up that way was because I deployed bluestore *very* early on (with ceph-disk) and the is_nvme() detection helper doesn't work with LVM. That's my theory at least.. can anybody with bluestore on NVMe devices confirm? Does anybody see class 'nvme' devices in their cluster?
Thanks! sage
Here's what we've got on the new performance nodes with Intel NVMe drives:
ID CLASS WEIGHT TYPE NAME -1 64.00000 root default -3 64.00000 rack localrack -2 8.00000 host o03 0 ssd 1.00000 osd.0 1 ssd 1.00000 osd.1 2 ssd 1.00000 osd.2 3 ssd 1.00000 osd.3 4 ssd 1.00000 osd.4 5 ssd 1.00000 osd.5 6 ssd 1.00000 osd.6 7 ssd 1.00000 osd.7
Mark
I should probably clarify that this cluster was built with cbt though!
Mark _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thu, Nov 21, 2019 at 11:25 PM Sage Weil <sage@newdream.net> wrote:
way was because I deployed bluestore *very* early on (with ceph-disk) and the is_nvme() detection helper doesn't work with LVM. That's my theory at least.. can anybody with bluestore on NVMe devices confirm? Does anybody see class 'nvme' devices in their cluster?
nope, never saw any disks automatically detected as "nvme". However, lots of users set the device class explicitly for NVMe disks (we do ask about the device class in our UI when creating OSDs, the default being 'auto detect') Paul
Thanks! sage
Hello.
22 нояб. 2019 г., в 01:25, Sage Weil <sage@newdream.net> написал(а):
Adding dev@ceph.io Does anybody see class 'nvme' devices in their cluster?
Thanks! sage
This is my production Luminous cluster: [root@r1flash1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 130.98889 root default -3 21.83148 host r1flash1 0 nvme 1.81929 osd.0 up 1.00000 1.00000 1 nvme 1.81929 osd.1 up 1.00000 1.00000 2 nvme 1.81929 osd.2 up 1.00000 1.00000 3 nvme 1.81929 osd.3 up 1.00000 1.00000 4 nvme 1.81929 osd.4 up 1.00000 1.00000 5 nvme 1.81929 osd.5 up 1.00000 1.00000 6 nvme 1.81929 osd.6 up 1.00000 1.00000 7 nvme 1.81929 osd.7 up 1.00000 1.00000 8 nvme 1.81929 osd.8 up 1.00000 1.00000 9 nvme 1.81929 osd.9 up 1.00000 1.00000 10 nvme 1.81929 osd.10 up 1.00000 1.00000 11 nvme 1.81929 osd.11 up 1.00000 1.00000 … 6 nodes, 6 Intel NVMe drives per server and 2 OSD per drive. An OSD was created with custom a script, no use LVM at all, no use ceph-disk or ceph-volume. A part create an OSD in script: <cut> # ID=$(echo "{\"cephx_secret\": \"$OSD_SECRET\"}" | ceph osd new $UUID -i - -n client.bootstrap-osd -k /var/lib/ceph/bootstrap-osd/ceph.keyring) sudo -u ceph mkdir /var/lib/ceph/osd/ceph-$ID ceph-authtool --create-keyring /var/lib/ceph/osd/ceph-$ID/keyring --name osd.$ID --add-key $OSD_SECRET echo bluestore > /var/lib/ceph/osd/ceph-$ID/type ln -s /dev/disk/by-partuuid/$PARTUUID /var/lib/ceph/osd/ceph-$ID/block ln -s /dev/disk/by-partuuid/$PARTUUID_DB /var/lib/ceph/osd/ceph-$ID/block.db chown ceph:ceph /var/lib/ceph/osd/ceph-$ID chown ceph:ceph /var/lib/ceph/osd/ceph-$ID/* chmod 600 /var/lib/ceph/osd/ceph-$ID/keyring chmod 600 /var/lib/ceph/osd/ceph-$ID/type ceph-osd -i $ID --mkfs --osd-uuid $UUID chown ceph:ceph /var/lib/ceph/osd/ceph-$ID/* <cut> We didn’t use LVM for maximize IO performance and latency and use the script because the ceph-volume don’t support RAW devices by now. — Mike, runs!
participants (6)
-
David Byte
-
Kyle Bader
-
Mark Nelson
-
Mike A
-
Paul Emmerich
-
Sage Weil