Introduce flash OSD's to Nautilus installation
Hi, We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup. The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash. We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively . But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating. But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong? I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all. The plan is to then change the crush rule for the exiting pools to the new ones with device classes. Thank you for any pointers or advice. [root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr --- crush_comp 2020-09-17 17:11:37.125310334 +0200 +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 @@ -574,6 +574,7 @@ root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily + id -29 class ssd # weight 2087.180 alg straw2 hash 0 # rjenkins1 @@ -710,5 +711,25 @@ step chooseleaf firstn 0 type host step emit } +rule replicated_ssd { + id 15 + type replicated + min_size 1 + max_size 10 + step take default class ssd + step chooseleaf firstn 0 type host + step emit +} +rule cephyrfs_data_hdd { + id 16 + type erasure + min_size 3 + max_size 12 + step set_chooseleaf_tries 5 + step set_choose_tries 100 + step take default class hdd + step chooseleaf indep 0 type host + step emit +} # end crush map [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 1 had 0/6144 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) rule 14 had 0/10240 mismatched mappings (0) maps appear equivalent Regards, Mathias Lindberg Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin@chalmers.se
Hi, AFAIR the device types adds a bunch of shadow devices e.g. osd.1~hdd or something like that... And those shadow devs have a different crush id than the original, untyped device. So, alas, I don't think your test is complete, and yes I expect that your data would move if you change the rule properly. Cheers, Dan On Thu, 17 Sep 2020, 18:36 Mathias Lindberg, <mathlin@chalmers.se> wrote:
Hi,
We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup.
The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash.
We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively .
But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating.
But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong? I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all. The plan is to then change the crush rule for the exiting pools to the new ones with device classes.
Thank you for any pointers or advice.
[root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr --- crush_comp 2020-09-17 17:11:37.125310334 +0200 +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 @@ -574,6 +574,7 @@ root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily + id -29 class ssd # weight 2087.180 alg straw2 hash 0 # rjenkins1 @@ -710,5 +711,25 @@ step chooseleaf firstn 0 type host step emit } +rule replicated_ssd { + id 15 + type replicated + min_size 1 + max_size 10 + step take default class ssd + step chooseleaf firstn 0 type host + step emit +} +rule cephyrfs_data_hdd { + id 16 + type erasure + min_size 3 + max_size 12 + step set_chooseleaf_tries 5 + step set_choose_tries 100 + step take default class hdd + step chooseleaf indep 0 type host + step emit +}
# end crush map
[root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 1 had 0/6144 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) rule 14 had 0/10240 mismatched mappings (0) maps appear equivalent
Regards, Mathias Lindberg
Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin@chalmers.se
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
... unless you use the reclassify tooling: https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/#migrating-... On Thu, 17 Sep 2020, 19:30 Dan van der Ster, <dan@vanderster.com> wrote:
Hi,
AFAIR the device types adds a bunch of shadow devices e.g. osd.1~hdd or something like that... And those shadow devs have a different crush id than the original, untyped device.
So, alas, I don't think your test is complete, and yes I expect that your data would move if you change the rule properly.
Cheers, Dan
On Thu, 17 Sep 2020, 18:36 Mathias Lindberg, <mathlin@chalmers.se> wrote:
Hi,
We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup.
The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash.
We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively .
But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating.
But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong? I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all. The plan is to then change the crush rule for the exiting pools to the new ones with device classes.
Thank you for any pointers or advice.
[root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr --- crush_comp 2020-09-17 17:11:37.125310334 +0200 +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 @@ -574,6 +574,7 @@ root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily + id -29 class ssd # weight 2087.180 alg straw2 hash 0 # rjenkins1 @@ -710,5 +711,25 @@ step chooseleaf firstn 0 type host step emit } +rule replicated_ssd { + id 15 + type replicated + min_size 1 + max_size 10 + step take default class ssd + step chooseleaf firstn 0 type host + step emit +} +rule cephyrfs_data_hdd { + id 16 + type erasure + min_size 3 + max_size 12 + step set_chooseleaf_tries 5 + step set_choose_tries 100 + step take default class hdd + step chooseleaf indep 0 type host + step emit +}
# end crush map
[root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 1 had 0/6144 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) rule 14 had 0/10240 mismatched mappings (0) maps appear equivalent
Regards, Mathias Lindberg
Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin@chalmers.se
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2020-09-17 18:36, Mathias Lindberg wrote:
Hi,
We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup.
The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash.
It depends on how many SSDs you want to add. And if you only use them for cephfs metadata or not. If you have a lot of dirs / files in your cluster, than you will also have a lot of OMAP / META data. If you put that data on a few SSDs, that will become a bottleneck. The rocksdb database will become large. On Mimic we have seen that this can become a big issue. We moved all metadata to a subset (25%) of the cluster (NVMe) and that did not work out well. So we ended up reverting the CRUSH change. During this period your OSDs will consume a lot of CPU. TL;DR: I don't think you want to do this and think it's better to have the RocksDB data spread out accross the cluster. Especially as you already have NVMe for WAL and DB. But maybe things have changed in Nautilus, but I wouldn't bet on it.
We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively .
But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating.
But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong?
That could work: https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/ If the crushtool doesn't see a difference it should work out.
I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all.
Try it out on a (virtual) test cluster if you can (you can manually assign device classes to OSDs). Better test and understand, than putting yourself (and the cluster) under a lot of stress. Gr. Stefan
participants (3)
-
Dan van der Ster
-
Mathias Lindberg
-
Stefan Kooman