Hi, We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup. The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash. We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively . But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating. But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong? I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all. The plan is to then change the crush rule for the exiting pools to the new ones with device classes. Thank you for any pointers or advice. [root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr --- crush_comp 2020-09-17 17:11:37.125310334 +0200 +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 @@ -574,6 +574,7 @@ root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily + id -29 class ssd # weight 2087.180 alg straw2 hash 0 # rjenkins1 @@ -710,5 +711,25 @@ step chooseleaf firstn 0 type host step emit } +rule replicated_ssd { + id 15 + type replicated + min_size 1 + max_size 10 + step take default class ssd + step chooseleaf firstn 0 type host + step emit +} +rule cephyrfs_data_hdd { + id 16 + type erasure + min_size 3 + max_size 12 + step set_chooseleaf_tries 5 + step set_choose_tries 100 + step take default class hdd + step chooseleaf indep 0 type host + step emit +} # end crush map [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 1 had 0/6144 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) rule 14 had 0/10240 mismatched mappings (0) maps appear equivalent Regards, Mathias Lindberg Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin@chalmers.se