I held off replying here hoping that someone more authoritative would step in, but I have a few thoughts that might help or stimulate conversation.
The recommendations for cephfs is to make a replicated default data pool, and adding any EC data pools using layouts: https://docs.ceph.com/en/latest/cephfs/createfs/
That's my understanding too. AIUI one reason is that head RADOS objects or some analogue thereof always live there, so there's a significant performance benefit.
I have an cephfs that unfortunately wasn't set up like this: they just made an EC pool on the slow HDDs as the default, which sounds like the worst case scenario to me.
It could be worse - those slow HDDs could be attached via USB 1 ;) I've seen it done.
I would like to add an NVMe data pool to this ceph fs, but recommended gives me pause on if i should instead go through the hassle of creating a new cephfs and migrating all users.
That wouldn't be a horrible idea. My understanding, which may be incomplete, is that one can't factor out and replace the default/root data pool. Something you could do easily would be to edit the CRUSH rule the pool is using to specify the nvme/ssd device class, and the pool will migrate. upmap-remapped.py could be used to moderate the thundering herd. EC still wouldn't be ideal, but this would limit client disruption.
I've tried to run some mdtest with small 1k files to see if i could measure this difference, but speed is about the same in my relatively small tests so far. I'm also not sure what impact I should realistically expect here. I don't even know if creating files counts as "updating backtraces", so my testing might just be pointless.
Are you running with a large number of files for an extended period of time? From multiple clients? Gotta eliminate any cache effects.
I guess my core question is; just how important is this suggestion to keep the default data pool on replicated NVME?
Setup: 14 hosts x 42 HDD + 3 NVMEs for db/wal 2*2x25 GbitE bonds 12 hosts x 10 NVME. 2*2x100 GbitE bonds
Old CephFS setup: - metadata: replicated NVME - data-pools: EC 10+2 on HDD (i plan to add a EC NVME pool here via layouts)
New CephFS setup as recommended: - metadata: replicated NVME - data-pools: replicated NVME (default), EC 8+2 on HDD via layout, EC 8+2 on NVME via layout.
Glad to see that you aren't making k+m = the number of hosts.
Ceph 18.2.7
Best regards, Mikael _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io