CephFS mirror very slow (maybe for small files?)
Hi all. I have successfully configured an operational mirror between 2 sites for Ceph FS. The mirroring is running but the speed of data transfer is varying a lot over time (200KB/s – 120MB/s). The network infrastructure between the two Ceph clusters is reliable and should not be the cause of this speed variation. The FS in question, has a lot of small files in it and I suspect this is the cause of the variability – ie, the transfer of many small files will be more impacted by greater site-site latency. If this suspicion is true, what options do I have to improve the overall throughput? * Is it possible to parallelise or “chunk” the transfers with some options to the mirror daemon? * Would the use of multiple snapshot mirror points help? * Note that I am currently forced to use only a single point ( ceph fs snapshot mirror add <fs_name> <path> ) because the part of the FS in use is managed by Openstack Manila which created a subvolume. Requests to add mirrors for sub-directories are therefore denied. Any suggestions for how I can improve this throughput would be most welcome. Currrently I am running Pacific (16.2.10) on the sender and Quincy (17.2.6) on the target. Stuart Cornell Cloud Development Director http://graphcore.ai ** We have updated our privacy policy, which contains important information about how we collect and process your personal data. To read the policy, please click here<http://www.graphcore.ai/privacy> ** This email and its attachments are intended solely for the addressed recipients and may contain confidential or legally privileged information. If you are not the intended recipient you must not copy, distribute or disseminate this email in any way; to do so may be unlawful. Any personal data/special category personal data herein are processed in accordance with UK data protection legislation. All associated feasible security measures are in place. Further details are available from the Privacy Notice on the website and/or from the Company. Graphcore Limited (registered in England and Wales with registration number 10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN. This message was scanned for viruses upon transmission. However Graphcore accepts no liability for any such transmission.
the speed of data transfer is varying a lot over time (200KB/s – 120MB/s). [...] The FS in question, has a lot of small files in it and I suspect this is the cause of the variability – ie, the transfer of many small files will be more impacted by greater site-site latency.
200KB/s on small files across sites? That's pretty good. I have seen rates of 3-5KB/s on some Ceph instances for reading local small files, never mind remotely.
If this suspicion is true, what options do I have to improve the overall throughput?
In practice not much. Perhaps switching to all-RAM storage (with battery backup) for OSDs might help :-). In one case by undoing some of the more egregious issues I managed to improve small file transfer rates locally by 10 times, that is to 40-60KB/s. In your case a 10 times, if achievable, improvement might get you transfer rates of 2MB/s. Often the question is not just longer network latency, but whether your underlying storage can sustain the IOPS needed for "scan" type operations at the same time as user workload. Perhaps it would go a lot faster if you just RSYNC, or even just 'tar -f - -c ... | ssh ... tar =f - -x' (or 'rclone' if you don't use CephFS) and it would be worth doing a test of transferring a directory (or bucket if you don't use CephFS) with small files by RSYNC and/or 'tar' to a non-Ceph remote target and a Ceph remote target to see what you could achieve. No network/sharded filesystem (and very few local ones) handles well small files. In some cases I have seen Ceph was used to store a traditional filesystem image of a type more suitable for small files, mounted on a loop device. https://www.sabi.co.uk/blog/anno05-4th.html?051016#051016 https://www.sabi.co.uk/blog/0909Sep.html?090919#090919
Hi Stuart, I would highly recommend you to have this [1] fix, so that the mirroring works as expected and uses the prev snapshot for syncing. Having multiple mirror daemons also improves the speed. [1] https://github.com/ceph/ceph/pull/54405 - Jos Collin On 13/11/23 21:31, Stuart Cornell wrote:
Hi all. I have successfully configured an operational mirror between 2 sites for Ceph FS. The mirroring is running but the speed of data transfer is varying a lot over time (200KB/s – 120MB/s). The network infrastructure between the two Ceph clusters is reliable and should not be the cause of this speed variation. The FS in question, has a lot of small files in it and I suspect this is the cause of the variability – ie, the transfer of many small files will be more impacted by greater site-site latency. If this suspicion is true, what options do I have to improve the overall throughput?
* Is it possible to parallelise or “chunk” the transfers with some options to the mirror daemon? * Would the use of multiple snapshot mirror points help? * Note that I am currently forced to use only a single point ( ceph fs snapshot mirror add <fs_name> <path> ) because the part of the FS in use is managed by Openstack Manila which created a subvolume. Requests to add mirrors for sub-directories are therefore denied. Any suggestions for how I can improve this throughput would be most welcome. Currrently I am running Pacific (16.2.10) on the sender and Quincy (17.2.6) on the target.
Stuart Cornell Cloud Development Director http://graphcore.ai
** We have updated our privacy policy, which contains important information about how we collect and process your personal data. To read the policy, please click here<http://www.graphcore.ai/privacy> **
This email and its attachments are intended solely for the addressed recipients and may contain confidential or legally privileged information. If you are not the intended recipient you must not copy, distribute or disseminate this email in any way; to do so may be unlawful.
Any personal data/special category personal data herein are processed in accordance with UK data protection legislation. All associated feasible security measures are in place. Further details are available from the Privacy Notice on the website and/or from the Company.
Graphcore Limited (registered in England and Wales with registration number 10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN. This message was scanned for viruses upon transmission. However Graphcore accepts no liability for any such transmission. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thankyou Jos. I will try the multiple daemons to see how that helps. It looks like I need to wait for the fix [1] to be in a release (currently pending review) before I can apply it. Stuart
Hi Jos, I have tried adding multiple daemons but it seems only 1 is active, and there is no improvement in throughput. On further reading, you suggestion conflicts with the docs (https://docs.ceph.com/en/reef/dev/cephfs-mirroring/#:~:text=Multiple%20mirro....) It recommends against multiple daemons and it also says that it "...rebalances the directory assignment amongst the new set thus providing high-availability." This sounds like it can only balance if there are multiple directories registered for snapshot. As mentioned in my OP we must use only 1.
participants (3)
-
Jos Collin
-
pg＠ceph.list.sabi.co.UK
-
Stuart Cornell