CephFS Snapshot Mirroring
Hi everyone, We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup? Thanks!
Hello Vladimir, Please contact croit via https://www.croit.io/contact for unofficial (not yet fully reviewed) patches and mention me. They are currently working on a similar problem and have solved at least the following: * Crashes when two directories are being mirrored in parallel * Data corruption (snapshots on the destination having missing files or files "from the future") due to incorrect cache maintenance in libcephfs * Confusion that results in the failed mirroring when a directory is replaced by a symlink * Useless work done to mirror old snapshots that nobody needs, while the latest snapshot has the highest business value * Useless stat() calls slowing down the mirroring significantly * They also introduced multi-threaded mirroring so that more than one file is copied at a time and more than one subdirectory is scanned at a time - this helps with directories containing many small files Due to the above, I absolutely cannot recommend using the unpatched cephfs-mirror and would rather (if you don't want to contact croit for patches) ask you to switch to rsync with some wrapper script that runs, in parallel, multiple copies operating on different subdirectories. On Fri, Mar 21, 2025 at 12:48 AM Vladimir Cvetkovic <vladimir.cvetkovic@hostpapa.com> wrote:
Hi everyone,
We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
[cc Jos] Hi Alexander, I have a few questions/concerns about the list mentioned below: On Fri, Mar 21, 2025 at 11:09 AM Alexander Patrakov <patrakov@gmail.com> wrote:
Hello Vladimir,
Please contact croit via https://www.croit.io/contact for unofficial (not yet fully reviewed) patches and mention me. They are currently working on a similar problem and have solved at least the following:
* Crashes when two directories are being mirrored in parallel
Are you talking about bi-directional mirroring for the same directory b/w two ceph clusters?
* Data corruption (snapshots on the destination having missing files or files "from the future") due to incorrect cache maintenance in libcephfs
There have been bugs identified in the mirror daemon and the snapdiff library. These have been fixed and merged in the main branch (or under review) waiting to get backported to releases.
* Confusion that results in the failed mirroring when a directory is replaced by a symlink
This is a bug in the mirror daemon when the sync mechanism uses snapdiff -- fix is under review.
* Useless work done to mirror old snapshots that nobody needs, while the latest snapshot has the highest business value
OK. So this is more of an enhancement (that you proposed) than a bug to avoid the mirror daemon starting synchronizing snapshots from the very first and also possibly synchronize the latest snapshot first since that snapshot has the most business value.
* Useless stat() calls slowing down the mirroring significantly
There are some (not many) unnecessary stats in the scan code path, however, most of those stat requests should be satisfied from the client's cache. If you have details on the slowness, please create a tracker with the details. Those unnecessary stats should be done away with irrespective of the requests satisfied from clients cache... Jos?
* They also introduced multi-threaded mirroring so that more than one file is copied at a time and more than one subdirectory is scanned at a time - this helps with directories containing many small files
Again, this is an enhancement feature being developed by Croit. A change has been posted for review.
Due to the above, I absolutely cannot recommend using the unpatched cephfs-mirror and would rather (if you don't want to contact croit for patches) ask you to switch to rsync with some wrapper script that runs, in parallel, multiple copies operating on different subdirectories.
The *real* slowness in the mirror daemon is the data sync for regular files over a slow link b/w clusters. This is due to the fact that the mirror daemon transfers the whole file even if only some bytes have been modified. As you can see, this is inefficient and scales badly with large files. For tentacle release, the MDS is introducing blockdiff operation support -- to be able to get a list of changes blocks for a file between snapshots and the mirror daemon would make use of this. This enhancement will also be backported along with other enhancements by Croit, so latest ceph releases should see a vast improvement in synchronization time.
On Fri, Mar 21, 2025 at 12:48 AM Vladimir Cvetkovic <vladimir.cvetkovic@hostpapa.com> wrote:
Hi everyone,
We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
On Mon, Mar 24, 2025 at 3:17 PM Venky Shankar <vshankar@redhat.com> wrote:
[cc Jos]
Hi Alexander,
I have a few questions/concerns about the list mentioned below:
On Fri, Mar 21, 2025 at 11:09 AM Alexander Patrakov <patrakov@gmail.com> wrote:
Hello Vladimir,
Please contact croit via https://www.croit.io/contact for unofficial (not yet fully reviewed) patches and mention me. They are currently working on a similar problem and have solved at least the following:
* Crashes when two directories are being mirrored in parallel
Are you talking about bi-directional mirroring for the same directory b/w two ceph clusters?
No, I am talking about https://github.com/ceph/ceph/pull/60667, already merged. Double-close, thus unexpectedly closing the fd open by another thread => crash.
* Data corruption (snapshots on the destination having missing files or files "from the future") due to incorrect cache maintenance in libcephfs
There have been bugs identified in the mirror daemon and the snapdiff library. These have been fixed and merged in the main branch (or under review) waiting to get backported to releases.
True.
* Confusion that results in the failed mirroring when a directory is replaced by a symlink
This is a bug in the mirror daemon when the sync mechanism uses snapdiff -- fix is under review.
* Useless work done to mirror old snapshots that nobody needs, while the latest snapshot has the highest business value
OK. So this is more of an enhancement (that you proposed) than a bug to avoid the mirror daemon starting synchronizing snapshots from the very first and also possibly synchronize the latest snapshot first since that snapshot has the most business value.
True.
* Useless stat() calls slowing down the mirroring significantly
There are some (not many) unnecessary stats in the scan code path, however, most of those stat requests should be satisfied from the client's cache. If you have details on the slowness, please create a tracker with the details. Those unnecessary stats should be done away with irrespective of the requests satisfied from clients cache... Jos?
That's interesting, as we had to tune the cache later for a different reason (bug identical to https://tracker.ceph.com/issues/58640#note-11 but affecting patched cephfs-mirror, not ceph-fuse) to avoid reads saturating the interface without any corresponding writes.
* They also introduced multi-threaded mirroring so that more than one file is copied at a time and more than one subdirectory is scanned at a time - this helps with directories containing many small files
Again, this is an enhancement feature being developed by Croit. A change has been posted for review.
Due to the above, I absolutely cannot recommend using the unpatched cephfs-mirror and would rather (if you don't want to contact croit for patches) ask you to switch to rsync with some wrapper script that runs, in parallel, multiple copies operating on different subdirectories.
The *real* slowness in the mirror daemon is the data sync for regular files over a slow link b/w clusters. This is due to the fact that the mirror daemon transfers the whole file even if only some bytes have been modified. As you can see, this is inefficient and scales badly with large files. For tentacle release, the MDS is introducing blockdiff operation support -- to be able to get a list of changes blocks for a file between snapshots and the mirror daemon would make use of this. This enhancement will also be backported along with other enhancements by Croit, so latest ceph releases should see a vast improvement in synchronization time.
Well, let me disagree. While the scenario above is definitely relevant, what we hit is completely different and would also manifest on a 100-gigabit network with zero latency and with the proposed blockdiff optimization applied. The HDD-based cluster has, in some folders, many small files written by many CephFS clients. As cephfs-mirror in its original version has only one sequential thread per directory, it is limited by Q1 IOPS of the cluster: it does not start reading a file until it finishes with the previous one. Therefore, it is limited by the seek latency of HDDs. For the same reason, it is limited by the throughput of one HDD per configured-to-mirror directory. What the patch does is transfer multiple files from the same directory in parallel (thus keeping more than HDD at a time busy), and, in later revisions, also scanning multiple subdirectories of a configured-to-mirror directory in parallel.
On Fri, Mar 21, 2025 at 12:48 AM Vladimir Cvetkovic <vladimir.cvetkovic@hostpapa.com> wrote:
Hi everyone,
We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
-- Alexander Patrakov
Hi Alexander, we are observing similar behaviour on two 19.2.1 clusters that are not in production yet. If you want to debug something, please let me know. We saw the following bugs: * missing data on the destination * data from the past on the destination Regards, Daniel Am 21.03.25 um 06:38 schrieb Alexander Patrakov:
Hello Vladimir,
Please contact croit via https://www.croit.io/contact for unofficial (not yet fully reviewed) patches and mention me. They are currently working on a similar problem and have solved at least the following:
* Crashes when two directories are being mirrored in parallel * Data corruption (snapshots on the destination having missing files or files "from the future") due to incorrect cache maintenance in libcephfs * Confusion that results in the failed mirroring when a directory is replaced by a symlink * Useless work done to mirror old snapshots that nobody needs, while the latest snapshot has the highest business value * Useless stat() calls slowing down the mirroring significantly * They also introduced multi-threaded mirroring so that more than one file is copied at a time and more than one subdirectory is scanned at a time - this helps with directories containing many small files
Due to the above, I absolutely cannot recommend using the unpatched cephfs-mirror and would rather (if you don't want to contact croit for patches) ask you to switch to rsync with some wrapper script that runs, in parallel, multiple copies operating on different subdirectories.
On Fri, Mar 21, 2025 at 12:48 AM Vladimir Cvetkovic <vladimir.cvetkovic@hostpapa.com> wrote:
Hi everyone,
We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444
Hello Daniel, I recommend that you use the same link to talk to croit: https://www.croit.io/contact Caveat: it goes to sales. As far as I know, croit has a production-ready patch for Ceph-18.2.4 and submitted most (all?) of their patches upstream, but there is no ready-made backport to your Ceph version (19.2.x). Sales guys might view this as an opportunity to sell you developer time necessary for backporting and/or try to sell a support contract. Or, if you don't want to talk to croit, the main patch against taking the wrong directory snapshot when mirroring is this one: https://github.com/ceph/ceph/pull/62281 Note that this does not contain the functionality needed to automatically clean up after bugs in old versions of cephfs-mirror. A patch provided by croit contains this functionality, but I could not find any equivalent in public pull requests (probably didn't search well enough). You also may want to check if your Ceph build contains this: https://github.com/ceph/ceph/pull/61100 On Wed, Apr 16, 2025 at 11:26 PM Daniel Schreiber < daniel.schreiber@hrz.tu-chemnitz.de> wrote:
Hi Alexander,
we are observing similar behaviour on two 19.2.1 clusters that are not in production yet. If you want to debug something, please let me know.
We saw the following bugs:
* missing data on the destination * data from the past on the destination
Regards,
Daniel
Hello Vladimir,
Please contact croit via https://www.croit.io/contact for unofficial (not yet fully reviewed) patches and mention me. They are currently working on a similar problem and have solved at least the following:
* Crashes when two directories are being mirrored in parallel * Data corruption (snapshots on the destination having missing files or files "from the future") due to incorrect cache maintenance in libcephfs * Confusion that results in the failed mirroring when a directory is replaced by a symlink * Useless work done to mirror old snapshots that nobody needs, while the latest snapshot has the highest business value * Useless stat() calls slowing down the mirroring significantly * They also introduced multi-threaded mirroring so that more than one file is copied at a time and more than one subdirectory is scanned at a time - this helps with directories containing many small files
Due to the above, I absolutely cannot recommend using the unpatched cephfs-mirror and would rather (if you don't want to contact croit for patches) ask you to switch to rsync with some wrapper script that runs, in parallel, multiple copies operating on different subdirectories.
On Fri, Mar 21, 2025 at 12:48 AM Vladimir Cvetkovic <vladimir.cvetkovic@hostpapa.com> wrote:
Hi everyone,
We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync
Am 21.03.25 um 06:38 schrieb Alexander Patrakov: the
snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup?
Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum
Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany
Tel: +49 371 531 35444
-- Alexander Patrakov
participants (4)
-
Alexander Patrakov
-
Daniel Schreiber
-
Venky Shankar
-
Vladimir Cvetkovic